[New version!] SD Card Migration Utility for MiSTer: An easy way to migrate to a new SD card, no PC required

[Natrox]

Like my work?
Support me on https://ko-fi.com/natrox!

[Changelog]
2022-11-04 - v0.2.0 - Added functionality to exclude folders from the migration, users can pick which ones to skip from an optional menu.
2022-10-30 - v0.1.0 - Initial release.

Original post:

Hello everyone!
I am happy to announce that I have released a new utility for MiSTer; the SD Card Migration Utility!

You can use this utility to migrate to another SD card. You can even migrate to a smaller card if you wish - as long as the space used on your source SD card does not exceed the capacity of the new SD card.

No more will you need to use MrFusion to set up a new SD card and copy files over manually! This utility runs from your MiSTer and does everything for you. After it finishes, the SD card is ready for use right away.

Limitation: It won't work with SD cards that are 2TB or larger - but good luck finding one of those!!

Description: (copied from my repository):
This utility can be used on a running MiSTer system to migrate to a new SD card.
A USB SD card adapter is required to be connected to your MiSTer.
The target SD card can be of any size - as long as it can fit all of the data on the source SD card.
It performs a number of checks to make sure that the migration will be successful.

A key feature of this utility is that it looks at the source SD card's disk geometry and reproduces the required geometry on the new SD card.
This means that the U-Boot partition located at the tail-end of the disk is copied over, while the rest of the card is formatted to exFAT for data use.
All of the original data is copied over by default - version 0.2.0 introduces folder filtering, giving you the option to skip folders if you so desire. This utility is designed to be run from the MiSTer main menu, it will guide you through disk selection and it performs the necessary checks. During the process, it will freeze the main MiSTer binary to ensure file integrity.
It can also be used to make a backup SD - although it is not suitable for incremental backups.

Download:
For wget: https://raw.githubusercontent.com/Natro ... rate_sd.sh
GitHub viewer: https://github.com/Natrox/MiSTer_Utils_ ... rate_sd.sh
Repository: https://github.com/Natrox/MiSTer_Utils_Natrox

Installation:
Download "migrate_sd.sh" in whatever way you please, and put it in the "/media/fat/Scripts" folder on your MiSTer!
You can then run it from the main menu like any other script.

I hope you will find this script useful. Please let me know if you have any questions, problems or nice words.

[Neocaron]

That is actually awesome! I wanted to move from an almost full 256gb to a 512gb sd card, but I was too lazy to go through the entire process of transfering all the files and games and formating the new card, so this is great! Perfect timing too! XD Thank you!

[toastboy]

same here 256->512. Bought the card, then put off the effort of moving the data for ages.

This will make it much easier, Ty!

[Natrox]

Happy to help! Let me know if you run into any problems, and also let me know if everything goes smoothly.

[Neocaron]

So it works absolutely great, I think it does an amazing job at creating additional sd cards and It's easier than mr fusion and faster, because everything is setup from the og card directly from the Mister with all my custom settings and content already! It's gonna be my new go to on that front.

However since the usb 2 is very slow, we should have an option to copy everything or having the choice to pick game categories. Because each PSX games/ Saturn/MSU/Ao486 are taking minutes to copy. The script is awesome but very much limited by the usb2 specification of the board.

Still it's possible if you don't have a huge cd library, or you are in no rush (I'm talking hours) to just start the process and let it be until it's done.

[Natrox]

So it works absolutely great, I think it does an amazing job at creating additional sd cards and It's easier than mr fusion and faster, because everything is setup from the og card directly from the Mister with all my custom settings and content already! It's gonna be my new go to on that front.

However since the usb 2 is very slow, we should have an option to copy everything or having the choice to pick game categories. Because each PSX games/ Saturn/MSU/Ao486 are taking minutes to copy. The script is awesome but very much limited by the usb2 specification of the board.

Still it's possible if you don't have a huge cd library, or you are in no rush (I'm talking hours) to just start the process and let it be until it's done.

I am glad to hear you like it so far!

Very true, it does take a long time to copy over everything. For me it took about 30 minutes for 42GB, so it's kind of an overnight job I suppose. :V
I like your suggestion! I think I will be able to add a filtering option for the next version. Thank you.

[Neocaron]

Awesome, glad to see my feedback will help

[toastboy]

I'm in the middle of running the migration now.

The script so far is excellent, it has clear colourful menus, it autodetects the new card inserted over USB, has the usual "don't erase the wrong thing" warnings. I really can't fault it.

In my case the migration will be 220GB, so yes it'll be running a long time, but it doesn't need my help while it's running, so no big deal.

[Longtime4321]

This came at the exact right time! I tried to migrate my SD card about a week ago manually, and gave up. Thanks!

[ntt]

Thank you @Natrox!

[toastboy]

Migration finished, SDcard swapped and works perfectly.

[kubbie]

Migration finished, SDcard swapped and works perfectly.

How long did it take to migrate 220GB? I'm thinking of migrating from 128GB to 256GB.

[PikWik]

thanks for making this!
i have a unique situation where my main MiSTer is tucked in an arcade cabinet and a pain to get the SD card out.
having this utility to plug in an SD card reader and transfer everything over with a USB cable is a very nice thing to have

[Longtime4321]

Mine won't detect my SD card. I'm using a sandisk Micro sd to usb converter, so I can't imagine it would be the converters fault.
Would it be possible to add support for the sd card slot on the I/O Board?
Edit: Does the SD card have to be formated as fat32 or anything? Mine is currently a corrupt MiSTer install
Edit2: Formating it as FAT32 did the trick.

[Natrox]

Mine won't detect my SD card. I'm using a sandisk Micro sd to usb converter, so I can't imagine it would be the converters fault.
Would it be possible to add support for the sd card slot on the I/O Board?
Edit: Does the SD card have to be formated as fat32 or anything? Mine is currently a corrupt MiSTer install
Edit2: Formating it as FAT32 did the trick.

Interesting, normally it should not matter whether the card is corrupt or not - I will have to run some tests to see why that is happening.
It seems likely this is the adapter's fault, but I will have to look into it - thanks for bringing it to my attention!

Sadly, the secondary SD card slot is not hooked up to the ARM processor - so it does not show up in the Linux side of things.

[toastboy]

How long did it take to migrate 220GB? I'm thinking of migrating from 128GB to 256GB.

Around 9 hours, I just left it running in an ssh session while I was working all day.

[Natrox]

I am happy to announce version 0.2.0! The major change of this version is that I have added Neocaron's suggestion.

The utility now asks you if you would like to filter out folders, and if you choose to do so, you are given a sorted list of folders that you can select/deselect for migration.

Due to some constraints with how scripts are run via the MiSTer binary, the approach may look a little janky. It is fully functional though!

To get the new version, all you need to do is re-download from the links in the OP. Thanks again for your patronage and I look forward to any feedback!

[ntt]

HI Natrox
I've just copied the mister SD card with your script: everything went well, I could even skip the contents of the games folder, I couldn't be happier!
Just one question - or perhaps a proposal for next version: would it be possible to flag the entire games folder as not to be copied, instead of having to go through all of its subfolders one by one? Curiously enough I could toggle off any other folders but apparently not the entire games folder itself (or possibly I've missed it among the other 120+ subfolders being listed )
Again, thank you for this wonderful piece of software!

[Natrox]

HI Natrox
I've just copied the mister SD card with your script: everything went well, I could even skip the contents of the games folder, I couldn't be happier!
Just one question - or perhaps a proposal for next version: would it be possible to flag the entire games folder as not to be copied, instead of having to go through all of its subfolders one by one? Curiously enough I could toggle off any other folders but apparently not the entire games folder itself (or possibly I've missed it among the other 120+ subfolders being listed )
Again, thank you for this wonderful piece of software!

Hey there,
Thank you for using my script! I could certainly add functionality for that - right now I leave out the "games" folder by-design, as it is scanned separately.
I will look into how to best handle this from a UX point-of-view. For example, say you accidentally toggle the games folder off, while you were doing very fine-grained toggling for its sub-folders, it would be annoying if you lose all of that. I have some ideas on how to handle it though!

There are more things I am planning for the next version:
- A slight change in flow, right now you select a disk and then filter your data, I want the following:
Disk scan and report -> Filtering with total space calculation -> (Filtering is saved in case the user quits out) -> Disk selection
- During filtering, I want to show a live view of space saved -or- total size of everything post-filtering.
- I might end up making this more like a guided install program, with a main menu and such, such that if you cancel out you can try again right away.
- A nice progress bar for the file copy including estimated time.

This will all take a while; I will probably end up polishing this into version 1.0.0 and I may end up switching to a compiled language rather than Bash.

[ntt]

Thank you for always considering user feedback.
I'm looking forward to the evolution of this tool, it's already very useful as it is

[evilsim]

Hi Natrox,

what a wonderful script, great idea too, nothing else does THIS job.
I love nothing more than a functional, well written bash script

I have a question, I would like to simply use this script, to "prepare" an SD card, then I can copy everything over myself on a device other than the mister itself.

To do this, if there were a 'de-select all' option when the listing of directories (step 3.5) to choose to migrate comes up, I could choose that - or an option to simply skip copying the 'other' folders. Then only the critical folders would be copied and the sdcard 'prepared'.

Once the process is finished (paritioning, u-boot, copying linux folder etc) then I can immediately check if the SD card works on a mister. I can then copy all the data myself on another linux/windows box, which will make copying the data much faster (if you use good quality sdcard and good quality sdcard read/write devices). In Windows I might use fastcopy or robocopy and in Linux rsync.

Fantastic tool, thanks so much for all your work on this

edit: I also noticed that the exfat partition has no label, this could possible be a feature added to the script as well.

[unclefunkle999]

Hi Natrox,

I tried running the script however I get the following error messages:

line 0: syntax error near unexpected token 'newline'
line 0: '<!DOCTYPE HTML>'

I'm a complete newb when it comes to programming so any help you can provide would be greatly appreciated. Thanks!

EDIT: I GOT IT WORKING!! This script is a godsend. THANK YOU FOR ALL OF YOUR HARD WORK!!!!

[Malor]

Who were you, DenverCoder9? What did you see?!

[MrMartian]

I love me a good xkcd reference..

[Natrox]

I have a question, I would like to simply use this script, to "prepare" an SD card, then I can copy everything over myself on a device other than the mister itself.

Hm, to be honest, it would probably be easier to just run Mr. Fusion again. If you need to prepare multiple MiSTers, you could image the SD after using Mr. Fusion and do it like that. Right now, you can still deselect everything by hand and it should work. You can also interrupt the file copy operation and then mount the SD manually to copy over what you need.

Hi Natrox,

I tried running the script however I get the following error messages:

line 0: syntax error near unexpected token 'newline'
line 0: '<!DOCTYPE HTML>'

I'm a complete newb when it comes to programming so any help you can provide would be greatly appreciated. Thanks!

EDIT: I GOT IT WORKING!! This script is a godsend. THANK YOU FOR ALL OF YOUR HARD WORK!!!!

Glad to see you got it working, and you're welcome.

For others running into this problem: you have to use "View Raw" on GitHub and download that file. Otherwise, you will download the HTML file instead.

[TuffyDK]

Hi Natrox.

I've a 128G SD card in my Mister and want to get a smaller SD card, like 32GB, cuz I moved all the games (/Games) folder to a USBKey (/media/USB0)

If I run your script, will It only look at SD card at (/media/fat) and copy it to a new SD card, or will it scan for other attachments like my USBKEY??

Perhaps its a good idea to pull out my USBKEY before running your script???

Regards
TuffyDK

[Natrox]

Hi Natrox.

I've a 128G SD card in my Mister and want to get a smaller SD card, like 32GB, cuz I moved all the games (/Games) folder to a USBKey (/media/USB0)

If I run your script, will It only look at SD card at (/media/fat) and copy it to a new SD card, or will it scan for other attachments like my USBKEY??

Perhaps its a good idea to pull out my USBKEY before running your script???

Regards
TuffyDK

Hey there Tuffy,

Technically it should only look at the /media/fat folder, but as far as I understand there is some functionality to overlap folders from USB storage onto the SD card root.

For safety, I recommend disconnecting your USB key before running the script.

Let me know if you run into any issues!

[TuffyDK]

Hi Natrox.

I've a 128G SD card in my Mister and want to get a smaller SD card, like 32GB, cuz I moved all the games (/Games) folder to a USBKey (/media/USB0)

If I run your script, will It only look at SD card at (/media/fat) and copy it to a new SD card, or will it scan for other attachments like my USBKEY??

Perhaps its a good idea to pull out my USBKEY before running your script???

Regards
TuffyDK

Hey there Tuffy,

Technically it should only look at the /media/fat folder, but as far as I understand there is some functionality to overlap folders from USB storage onto the SD card root.

For safety, I recommend disconnecting your USB key before running the script.

Let me know if you run into any issues!

I'll get back when I have tried it - Not today, but sometime in the future - I think I will make a backup with Clonezilla first - Just to be sure

/Tuffy

[Orfax]

Thank you Natrox for a fantastic script. It was painless swapping over SD cards.

[Longtime4321]

It's a real shame that the Linux side of MiSTer can't access the second SD card slot, that's a huge waste of potential. I suppose it is what it is but it's still unfortunate. Regardless, excellent script.
Has the issue been fixed where it won't detect the SD card if the SD card isn't Fat32?