Backup of SD Card + USB Disk to NAS

[zlyclovek]

I have Mister for several months and it is a great tool that is giving me more and more joy. I am trying to look into all news and Mister has already brought me to several interesting systems that I did not even know they existed.

Now when I have a system with several ROMs and VHDs I was thinking about a backup plan when something fails.

So the question is how do you backup Mister file system. I have QNAP NAS and I was trying to schedule daily job to synchronize files. It works, but it takes several hours even after first backup was successful. I was hoping that next diff backups will be quick but they are as slow as first backup.

I tries to use SMB and rsync connections but with very similar results... very slow (about 11 hours for 120 GB).

I tried to search forum and I could not find any relevant topic, so I am starting this new one. I know that it is possible to store data on NAS and use it on Mister, but I would prefer to have primary data on Mister itself.

The question is: what is the preferred backup plan and how to make it reasonably fast for diff backups? Is there a preffered way?

[Pitou]

Basically, you just need to backup the /media/fat directory
If your SD fails, you can re-install from start and then copy back the fat dir.

I'm doing the setup on mine and doing backups on my NAS as I progress. At the end it's only a matter of a couple of files to update (like the arcade cores)

All of this is a manual process, as you mentioned the other way is to use rsync, it should have an option to only update what's missing.

Pitou!

[fierman]

rsync is the best way really. Just don't use compression (-z) , that will significantly impact the transfer speed.

[PistolsAtDawn]

I have not had good experiences with rsync over samba outside of the MiSTer. Some advice I found a while back that seems to have been true in my experience:

Rsync's performance advantage comes from doing delta transfers-- that is, moving only the changed bits in a file. In order to determine the changed bits, the file has to be read by the source and destination hosts and block checksums compared to determine which bits changed. This is the "magic" part of rsync-- the rsync algorithm itself.

When you're mounting the destination volume with SMB and using rsync to copy files from what Linux "sees" as a local source and a local destination (both mounted on that machine), most modern rsync versions switch to 'whole file' copy mode, and switch off the delta copy algorithm. This is a "win" because, with the delta-copy algorithm on, rsync would read the entire destination file (over the wire from the NAS) in order to determine what bits of the file have changed.

The "right way" to use rsync is to run the rsync server on one machine and the rsync client on the other. Each machine will read files from its own local storage (which should be very fast), agree on what bits of the files have changed, and only transfer those bits. They way you're using rsync amounts of a trumped-up 'cp'. You could accomplish the same thing with 'cp' and it would probably be faster.

If your NAS device supports running an rsync server (or client) then you're in business. If you're just going to mount it on the source machine via SMB then you might as well just use 'cp' to copy the files.

https://serverfault.com/questions/68270/speeding-up-rsync-over-smb

[JamesH]

rsync is quite complex piece of software and for any non-trivial use case reading the docs is mandatory. For example, speaking of whole file transfers, IIRC, one can always force delta-transfers with --no-whole-file

However, even without any other benefits, rsync brings "interruptable" backups to the table. But you have to explicitly request it with --partial

One thing to consider, though. The "best" option depends on the actual files. If there are quite a lot of hard disk images there, a backup scheme which performs compression and/or deduplication may be faster and reduce backup size. Apparently, compressing ZIP's is useless.

[Stupid Dufus]

Not sure if it'd work for your needs, but could you store the ROMs and cores on your NAS and load them using cifs on the mister?

That way the bulk of the data is already on the NAS and the only things you'd be backing up from the SD card would be config, scripts, etc., which I imagine would sync pretty quickly since they're not too large of files in comparison.

[Lightwave]

If you have a computer with network access to the MiSTer and the NAS, you could use something like FreeFileSync (open source and free for Windows/Max/Linux). You can set it to only copy the changed files based on date and file size (checksum as well, but that takes quite long to generate for large files). The benefit of doing it this way is that you get to see exactly what has been added/updated, and can choose to ignore certain changes if you wish (also helpful if you were fooling around with scripts and configurations that you wish to revert).

I run this manually to backup after an update all, but you can also schedule it to run automatically.

[cheddargo]

I store all games and cores in a shared Synology NAS folder and access them via CIFS mount. Works like a charm. The NAS is connected to an external USB Drive for regular backups. The only thing left to backup on actual MiSTer SD Card are save games, custom scripts, settings, ini-Files or menu background images in /media/fat.

[chunky_tesco]

rsync -avhP --delete --backup --backup-dir={destination}/MiSTer_DELETED_FILES/$(date "+%Y%m%d-%H%M%S") --recursive --times root@{MiSTer_IP_address}:/media/fat/ destination}/MiSTer_backup/

this is the command I use to backup my SD card to a network drive in a folder name MiSTer_backup. Files to be deleted are moved to MiSTer_DELETED_FILES in a datetime stamped folder. You can add an exclude flag for individual files e.g. --exclude='MegaAGS.hdf'