MiSTer FPGA Forum

Hello,
On a whim I pulled the SD card from my mister, ran a chkdisk and found undreds of errors (300+)!
On Windows, the OS would warn of unclean disks and suggest a check, but this doesn't happen on Linux.

I fear in the longterm something will be corrupted or damaged, especially if the SDcard is used for savestates.

Does someone have a suggestion to avoid these errors, or Is there any way to have the Mister_data
in a Linux compatible FS (Ext4 or other)?

I use a USB HDD formatted ext4.
On Linux you can check a filesystem using fsck or e2fsck but I do not know if it works with ExFAT. The partition has to be unmounted to run these commands so you would have to unmount /media/fat, run fsck or e2fsck on /dev/mmcblk0p1 if it is even possible to do that on ExFAT. and remount /media/fat when done (I would just do a reboot.

exFAT is just not a terribly reliable filesystem. It has no protection whatsoever against power interruptions or bugs, and it's not native to Linux, really. Its only real strength is interoperability, since all current operating systems can read from and write to it.

Using it as the basis of the Mister distro means that anyone can get it going and load ROMs, no matter what OS they use, so it's an important choice for onboarding newbies and expanding the Mister community. But between FAT and SD cards both being unreliable, it can be a poor choice for longer-term usage.

If you use an outboard hard drive or SSD, you can use any filesystem you like. Linux's native ext4 is a pretty good choice, because it's got a journal, and can thus recover from power failures... usually. It's not perfect, but it's way way better than FAT.

You can also set up a NAS and use any filesystem you like on the server. Ext4 remains a good choice there, too. I'm using ZFS on my server, which is incredibly reliable, but I'm seeing terrible performance at writing hard drive images. The AO486 core, for instance, is glacially slow at writing files to storage. I plan to set up an SSD running ext4 on my server to see if images work better there; I'm suspicious that ZFS's never overwriting blocks is causing the extremely bad performance.

I'd avoid btrfs; they've never really gotten that thing fully reliable. Most folks running desktops don't have too much trouble with it anymore, but I've heard so many horror stories about people who've depended on it, and lost data. ext4 seems much more trustworthy. It's been holding up Linux systems successfully for a long, long, long time.

edit: out of interest, I used Windows to check the Mister drive, and it had "volume bitmap errors", which I think are sectors incorrectly marked as used, with no associated files or directories. There was no data damage, and Windows fixed it fine, but the filesystem was slightly corrupt.

I already have an external USB drive for the ROMs (ext4), but the problem is that the SDcard is still mandatory.

Zagg3r wrote: ↑Tue Jan 03, 2023 5:04 pm

I already have an external USB drive for the ROMs (ext4), but the problem is that the SDcard is still mandatory.

It is still mandatory but you can limit its usage to the bare minimum.
@theypsilon will scream but this is what I did:
I moved everything (almost) to the USB drive and for each directory on the USB drive I created a symlink SD Card -> USB drive.
The only things left on the SD cards are config, linux, Scripts, MISTer, MISTer.ini, menu.rbf and downloader.ini. Not much left on the SD card and easy to redo/replace if something goes wrong.

MiSTer does have exfat utilities installed. Log in and you can scan the SD card even when mounted:

/root# fsck.exfat /dev/mmcblk0p1
exfatprogs version : 1.0.4
/dev/mmcblk0p1: clean. directories 2994, files 43687

You going to have to umount the SD card for repairs, however. Perhaps someone can chime-in if that's possible while the MiSTer is running.

Zagg3r wrote: ↑Tue Jan 03, 2023 5:04 pm

I already have an external USB drive for the ROMs (ext4), but the problem is that the SDcard is still mandatory.

Unfortunately, that's a hardware requirement; the DE-10 will only boot from a micro SD. You cannot use that board without providing one.

If you know enough about Linux, you could replicate the file structure onto a USB drive, and add u-boot arguments to force it to boot from the filesystem on your chosen storage medium. But that could very possibly blow up spectacularly if update_all ever tried to do a Linux system update.

An easier method would be to reburn the Mr. Fusion image to the SD, connect the external storage, and set it up to mount as /media/fat/usb0. At least hypothetically, there's a priority system to use /media/fat/cifs (for a NAS server) or /media/fat/usb0 (for an external drive) before using regular /media/fat. Update_all is supposed to download everything into those dirs, if one exists. But when I was last doing this, early last year, update_all resolutely refused to put lots of stuff into /media/fat/cifs. It may have been fixed since, and I'd definitely give it a try since you have a mostly new install, but it may not be reliable. If it does work correctly, then you have a minimum install on the SD, with almost everything external, so if the exFAT SD filesystem blows up, you can return to where you are now very quickly, and without losing data.

If update_all is still incomplete/broken, then as jca says, you can use symlinks in /media/fat to force redirection into /media/fat/usb0. FAT symlinks aren't "real", but a text file in a specific format. The Linux kernel understands the format and will use them transparently, but other OSes will not, so if you mount the microSD under Windows or MacOS, it'll look weird, inconsistent with what you see from the Linux CLI. You'll see short text files instead of symlinks. If you have to rebuild the SD card, you will have to recreate any symlinks you made from Linux.

And, of course, if you format your external storage with ext4, then it will be much more reliable, but you won't be able to store console ROMs on it easily from Windows or a Mac. You'll have to either run a server on the Mister (like FTP or SMB), and transmit them over the network, or download your ROMs using the Linux command line.

@Krypto Thanks for fsck.exfat.
Real fast when you have almost everything on USB as I mentioned above.

/root# fsck.exfat /dev/mmcblk0p1
exfatprogs version : 1.0.4
/dev/mmcblk0p1: clean. directories 16, files 184

Oh, a note: if you create an external ext4 filesystem, you'll probably want to use the 'sync' argument as a mount option. This will slow down writes fairly substantially, but it will mean that the filesystem should stay consistent over time, even if the Mister crashes or you do a cold reboot. (which I do pretty frequently.) ext4 is pretty robust, but it's not really designed around the constant cold reboots that Mister users often do.

If your ext4 filesystem is on an SSD, you probably also want the 'discard' option, which will send TRIM commands as sectors are freed up. This will help maintain the SSD's speed over time, because it will know what sectors are free. SSDs move things around and coalesce free space when they're not being actively written to, and TRIMs let them do this much more intelligently.

I keep my library and config backups on a USB drive formatted as EXT4, it is mounted by default as rw,nodev,noexec,noatime,nodiratime,sync. I think that is sufficient for a USB thumb drive, noting that there isn't a lot of churn on any volume in MiSTer as say a desktop PC drive would have.

I mount a folder on my NAS with NFS. The ZFS array is basically RAID5.

I am still on my original install from over 3 years ago and even migrated it to a larger SD card over the years.

I always though the MiSTer backend is designed to prevent SDcard corruption ?

Armakuni wrote: ↑Sat Sep 23, 2023 12:27 pm

I always though the MiSTer backend is designed to prevent SDcard corruption ?

That really isn't possible - that's the job of the filesystem.

What MiSTer (Main) has done is kept to a minimum how much writing it does to the SD card to reduce the possibility of SD card corruption and to also extend the life of it since flash media has a finite amount of writes.
It doesn't actively prevent corruption....it just tries really hard not to be the cause.

Having cores with virtualized HDD's should be carefully shut down much like you would treat a real disk. Corruption happens in the virtual medium, but also to the image file as Linux sees it. So switch to a core without any writable media, then turn off. Still: Linux was never made to have the power just yanked out from under it. That's why I have my writable media stored on a NAS and treat the SD as mostly read-only except during updates.

Armakuni wrote: ↑Sat Sep 23, 2023 12:27 pm

I am still on my original install from over 3 years ago and even migrated it to a larger SD card over the years.

I always though the MiSTer backend is designed to prevent SDcard corruption ?

Not really possible on an SD card. It's up to the filesystem and the controller built into the card. FAT32 doesn't do wear-leveling and neither do most micro SD cards. If you look through the settings script, there are a handful of settings that have warnings attached to them saying that enabling them may wear out your SD card because they write data to the same locations every time you load a core or a ROM.

Antoine.WG wrote: ↑Sat Sep 23, 2023 8:50 pm

Armakuni wrote: ↑Sat Sep 23, 2023 12:27 pm

I am still on my original install from over 3 years ago and even migrated it to a larger SD card over the years.

I always though the MiSTer backend is designed to prevent SDcard corruption ?

Not really possible on an SD card. It's up to the filesystem and the controller built into the card. FAT32 doesn't do wear-leveling and neither do most micro SD cards. If you look through the settings script, there are a handful of settings that have warnings attached to them saying that enabling them may wear out your SD card because they write data to the same locations every time you load a core or a ROM.

I am not talking about wear though, if you have ever used a Retropie build that is just a pot luck until the SD card gets corrupted

I would have thought most people use common sense practice and replace the SD card after a certain amount of time like any other back up media mechanical hard drives for example ?

In general use I have not had a single issues in over three years, never had to reinstall or repair, which is some feat alone imho

Excluding wear, an SD does not get corrupted. The filesystem does. That's minimized (like wear) by preventing writes. That's done by moving all write tasks to alternative media like a NAS. A NAS manages its own integrity mostly independent of its clients.
If you do need writes, make sure they get a chance to finish and file handles are properly terminated on the Linux OS so the filesystem doesn't remain in a wonky state. That actually calls for a clean shutdown (sync before termination) function, which I haven't yet seen on MiSTer.
Another option is to use a filesystem other than FAT-variants that's more resilient against power failure, like a journaling FS of some type that always keeps consistent metadata. You may miss a write from time to time then, but the FS structure remains valid.

Bas wrote: ↑Sun Sep 24, 2023 7:32 am

Excluding wear, an SD does not get corrupted. The filesystem does. That's minimized (like wear) by preventing writes. That's done by moving all write tasks to alternative media like a NAS. A NAS manages its own integrity mostly independent of its clients.
If you do need writes, make sure they get a chance to finish and file handles are properly terminated on the Linux OS so the filesystem doesn't remain in a wonky state. That actually calls for a clean shutdown (sync before termination) function, which I haven't yet seen on MiSTer.
Another option is to use a filesystem other than FAT-variants that's more resilient against power failure, like a journaling FS of some type that always keeps consistent metadata. You may miss a write from time to time then, but the FS structure remains valid.

Yes and MiSTer Main is designed to reduce the amount of SD writes which helps prevent corruption. You also dont need to do a clean shutdown due to this

This is also why it was recommended to run the ZXNext core from the secondary SD card slot originally as the core needed to write back to SDcard all the time

Armakuni wrote: ↑Sun Sep 24, 2023 10:33 am

Bas wrote: ↑Sun Sep 24, 2023 7:32 am

Excluding wear, an SD does not get corrupted. The filesystem does. That's minimized (like wear) by preventing writes. That's done by moving all write tasks to alternative media like a NAS. A NAS manages its own integrity mostly independent of its clients.
If you do need writes, make sure they get a chance to finish and file handles are properly terminated on the Linux OS so the filesystem doesn't remain in a wonky state. That actually calls for a clean shutdown (sync before termination) function, which I haven't yet seen on MiSTer.
Another option is to use a filesystem other than FAT-variants that's more resilient against power failure, like a journaling FS of some type that always keeps consistent metadata. You may miss a write from time to time then, but the FS structure remains valid.

Yes and MiSTer Main is designed to reduce the amount of SD writes which helps prevent corruption. You also dont need to do a clean shutdown due to this

This is also why it was recommended to run the ZXNext core from the secondary SD card slot originally as the core needed to write back to SDcard all the time

Yes and no. Most cores aren't sensitive to being shut down incorrectly. Cores that use VHDs are just as vulnerable to improper shutdowns as whatever you're running on them. Windows 95 running on the AO486 core will still detect an improper shutdown and scandisk will probably find errors for whatever files were open, just like powering off a real PC without doing a shutdown first.

Antoine.WG wrote: ↑Mon Sep 25, 2023 2:50 am

Armakuni wrote: ↑Sun Sep 24, 2023 10:33 am

Bas wrote: ↑Sun Sep 24, 2023 7:32 am

Excluding wear, an SD does not get corrupted. The filesystem does. That's minimized (like wear) by preventing writes. That's done by moving all write tasks to alternative media like a NAS. A NAS manages its own integrity mostly independent of its clients.
If you do need writes, make sure they get a chance to finish and file handles are properly terminated on the Linux OS so the filesystem doesn't remain in a wonky state. That actually calls for a clean shutdown (sync before termination) function, which I haven't yet seen on MiSTer.
Another option is to use a filesystem other than FAT-variants that's more resilient against power failure, like a journaling FS of some type that always keeps consistent metadata. You may miss a write from time to time then, but the FS structure remains valid.

Yes and MiSTer Main is designed to reduce the amount of SD writes which helps prevent corruption. You also dont need to do a clean shutdown due to this

This is also why it was recommended to run the ZXNext core from the secondary SD card slot originally as the core needed to write back to SDcard all the time

Yes and no. Most cores aren't sensitive to being shut down incorrectly. Cores that use VHDs are just as vulnerable to improper shutdowns as whatever you're running on them. Windows 95 running on the AO486 core will still detect an improper shutdown and scandisk will probably find errors for whatever files were open, just like powering off a real PC without doing a shutdown first.

I meant the Linux backend and filesystem handling is designed for lack of a proper shutdown, The cores cannot write back to the filesystem unless you bring up the OSD. This will be what causes the issues with windows in the AO486 core

Armakuni wrote: ↑Mon Sep 25, 2023 2:49 pm

Antoine.WG wrote: ↑Mon Sep 25, 2023 2:50 am

Armakuni wrote: ↑Sun Sep 24, 2023 10:33 am

Yes and MiSTer Main is designed to reduce the amount of SD writes which helps prevent corruption. You also dont need to do a clean shutdown due to this

This is also why it was recommended to run the ZXNext core from the secondary SD card slot originally as the core needed to write back to SDcard all the time

Yes and no. Most cores aren't sensitive to being shut down incorrectly. Cores that use VHDs are just as vulnerable to improper shutdowns as whatever you're running on them. Windows 95 running on the AO486 core will still detect an improper shutdown and scandisk will probably find errors for whatever files were open, just like powering off a real PC without doing a shutdown first.

I meant the Linux backend and filesystem handling is designed for lack of a proper shutdown, The cores cannot write back to the filesystem unless you bring up the OSD. This will be what causes the issues with windows in the AO486 core

I just tested this by creating a file on my Windows 95 desktop, then power cycling my MiSTer without bringing up the OSD.
Not only did Windows detect an improper shutdown and run SCANDISK, but the file is still there.

Indeed the cores can and do write to the SD through Main. They get offered a "bunch of sectors" that they treat as a disk. That bunch of sectors lives in a file on the SD and ExFAT -to my knowledge- has no provisions against power failure to maintain integrity. So you have risk of corruption both inside and outside the core's context. Inside is what Windows 95 told you about the unclean shutdown. The FAT structure on the SD won't be damaged by writing within an existing file. No new sectors got allocated or deallocated while you were working in Windows. The biggest risk to the ExFAT is when running updater scripts that do a lot of shuffling things around.