MiSTer FPGA Forum

There is a good and bad "news" about symlinks on exFAT used on MiSTer.
Good: I guess, many already noticed you can create a symlink on exFAT.
Bad: it's not native feature of exFAT (and any other FAT) file system and Linux uses undocumented reserved flag for that. It works till you try to copy files on Windows and other OS supporting exFAT. Once you copy symlink, it becomes a generic text file. It won't be recognized as symlink anymore.

What i plan to do is to change the symlink flag to system flag (S). It's documented file attribute and pretty much useless in Linux. When you will copy such file, it should be preserved - it of course depends on app where you copy, but at least you can find the way to copy and keep the attributes.

As a linux user accustomed to symlinks, I think it is good news that one has found a way to implement them on exFAT. With the planned change, will symlinks created by the vanilla linux driver still be recognised as such by MiSTer and vice versa?

I don't know about other systems. FAT file systems don't support linux links. So it's workaround just for MiSTer.

Is there some good reason to keep using exFAT instead of moving to an ext* filesystem? Is it just for convenience for Windows users? There are ext* filesystem drivers available for Windows these days.

Keith, really, can you point me to a usable ext* driver for Windows?

I think exFat is a nice solution as it is read-/writeable on all mayor systems natively these days.
Only caveat are the symlinks, but honestly I for my part don't use a single one on my cards. But as always YMMV

exFAT is pretty much right file system for MiSTer.
It doesn't have complexities such as access rights, case sensitivity and supported in all OS.
Links are basically not supposed to be used in ROM folders. They give some convenience but not necessary.

The Nintendo Switch modding community largely dissuades the use of exFAT on the SD card because it's prone to corruption. My understanding of the reason for it is down to the driver implementation. I assume the driver being used for MiSTer is more robust.

tRAcE wrote: ↑Sun Jun 07, 2020 8:02 am Keith, really, can you point me to a usable ext* driver for Windows?

https://www.paragon-software.com/us/hom ... s-windows/

So, will something bad happen if I use chkdsk /F against the exFAT partition (given that it is somehow non-standard) with the microSD card inserted into my Windows 10 machine? Because I just ran it without the /F flag, and it claims to have found problems:

I had a recent backup, so I went ahead and ran chkdsk /F against the exFAT partition, and it seems to have fixed whatever problem it found without messing up any of my symlinks. So I guess that was a safe enough thing to do

It appears that if a chkdsk-validated microSD exFAT partition is used in MiSTer, and then MiSTer is simply powered off from the OSD, it somehow corrupts the "volume bitmap" of that partition. At least, that's what chkdsk again reports if I remove the microSD from MiSTer and check it again on my Windows 10 PC. I repeated this cycle a couple times to confirm the issue.

So then I wrote a quick /media/fat/Scripts/shutdown.sh that I can easily run from the OSD: As long as I use that shutdown script before powering down each time, then the exFAT partition still reads as good when I check the microSD again on my Windows PC.

So it seems the standard guidance about "just go to the OSD and power off" is still not great. It would be better to add a proper shutdown command to the main screen of the OSD, so the user doesn't have to go hunting through the Scripts folder just to shut down. (For now, I created a symlink /media/fat/Scripts/_shutdown.sh just to force it to appear first in the sort order, but that's still a bit of a kludge.)

There is some flag in exFAT telling that not all writeable files are closed before unmounting. It doesn't mean corruption.
So general guide remains the same. There WON'T be shutdown procedure for MiSTer. But if your config went too far and something constantly is writing to files, then you can invent some specific options like shutdown script. Up to you, but unnecessary for normal use.

Also chkdsk doesn't check extended unused attributes for files on exFAT. So it won't find anything. Just create some link file on MiSTer then plug SD card to Windows and copy this link file to other place on the same SD card. Then plug it back in MiSTer and you will see it's no longer a link but normal text file.

The only issue with symlinks is that each file will occupy the size of one block on the SD Card (128K for SD cards larger than 32GB). Sadly, this cannot be avoided as there is a strict 1 file per block limitation and symlink files are usually less than 100 bytes. Therefore symlinks on MiSTer right now only save space if the file is greater than 128K.

Abusing feature is usually a bad idea. Links on exFAT give some convenience, but making a tons of links isn't good.

The block size seems to effect more than symlinks. Small files in general are effected. As they are forced to be at 128K minimum regardless of their size.

On larger SD cards this is not as felt but on smaller SD cards a lot of space can be eaten up by the block size.

One example that was recently pointed out to me was a 40MB NES set taking 70MB because of the block size.

I'd like to understand more of the reasons to set block size smaller or larger and what's effected and what is a good fit for MiSTer and it's users?

mkfs.exfat sets defaults at:

Number of physical sectors per cluster (cluster is an allocation unit in exFAT). Must be a power of 2, i.e. 1, 2, 4, 8, etc. Cluster size can not exceed 32 MB. Default cluster sizes are: 4 KB if volume size is less than 256 MB, 32 KB if volume size is from 256 MB to 32 GB, 128 KB if volume size is 32 GB or larger.

Does someone know the defaults for MiSTer in the SD installer and why are they choosen?

I checked out the SD Installer and it looks like BlockSize is not specified. So it'll just use the default you listed there.

The main issue you have with BlockSize is it's a compromise between used space per file and read/write speeds. If Blocks are large, then large files read/write faster but small files take up more space. If blocks are small, then small files take up less space but large files need more read/write time.

I'm wondering if there's a case to be made for allowing different FileSystems in MiSTer. exFAT could remain the default, but giving the option for ext4 would be a good power-user option for people to use more advanced techniques like file links to manage their ever-growing collection. The more cores are added on to MiSTer, the more cramped for space these SD cards will be thanks to the large block size and inability to use file links.

32KB should be ideal for MiSTer needs. Probably sources of installer can be tweaked to set 32KB permanently. I just can't handle everything myself. So if someone will adjust and make PR, then it will be perfect.

MiSTer supports USB drives where you can keep large collection of ROMs and CDs, so you don't have to stick to SD card. It can be like 8GB only to hold just cores and essential files. Then you can plug some tiny USB stick with something like 512GB where you can use different kinds of file systems: FAT32/exFAT/NTFS/ext4. But be careful with ext4 as it's native Linux system and automatically applies restrictions such as case sensitivity, access rights and other redundant features not used by ROM/CD storage.

SD card file system is tightly bound to booting process and requires to be exFAT.

You also have to be careful with NTFS. I recently discovered that NTFS is also case sensitive and I think a lot of people do not know about it as Windows ignores the case sensitivity in order not to disorient Windows users. The way NTFS files systems are mounted under MISTer preserve the case sensitivity. It is possible to tell Linux to be case insensitive with NTFS using the mount option -o ignore_case but it also requires mounting the drive using the type -t lowntfs-3g. I did not try it yet.