I Think “/media/fat” Should Be Depricated

[held]

I find "/media/fat" to be a problem when data resides elsewhere. Should we start using a symlink "/media/MiSTer" to point to the actual data folder?

Benefits:

• path independent scripts that work on sdcard and usb
• easily switch from sdcard to usb storage by updating the symlink

Code: Select all

drwxr-xr-x  2 root root     40 Jan  1  1970 cifs
drwxr-xr-x 32 root root 131072 Jan  1  1970 fat
drwxr-xr-x  2 root root     40 Jan  1  1970 rootfs
drwxr-xr-x  2 root root     40 Jan  1  1970 usb0
drwxr-xr-x  2 root root     40 Jan  1  1970 usb1
drwxr-xr-x  2 root root     40 Jan  1  1970 usb2
lrwxrwxrwx  1 root root     40 Jan  1  1970 MiSTer -> fat

Code: Select all

drwxr-xr-x  2 root root     40 Jan  1  1970 cifs
drwxr-xr-x 32 root root     40 Jan  1  1970 fat
drwxr-xr-x  2 root root     40 Jan  1  1970 rootfs
drwxr-xr-x  2 root root 131072 Jan  1  1970 usb0
drwxr-xr-x  2 root root     40 Jan  1  1970 usb1
drwxr-xr-x  2 root root     40 Jan  1  1970 usb2
lrwxrwxrwx  1 root root     40 Jan  1  1970 MiSTer -> usb0

[jca]

A long time ago Sorge mentioned that "Switch to USB" was kind of obsolete and one should use symlinks. This is what I did and made many posts about it.
You can place about any folder on USB and use a symlink in media/fat.
This is what I did: I have everything on USB except config and Scripts.
A little annoyance if you place _Arcade, _Computer, .. on USB and the USB drive is not yet mounted when MISTer displays the menu you will get No Files!
Wait a little, press twice F12 to flip the OSD back and forth et voila.

[FPGA64]

You dont need to symlink anything. The USB external drive overlays the SD card. If a directory is on the external drive it replaces whatever is on the SD. if the directory is not present on the external drive then the SD directory is used. Its all automatic

The switch to USB option should not be touched

[jca]

I did what Sorge said to do at the time and it works for me.
Good to know that you don't need the symlinks, I was under the impression that this automatic switch was only for the games directory.
It is hard to know what is going on on the Linux side of MISTer due to the paucity of information.
I have a few questions:
Are the directories located on USB mounted in /media/fat?
What about the update script, do you leave the default setting to the SD card? On my system with symlinks I left it to the SD card.

[FPGA64]

The external I only store Games om. There was no point moving the cores from the SD as they are small

The /media/usb0 is the external drive and on there I have a games directory exactly as it was on the SD

The Update script happily copies files to the external drive

[jca]

The external I only store Games om. There was no point moving the cores from the SD as they are small
...

They are small but are updated frequently -> SD card wear.
Do you still have an emptys games on the SD?
When I am close to my MISTer I will make some test.

[FPGA64]

No I have a full games directory on the SD card so it can be moved around without the external HD, well full bar the PSX. The external has it all and as such no SD card directory is observable in the cores

[held]

It still leaves the scripting argument.
And how do we measure sd-card writes?

[jca]

It still leaves the scripting argument.
And how do we measure sd-card writes?

If scripting refers to the location of the Scripts folder the problem is due to the fact that the USB drive is mounted with the NOEXEC option. If you want to run a script via SSH and the script is on USB you have to use "bash <scriptname>". I am not sure how a script is run if you run it via the Menu.

[riblad_om]

I think you guys may be a bit paranoid about sd card wear. I have a Micro Center card I've been running update all and mame getter on weekly for years with no issue. I'm sure it'll wear out eventually but then I'll just buy a new one.

[held]

Maybe its just me but I'd rather have my roms on a SSD.

@jca
Yeah and no, so I'll gather my thoughts and do some more investigation before I say something stupid

[Newsdee]

Just backup your card once in a while

Wish the google drive scripts worked again, they made it very easy to save amd restore save data.

[theypsilon]

It still leaves the scripting argument.

USB external drive also overlays the SD card with the updaters, the same as in the firmware.

[riblad_om]

Maybe its just me but I'd rather have my roms on a SSD.

I don't keep my non-arcade roms on the sd card, if that's what you're implying. Those go on the usb drive.

It's really the path of least resistance. Let update_all manage the system files and arcade roms on the sd card. Plug in a usb drive with a games directory for console roms and those will automatically be mounted at boot.

[jca]

I think you guys may be a bit paranoid about sd card wear. I have a Micro Center card I've been running update all and mame getter on weekly for years with no issue. I'm sure it'll wear out eventually but then I'll just buy a new one.

It is not a question of being paranoid on my side: I was using "Switch to USB" until Sorge said it was better to use symlinks which is what I did. Why not do it for almost every directory. It also has the advantage of making things faster if I have to redo my SD card. But it is just me.

[jca]

So I made a test to see if this automatic switch worked also for directories other than the games directory: it does not.
I removed the symlink SD -> USB for _Arcade and rebooted: no more Arcades.
I created an empty _Arcade on the SD card and rebooted: I have an empty _Arcade directory.

[riblad_om]

The Arcades collection is so small that there is no reason not to keep it on the sd card.

[jca]

@theypsilon Thanks for your post. It made me think and leads me to a question:
When I placed everything on USB I created symlinks SD -> USB, includings games. I do not have a games directory on the SD card, just a symlink. After that In had some problems with Mame Getter and HB mame Getter as they did not find any mra. To fix the problem I created ini files for the getter to point the Arcade directory and the ROM directories to USB.
Would getting rid of the games symlink and creating an empty games directory on the SD card allow me to also get rid of these ini files?

[jca]

The Arcades collection is so small that there is no reason not to keep it on the sd card.

1. We live in a free world or may be not but we can pretend we do.
2. Arcades was just for a test after FPGA64 sowed doubt in my mind. Shame on you FPGA64.
As I said earlier I have almost everything on the USB drive. This was after using "Switch to USB" and switching to symlinks. I also like experimenting.
Here is the contents of my SD card.

[theypsilon]

@jca, if I were you, I'd get rid of that symlink setup. All folders should be on the SD card, except games and docs that can stay on your external drive if you want. Once you have that, the default downloader.ini options will install "games" files on your external drive for each system that already has its folders there (same as what the firmware does to load the games).

[Malor]

You dont need to symlink anything. The USB external drive overlays the SD card. If a directory is on the external drive it replaces whatever is on the SD. if the directory is not present on the external drive then the SD directory is used. Its all automatic

The switch to USB option should not be touched

In my experience, that's what the docs say should happen, but it doesn't actually work, at least not with /media/fat/cifs. Update_all, for instance, was particularly annoying about insisting that the AO486 BIOS files always ended up in /media/fat/games/AO486. There seemed to be numerous papercuts of that nature.

Symlinks, on the other hand, instantly fixed the problem. Smart programs go into /media/fat/cifs/whatever, and stupid programs are forced into /media/fat/cifs via symlinks.

@jca, if I were you, I'd get rid of that symlink setup. All folders should be on the SD card, except games and docs that can stay on your external drive if you want. Once you have that, the default downloader.ini options will install "games" files on your external drive for each system that already has its folders there (same as what the firmware does to load the games).

My ZFS NAS is enormously more robust than a simple microSD card, and offers timed snapshots, so I can easily restore older versions of save files or savestates. (This has saved my butt at least once.) It's also a lot faster; the gigabit Ethernet port seems to actually have a dedicated gigabit, where the USB devices all have to share 480Mbps between them. Putting as many things as possible on the NAS makes things snappier now, and also makes it very easy to rebuild the card if it ever fails. I just have to connect the /media/fat/cifs share and remake the symlinks.

I should probably write a script to do that, but haven't bothered.

edit: well, no, I'd have to do some mucking around in /etc/, too. A card rebuild would still be pretty painful. But it wouldn't take as much downloading, at least.

[FPGA64]

I would not use switch to USB option. Its simply not needed for an External drive. Its much simpler to keep all the cores/ saves/ scripts and executables on the SD card and use the external HD just for the Games. I find it best to format the external drive as exFat32 to avoid having to deal with case sensitivity of filenames.

Some of the automatic updates such as A0486 may not work, but its simple to run it and then copy the new file to the external drive

[held]

It still leaves the scripting argument.

USB external drive also overlays the SD card with the updaters, the same as in the firmware.

Does it use bind mount ? because that is a far better option than I initially suggested, and would negate my deprecation request/suggestion entirely.

[Malor]

I would not use switch to USB option. Its simply not needed for an External drive. Its much simpler to keep all the cores/ saves/ scripts and executables on the SD card and use the external HD just for the Games. I find it best to format the external drive as exFat32 to avoid having to deal with case sensitivity of filenames.

Some of the automatic updates such as A0486 may not work, but its simple to run it and then copy the new file to the external drive

IMO, this is bad advice. Hard drives are typically much more reliable than SD cards. They also will retain data for very long periods while unplugged, where both SDs and SSDs will often start losing data after about a year without power.

[cursedverses]

I would not use switch to USB option. Its simply not needed for an External drive. Its much simpler to keep all the cores/ saves/ scripts and executables on the SD card and use the external HD just for the Games. I find it best to format the external drive as exFat32 to avoid having to deal with case sensitivity of filenames.

Some of the automatic updates such as A0486 may not work, but its simple to run it and then copy the new file to the external drive

IMO, this is bad advice. Hard drives are typically much more reliable than SD cards. They also will retain data for very long periods while unplugged, where both SDs and SSDs will often start losing data after about a year without power.

That's the first I've ever heard of this. Do you have anything to back up that claim?!

[Malor]

Yeah, they absolutely do; they need to stay connected to power so that the card controller can, over time, move the data around. The storage cells work by being charged up, and the charge leaks out again. If the card isn't powered, this can result in data loss.

It has a lot to do with how hot the card was when the data was written. Weirdly, the hotter the better. You want the chipset cool so it runs well, but you want the memory banks hot while being written, but then cold while being stored.

This link has some data:

https://www.partitionwizard.com/clone-d ... orage.html

Search for "Is SSD Good for Long-Term Storage?" The table shows that if the SSD is stored, without power, in 55C heat, then data can be lost in just a week if it was written at 25C, and can be lost in 8 weeks if it was written at 55C.

On the other hand, the best case in that chart is writing at 55C and storing at 25C, where data can last over 400 weeks.

This link has more graphs, from a different vendor, but showing roughly comparable figures:

https://www.anandtech.com/show/9248/the ... -retention

Regardless, the underlying lesson is this: cards and SSDs that are not connected to power degrade. They can degrade very quickly in hot temperatures. Their retention will be fine as long as they have power, as the controller can detect the weakening charge and cycle the data to different cells, starting the leak time over again.

You should expect SD cards to be markedly worse than this; SSDs are built to a much higher quality standard.

Hard drives, on the other hand, deteriorate extremely slowly. The failure point there is that the drive might not spin up after being stored a long time, as opposed to the magnetic media losing its charge. You can stick one on the shelf and usually leave it there a decade. This, IMO, is a *much* better place to put your save games.

[cursedverses]

Wow! That's some really useful and interesting content!

I guess if we're turning our MiSTers at least every couple of weeks or so we wouldn't see data loss, right? I guess from a data view, you'd want to keep games and saves on your external drive along with a copy of your mister.ini/downloader.ini. Cores can be redownloaded and the same for Scripts.

Using the USB drive for everything doesn't seem to need to be the case from my perspective.

[Malor]

Yeah, if you're turning on your Mister every few weeks, that should be fine, as long as it's on for a few hours to give the controller time to refresh anything that needs it. And there's no critical need to store cores on the USB drive, unless you think the project might get nailed by someone claiming copyright infringement. It might save you a bunch of downloading if your card dies... but you might be trading away time fiddling with creating symlinks before running update_all.

Storing ROMs and saves on spinning media seems like an extremely good idea to me.

FWIW, so far I've left the actual cores and the config directory on the SD, but put everything else under /media/fat/cifs. I thought I'd moved the cores, but I looked last night, and haven't done that yet. As you say, that can be re-downloaded, and I don't have a bandwidth cap, so maybe I won't bother.

You probably want the config dir on the same filesystem with the Mister, because it needs to be available when the menu launches. On a cold boot, network shares don't come up until a few seconds after the menu starts, which is too late, so it drops back to default settings. (but then remembers its settings after a warm boot, which is confusing if you don't understand what's going on.) I'm not sure if USB filesystems would come up fast enough or not.... I'd err in the direction of just keeping the config dir on the same filesystem as the main MiSTer binary.

Taking occasional backups of config probably wouldn't hurt. Each file usually doesn't take that long to recreate, but there can be a lot of them.

[theypsilon]

My ZFS NAS is enormously more robust than a simple microSD card, and offers timed snapshots, so I can easily restore older versions of save files or savestates. (This has saved my butt at least once.) It's also a lot faster; the gigabit Ethernet port seems to actually have a dedicated gigabit, where the USB devices all have to share 480Mbps between them. Putting as many things as possible on the NAS makes things snappier now, and also makes it very easy to rebuild the card if it ever fails. I just have to connect the /media/fat/cifs share and remake the symlinks.

I should probably write a script to do that, but haven't bothered.

edit: well, no, I'd have to do some mucking around in /etc/, too. A card rebuild would still be pretty painful. But it wouldn't take as much downloading, at least.

Until your network is down for any random reason. What you mention is a good option for power users that know what they are doing, know well the drawbacks, and can troubleshoot everything in case they face some problems. But it's not a setup that I would recommend to 99% of the users, who only want to install everything and play. If you opt into that setup, you are on your own.

Btw, update_all and downloader work as described before (games folder overlays) since May/June as long as you have the default options.

[Malor]

This isn't your problem, but it seems to me that making the use of a NAS easy would be a lot more user friendly than leaving people out to hang with possibly bad SD cards. There are a lot of counterfeits out there, and even good SDs aren't that good.

If update_all is now properly using multiple targets correctly, that's a great step in the right direction, but when first transitioning to the NAS, I was having lots of issues with various different parts of Mister. Symlinks made all that pain go away.