request: squashfs in the linux kernel

[Johnny Love]

Just an example, the 2019-03-04 Neo Geo Roll-UP BETA 3.1 7zip file is 2.5GB

MiSTer can't read it, so you have to extract it, then it ends up being 7.3GB

After a few minutes of compression with "gensquashfs -D . -c xz -b 1048576 ../squash.neogeo.img" it goes from being 1447 uncompressed files, down to a single file that is only 1.8GB, smaller than the original 7zip file.

Unlike zipped roms, this would require no changes to cores or MiSTer itself. There are no downsides when it comes to rom files other than slightly slower rom loading since the ARM core would have to do the work, transparently behind the scenes. It might be an issue with ISO images, and of course would not work with hard drive disk images that need the ability to write back to them unless copy-on-write was used to write the changes to another file. For any kind of ROM based system like Neo Geo or below, this is the only way I would do it if I had the option.

[Johnny Love]

If anyone is interested, I have compiled a kernel with squashfs support. See https://github.com/MiSTerJohnnyLove/MiSTerSquashfs and https://redd.it/ibl8j3

[ZigZag]

If anyone is interested, I have compiled a kernel with squashfs support. See https://github.com/MiSTerJohnnyLove/MiSTerSquashfs and https://redd.it/ibl8j3

Thanks, I'll have to test it sometime.

[retrorepair]

Very interesting, especially as it is optional and only runs on the linux side.

How easy is it to compress/decompress these files and how does the menu handle them? Like folders? I imagine it would have to if you have the entire Neo Geo set in one file.

[Johnny Love]

It takes quite a lot of CPU resources to compress them, but I only plan on doing it once a year or so. Probably RAM too, but I didn't check. So far I haven't really noticed much of a change at all, and I have most cores working now, excluding the cdrom based ones. Decompression is always easier than the initial compression.

It is literally a filesystem, so it looks the same from within the menu. The squashfs images get mounted out of the way, and then whatever is in them gets symlinked into the Games/CORE/ folder. It is best for the only thing to appear in the root of the squashfs image to be the folder/directory for a rom pack so only one symlink has to be made. it Doesn't matter, there can be more than one thing and they will all get symlinked in. You can also have more than one squashfs per core.

[Sorgelig]

Looks like linux style: do something a whole day digging in config files.
While MiSTer becomes more and more complicated i still try to keep it in plug-and-play form.
Mounting additional images is not user friendly. Pressing on zip to open it as a folder is more user friendly even though it can be a little bigger.
Your original problem with oversized zip can be solved by splitting to smaller zips like A.zip B.zip .. Z.zip.

[Johnny Love]

I think you may be misinterpreting what my goal is here. Squashfs is part of the linux kernel. All I had to do was grab the MiSTer kernel source code from github, go into the filesystem settings of the kernel, and turn on squashfs support before compiling it. I enabled all of the compression options so people can experiment, but xz seems to be the best so far.

There's no digging through config files all day.You simply take a squashfs image file (which is about as easy as making a zip file at a command prompt, really incredibly easy) and upload it to /media/fat/squashfs/CORE/ and the next time you reboot, you now have access to all of those files that were compressed with XZ compression, not zip files which are hardly compressed at all.

It might not seem user friendly to "mount additional images" and I get that, that's why I made some bash scripts that do it for you. It doesn't even have to be run manually once it is sitting in /etc/init.d, it just runs at boot.

In any case I don't mind compiling a kernel for this purpose, it just seems easy enough to add to the default kernel for those that want it. It saves a lot of sd card space and if you don't know its there, it can't do anything to make your experience worse. It also isn't proprietary or hard to access, the windows executable to make these images works the same as the linux one.

[KremlingKuthroat19]

I'm no expert whatsoever on the topic of compression and decompression. I have experience using CHD and love using it in Retroarch because it not only shrinks the file sizes drastically, but it conveniently makes each bin/cue into a single file with a CHD extension. It's very clean and snappy. I'm wondering if you created a CHD and zipped it if it would make the file even smaller? (obviously at the cost of booting time). I understand that CHD might be too obscure of a compression method to implement, but I've used it and can vouch for it.

I digress. The point I'm making is that I've never heard of squashfs until reading this thread, but if it's a good compression method then I'm on board. However, there comes a certain point in time that I believe there should be a compression method that's simple and compatible with all bin/cues and ideally works with all other rom extensions for the MiSTer project. I say this because MiSTer currently supports Sega CD and PC Engine CD libraries which each have a few hundred games. However, once the PS1 core comes out (who knows when exactly honestly), then we may have a problem that needs to be addressed. The PS1 has 2500+ games. That library is absolutely massive and each game is around 700 MB.

TLDR: If there was a universal compression method that MiSTer adopted, it could futureproof this issue by the time the PS1 core becomes available. Obviously I don't want to undermine the work requied to make this feature a reality, but just wanted to give my two cents.

[tankjr]

Great work! Your update and mount scripts are finicky and didn't want to play nice with my system.

I compressed all the data on my main Linux machine and transferred it onto my SD card. I managed to get everything working by reading your scripts and doing all the heavy lifting myself over an ssh terminal. I've easily freed up 20gb of space on my 128gb card and my load times have only increased slightly. It's not like I'm swapping games every minute... so the change in load time doesn't bother me. Big thumbs up overall.

I think there are ways to make it more user friendly. As is, it works great, but it's definitely tailored for someone who knows there way around a Linux machine.

[jrronimo]

Great work! Your update and mount scripts are finicky and didn't want to play nice with my system.

I compressed all the data on my main Linux machine and transferred it onto my SD card. I managed to get everything working by reading your scripts and doing all the heavy lifting myself over an ssh terminal. I've easily freed up 20gb of space on my 128gb card and my load times have only increased slightly. It's not like I'm swapping games every minute... so the change in load time doesn't bother me. Big thumbs up overall.

I think there are ways to make it more user friendly. As is, it works great, but it's definitely tailored for someone who knows there way around a Linux machine.

Is this with CHD or with SquashFS? Either way, it sounds like a win to me!

[tankjr]

Is this with CHD or with SquashFS? Either way, it sounds like a win to me!

Sorry for the late reply. This was with SquashFS.

[zakk4223]

You can compile squashfs and a few dependencies as kernel modules and load them at runtime on the mister. You still have to worry about kernel version matching, so it's not 100% foolproof. Still, it might be easier to distribute a few kernel modules via a script than it is to replace the whole kernel.

It also means the users are still running a stock kernel

[Johnny Love]

I'm no expert whatsoever on the topic of compression and decompression. I have experience using CHD and love using it in Retroarch because it not only shrinks the file sizes drastically, but it conveniently makes each bin/cue into a single file with a CHD extension. It's very clean and snappy. I'm wondering if you created a CHD and zipped it if it would make the file even smaller? (obviously at the cost of booting time). I understand that CHD might be too obscure of a compression method to implement, but I've used it and can vouch for it.

I don't use any of the CD cores yet, and am unfamiliar with the CHD format. It sounds like it would be better than squashfs, and that trying to use squashfs to further compress the files would end up not saving any meaningful amount of space, while adding to the CPU load. It is very uncommon for already compressed files to compress more, even if the second format is better, it can sometimes even increase the file size. So far I'm not aware of anyone doing tests with squashfs with CD images, but I would be very interested in any success or failure with it. I would recommend starting with uncompressed images that you know to work, and then compressing them with squashfs.

Great work! Your update and mount scripts are finicky and didn't want to play nice with my system.

Thank you, I know there were several bugs, and I thought I got them all, but apparently at least one slipped by while i was running it multiple times trying to get them. Knowing there is at least one left is very helpful. I'll try to get that fixed.

You can compile squashfs and a few dependencies as kernel modules and load them at runtime on the mister. You still have to worry about kernel version matching, so it's not 100% foolproof. Still, it might be easier to distribute a few kernel modules via a script than it is to replace the whole kernel.

It also means the users are still running a stock kernel

That was something I looked into, I was originally going to do it that way, but I think I noticed that the kernel doesn't have loadable module support. As it turns out, it is easier for me to just to replace the kernel, there is only one file this way, and it really is the stock kernel because I grabbed it from the official MiSTer github. That is why I wanted it added to the kernel officially though, either built in or as a module, there would be no script needed to screw around with the kernel at all.

[zakk4223]

The kernel does have loadable module support; I successfully built and loaded squashfs+dependencies on a stock mister.

I think the main advantage of the module approach is that it is nearly impossible (via script error or user error) to end up with an unbootable mister. I mean, it's unlikely either way, but I'm always amazed at what manages to happen 'in the wild'.

regarding CHD: chd uses (I think) lzma for data and FLAC for audio. If zipping isn't recommended for ISOs due to access speed issues, I'm not sure CHD is going to be any better.

[rhester72]

regarding CHD: chd uses (I think) lzma for data and FLAC for audio. If zipping isn't recommended for ISOs due to access speed issues, I'm not sure CHD is going to be any better.

ZIP isn't recommended because it's linear compression - CHD is block-level compression, designed from the ground up for random (and compressed) access.

[Johnny Love]

That is good to know about the modules. It has been a few months so i wasn't sure. Now I'm tempted to just compile all of the modules and upload them along with the squashfs ones, that way people can use different linux features that aren't available in a default install.

I would think FLAC would be much easier to decompress, especially since these CD games load up the data for a level and then stream CDDA from the CD throughout the level just like it was in an audio CD player. FLAC is optimized for this, although opus seems like it would be a better choice.

[KremlingKuthroat19]

Glad to see this conversation is still ongoing. I'm up for any compression method for CD games, as long as it works across all cores and is an easy command line type of conversion so I can convert all my CD games in one or two gos.

[zakk4223]

Over the last few days I've managed to integrate libchdr into the main mister code. I can now load PCE-CD and Mega-CD CHD images.

I still have some cleanup and a lot more testing to do, but so far everything I've tried works.

As far as performance: CHD stores data in 'hunks', for a CD image those hunks container 8 sectors. For CDDA hunks it takes about 1.5ms to read the hunk. So basically every 8 sector reads there's a 1.5ms read penalty, which doesn't seem to be causing any odd audio issues.

The space savings is pretty good, for my MegaCD collection it went from around 145G bin/cue to 75G CHD

Code is up on this branch: https://github.com/zakk4223/Main_MiSTer/tree/isochd

[elvis]

On the subject of realtime compression, I've been running Linux file servers with BtrFS+zstd block level compression, or embedded devices like Raspberry Pis as well as old laptops with compact flash replaced hard disks with F2FS+lzo block level compression for some time now.

BtrFS benefits twice over, because I can also deduplicate large disk/ROM collections and compress the resulting blocks transparently to save considerable space (several terabytes across my collections) without any manual effort needed at hand loading time (no need to manually compress ROMs or disk images).

F2FS also benefits SD style storage, as it dramatically increases the life span of regularly written filesystems (anecdotal evidence suggests a 25% improvement in storage life span).

Both of these file systems would be a welcome addition to the standard kernel supplied with MiSTer. I've compiled them in manually a couple of times, but resulting updates mean I have to do it over. So I've since resorted to network mounting my files instead.

------------

Examples of BtrFS in action.

I created a volume of type BtrFS, data=single, metadata=single. Mounted with "compress-force=zstd". I copied a USA SNES set of ROMs to the file system (uncompressed ROM files).

Raw logical data = 1026 MB

Code: Select all

root@server:~# du -smx /mnt/1
1026    /mnt/1

On-disk, BtrFS realtime compresses this down to around 661MB transparently:

Code: Select all

root@server:/mnt/1# compsize .
Processed 767 files, 8129 regular extents (8129 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       64%      661M         1.0G         1.0G       
none       100%       15M          15M          15M       
zstd        63%      646M        1010M        1010M   

root@server:/mnt/1# df -hT ./
Filesystem                Type   Size  Used Avail Use% Mounted on
/dev/mapper/vg0-btrfstest btrfs  5.0G  668M  4.4G  14% /mnt/1

From here I run an offline deduplication. This scans for identical blocks of information, and uses BtrFS's "reflink" to deduplicate identical blocks. I run the command:

Code: Select all

duperemove -b 8K -rdhv --hashfile=/dedup/tmp.db /mnt/1

To scan and deduplicate any block of data at 8KB in size. The end result according to the log file:

Code: Select all

Comparison of extent info shows a net change in shared extents of: 71.8M

Using BtrFS's reporting tools:

Code: Select all

root@server:~# btrfs fi du -s /mnt/1
     Total   Exclusive  Set shared  Filename
   1.00GiB   932.39MiB    40.76MiB  /mnt/1
 

They say there's around 40MB of "set shared data" (i.e.: deduplicated data), but worth remembering that this can be shared multiple times, saving more than reported there (again, the log file suggested 71MB).

And a final df which includes all metadata, file system overhead, etc:

Code: Select all

root@server:~# df -hT /mnt/1/
Filesystem                Type   Size  Used Avail Use% Mounted on
/dev/mapper/vg0-btrfstest btrfs  5.0G  649M  4.4G  13% /mnt/1

That puts us somewhere in the order of 64% or so of our original 1026MB, saving 36%. Almost as good as zipping (I tested with "zip -9r", and the resulting file was 628M - almost identical in size as the realtime zstd compression+dedup), but it's all transparent (i.e.: no zip/unzip process to read the raw data). More importantly, this can apply to ANY file on disk - VHD images for your computer cores, CD images for your optical based consoles and ao486 cores, etc, etc. Again, these latter file types can't be stored inside .zip files and accessed by the cores, but on BtrFS with zstd compression and deduplication, they don't know the difference.

I run my entire NAS this way, as well as several Raspberry Pi systems, and it saves me real world somewhere in the order of 30% across a wide range of data (a bit difficult to calculate exactly as I also have snapshots which make measuring deduplication space saved difficult).

[rgn]

Just grab a larger SD card! And btw, I am absolutely sure you won't play ALL the NeoGeo games, so why don't you just copy the only games that you desire?..

[elvis]

why don't you just copy the only games that you desire?..

MiSTer is a literal "museum in a box". I consider myself pretty well versed in old video games (because I'm old and lived through these titles in their heyday, and I research them today, and help other cultural historians and researchers do the same), and I'm still discovering new titles every month.

One of the most convenient features of the device is having complete collections at your disposal. When you read about games you've not heard of, or you're searching for titles you've not seen before, not having to manually curate things is precisely why you have these complete collections.

Compression helps three fold, not just in saving space, but also in loading ROMs faster (CPU decompression speeds are faster than SD card read speeds), and in device longevity (fewer IOs and advanced flash-friendly file systems mean SD cards last longer). Plenty of worthwhile reasons to bother with better file system options.

[redsteakraw]

Just a reminder there are windows drivers for some linux file systems, BtrFS being the latest. Given MiSTer supports FTP and SMB network filesystems, maybe a pure BtrFS / squashFS could make for a better default. Windows users just need to install the drivers so they aren't locked out of accessing the SD card directly.

Here is the BtrFS for windows project as being used by the Russian funded ReactOS project
https://github.com/maharmstone/btrfs

Here is the EXT filesystem drivers for windows
https://sourceforge.net/projects/ext2fsd/

Windows based file systems are extremely limiting and lacking ExFat doesn't support symlinks nor does it take advantage of any real modern feature sets. It is a better version of Fat but that really isn't saying much.