MiSTer FPGA Forum

Lightwave wrote: ↑Mon Dec 19, 2022 11:23 pm

XtraSmiley wrote: ↑Mon Dec 19, 2022 11:02 pm

This question is a simple run through of the most modern ROM pack available online being in the 7zip formate, which MiSTer cannot handle (currently).

The simple answer is that 99.9% of people using ROMs don’t use or care about MiSTer. They are using a PC, Raspberry Pi, or some other device for emulation with Retroarch, etc.

Right... the Mister is a teeny-tiny project compared to the big ones. We're fortunate when projects like the MT32-Pi decide to support it. As users, it's mostly up to us to work around issues like this.

I mean, sure, it would be nice to have 7z support built into the framework, but we don't actually need it. Batch conversions from one format to another take quite awhile (especially if you do them on the Mister itself, with its weak and flabby CPU), but they're not actually difficult. It's an inconvenience, but it's far from a showstopper. You can manually convert a few games, and then use a script to batch up the rest, while playing the ones you did first.

That said, I haven't really run into issues with compression formats. The TOSEC libraries seem to mostly (wholly?) use regular ZIP, so those have been good sources for me. The Internet Archive also seems to have lots of Mister-targeted collections, and those are always set up correctly.

7zip support would be nice to the users to some degree, but it would add some cpu overhead since 7zip compression requires more cpu time to decompress. In general sorgelig doesn't want to add support to it because he doesn't see it as necessary. It could also introduce more code maintenance issues long term for him when he's the primary maintainer of all the code.

The implementation cost could be painful, but the ongoing maintenance burden, after the initial bugs were squashed, probably wouldn't be significant. It's not like the archive format changes, so once the code works, it should keep working without much, if any, attention.

But it's still a 'nice to have' feature, and I'm sure Sorgelig has many more of those that he'd rather work on. Converting really isn't that difficult, or just sourcing the ROMs in ZIP format in the first place.

On second thought I'm not sure I even understand the issue. All of Linux, macOS and Windows can decompress 7z files. What's the problem?

thisisamigaspeaking wrote: ↑Thu Dec 22, 2022 5:46 pm

On second thought I'm not sure I even understand the issue. All of Linux, macOS and Windows can decompress 7z files. What's the problem?

Example: when you load a rom in a console the rom cam be uncompressed or compressed. If it is compressed it is uncompressed on the fly by the MISTer software and only zip is supported.

thisisamigaspeaking wrote: ↑Thu Dec 22, 2022 5:46 pm

On second thought I'm not sure I even understand the issue. All of Linux, macOS and Windows can decompress 7z files. What's the problem?

The Mister system can load either uncompressed ROMs or files out of ZIP archives. It doesn't support any other format. If you've got a ROM packed by 7zip, you'll have to unpack it, and then optionally repack it with the less efficient ZIP.

People use 7z format because it packs tighter. But I don't think there's any OS-level support for it. Any program that wants to directly use a 7zip has to either write the code to understand the format, or import it from a library. The Mister is no exception.

Windows understands ZIPs natively, or at least File Explorer does, but I'm pretty sure it's clueless on 7zip. And the Linux kernel and shell programs know nothing about most compression. It's easy to get programs to manipulate pretty much any compression format, but I don't there's any OS-level facilities to help. AFAIK, to the OS, all of them look like random binary data, and are exposed to user programs just like any other file.

The kernel understands how to mount and manipulate many hard drive image formats, like VFAT images or ISO files, but I don't think it has any way to work with them in compressed format.

Probably some of the desktop environments can deal with compressed files like Windows can with Explorer, but the Mister's just running directly on the kernel in text mode.

Ah, I just noticed that on my Mac if I doubleclick on a .7z file in Finder it expands it into a folder without any addon. I guess on Windows I am handling that with WinRAR, I'd forgotten that Explorer doesn't natively (if that's the case). 7z is standard for Linux and 7Zip is FOSS originally. So I was wondering why this is a huge issue for OP, he's on Mac, just expand the archive with Finder/Archive Utility and put it on the MiSTer. Or repack to zip if one wants?

I understand that it would be nice for MiSTer to handle ROMs in .7z format, and I don't see why that would be hard to add given that it's available under LGPL, but what I didn't understand was why OP was so stressed about it. I went to look at a .7z on my Mac and thought "well that was easy." MegaAGS is compressed with .7z and that is intended for MiSTer...

Malor wrote: ↑Wed Dec 21, 2022 2:55 pm

The implementation cost could be painful, but the ongoing maintenance burden, after the initial bugs were squashed, probably wouldn't be significant. It's not like the archive format changes, so once the code works, it should keep working without much, if any, attention.

But it's still a 'nice to have' feature, and I'm sure Sorgelig has many more of those that he'd rather work on. Converting really isn't that difficult, or just sourcing the ROMs in ZIP format in the first place.

Kind of incorrect. The maintenance also includes support. Each new factor you introduce as to why someone may be having a problem can lead to more branches in the tree of trying to figure out what the root cause of someone's issue is. It's not a major issue, but it's just one extra contributing factor that you have to always consider. As a system gets more complex you introduce the potential for more causes behind a bug.

Once the implementation was solid, I can't imagine 7z code generating support problems that wouldn't also be generated by ZIP.

However, as I said, getting to the point that the implementation was solid could be painful. The interim process between 'first version released' and 'last bug squashed' could be extended and un-fun.

Malor wrote: ↑Fri Dec 23, 2022 2:21 pm

Once the implementation was solid, I can't imagine 7z code generating support problems that wouldn't also be generated by ZIP.

However, as I said, getting to the point that the implementation was solid could be painful. The interim process between 'first version released' and 'last bug squashed' could be extended and un-fun.

Since 7z is available in an LGPL library, I have found this exact thing fairly painless to implement (with zip but I assume 7z would be the same). In other words reading compressed files, navigating compressed archives, as if they were standard data files. But that was in C++ with a modular codebase, I agree this could be a bit more complicated in C.

I agree with the fact that it is not necessary.
If you put aside the fact that you have to implement it, debug it and maintain it. I don't think that the extra KB you save using 7z instead of zip is worth the extra time it would take to run a rom inside a 7z instead of a zip file.
Some people even prefer not using zip at all just because of the extra time it takes to run a rom from within a zip file.

Libarchive supports the whole shebang from a single library, but I'm not going to put in the effort. Still fudging around with my vhd generator in my scarce free time.