MiSTer FPGA Forum

Hello nerds,

I wrote myself a little tool for finding and downloading individual game ROMs on the MiSTer. This is mainly intended for CD based systems, when having local copies of full libraries isn't sensible. Or when my local library just happens to be missing something I very urgently must play.

I'd appreciate any feedback from users, coders etc. if you think it'd be worthwhile developing it further or how. Download from MiSTer shell:

(Note: Project moved to Github)
https://github.com/Koston-0xDEADBEEF/MiSTer-ROMweasel

cd /media/fat/Scripts && curl -kLO https://github.com/Koston-0xDEADBEEF/MiSTer-ROMweasel/raw/main/romweasel.sh

There's no documentation yet, and hopefully never needs to be. This is intended to be simple enough to use, to not require a manual.

There are few instructions now on Github, based on feedback from this thread.

netbeui wrote: ↑Thu Nov 03, 2022 9:40 pm Is this supposed to go in the scripts folder and then you run it from within MiSTer? Or just download the file in a shell and run it from there?

LamerDeluxe wrote: ↑Fri Nov 04, 2022 4:32 pm So far it gives me 0B size for the TGFX16 CD and PSX games I selected and then it doesn't download anything.

Koston wrote: ↑Fri Nov 04, 2022 9:21 pm

LamerDeluxe wrote: ↑Fri Nov 04, 2022 4:32 pm So far it gives me 0B size for the TGFX16 CD and PSX games I selected and then it doesn't download anything.

Did you actually select anything before trying to download? You need to select items with spacebar.

If that is the issue, it's an oversight on my behalf - I realised too late the script cannot be used at all without a keyboard. I intend to investigate how to make it usable with just a controller.

Koston wrote: ↑Fri Nov 04, 2022 9:21 pm

LamerDeluxe wrote: ↑Fri Nov 04, 2022 4:32 pm So far it gives me 0B size for the TGFX16 CD and PSX games I selected and then it doesn't download anything.

Did you actually select anything before trying to download? You need to select items with spacebar.

If that is the issue, it's an oversight on my behalf - I realised too late the script cannot be used at all without a keyboard. I intend to investigate how to make it usable with just a controller.

Neocaron wrote: ↑Sat Nov 05, 2022 12:47 am I'd be happy with a "select all" as well. It might come in handy.

Koston wrote: ↑Sat Nov 05, 2022 1:35 am

Neocaron wrote: ↑Sat Nov 05, 2022 12:47 am I'd be happy with a "select all" as well. It might come in handy.

This feature is currently absent on purpose.

I'm considering adding a separate code path for downloading full libraries, but only if the total library size is reasonable; I don't want to encourage wasting archive.org's bandwidth. They're struggling as it is, and their work preserving digital history is very important.

Koston wrote: ↑Sat Nov 05, 2022 1:35 am I'm considering adding a separate code path for downloading full libraries, but only if the total library size is reasonable; I don't want to encourage wasting archive.org's bandwidth. They're struggling as it is, and their work preserving digital history is very important.

capitaineflam25 wrote: ↑Sat Nov 05, 2022 7:30 am What is the recommanded way to configure downloads to /media/usb0/games/ instead of /media/fat/games/

Koston wrote: ↑Sat Nov 05, 2022 10:44 am The script sources /media/fat/Scripts/.config/romweasel/settings.sh for user settings, or if the file doesn't exist, creates it.

georgemog wrote: ↑Sat Nov 05, 2022 12:03 pm I did find one issue with downloading PSX files and that is that they require a unique subdirectory for each game. I have added the following lines in bold to the download_roms procedure.

georgemog wrote: ↑Sat Nov 05, 2022 12:03 pm I did find one issue with downloading PSX files and that is that they require a unique subdirectory for each game.

Games that are in their own folder will create it's own memory card in media/fat/saves/psx as .sav

Koston wrote: ↑Sat Nov 05, 2022 2:26 pm I've been cursing the cumbersome handling of save files with the PSX core, but this should get rid of the problem entirely.

I will modify your patch a bit and push an update to Github.

Place CD images into game-specific subdirectories

CD based systems benefit from having each game in their own directory,
especially so with multi-CD games.

Implemented heuristics for determining a suitable unique path for
each game. There are some corner cases when it doesn't work, in such
case the files are simply placed to the ROM root and left to the user
to relocate as they see fit.

Also fixed 4-byte integer overflow in total ROM size calculations.

• Enable controller-only operation DONE
• Full library download function (for reasonably sized libraries only)
• Keep session state in disk cache for automatically continuing unfinished downloads (currently user needs to manually re-select same games to continue download from where it left off)
• Re-using CHD placement logic for reorganizing already existing local files DONE

Koston wrote: ↑Sun Nov 06, 2022 9:41 am Commit 076fd2b:

Place CD images into game-specific subdirectories

CD based systems benefit from having each game in their own directory,
especially so with multi-CD games.

Implemented heuristics for determining a suitable unique path for
each game. There are some corner cases when it doesn't work, in such
case the files are simply placed to the ROM root and left to the user
to relocate as they see fit.

Also fixed 4-byte integer overflow in total ROM size calculations.

This change turned out to be a lot more complex than anticipated.

import os
import sys
import re

MULTI_DISK_PATTERN = r'\(Disc \d+\)'

def gamedirname(filename):
    multidisk = re.search(MULTI_DISK_PATTERN, filename)
    if multidisk:
        return(filename[0:multidisk.start()].strip())
    else:
        return(filename[0:filename.rfind(".")].strip())

if len(sys.argv) != 2:
    sys.exit("Usage: chd2folders <CHDs dir>")

for file in os.listdir(sys.argv[1]):
    if file.lower().endswith(".chd"):
        file = os.path.join(sys.argv[1], file)
        dir = gamedirname(file)
        if not os.path.exists(dir):
            os.mkdir(dir)
        os.rename(file, os.path.join(dir, os.path.basename(file)))

LamerDeluxe wrote: ↑Sun Nov 06, 2022 11:52 am

Koston wrote: ↑Sun Nov 06, 2022 9:41 am Commit 076fd2b

This change turned out to be a lot more complex than anticipated.

Don't know if this helps as an example, but this is a (Python) script made by user @bootsector on the MiSTer Discord. I used this to correctly organize my chd files for multiple cores.

% for f in *.chd; do d=${${(Q)f%.chd}// \(Disc [0-9]\)*/} ; [[ -d $d ]] || mkdir "$d" ; mv "$f" "$d" ; done

LamerDeluxe wrote: ↑Tue Nov 08, 2022 1:23 pm I've been testing the chd code and simple mode a bit and so far it has been working fine. This is a really handy script, thanks for all the effort you've put into it.

Koston wrote: ↑Tue Nov 08, 2022 1:55 pm

LamerDeluxe wrote: ↑Tue Nov 08, 2022 1:23 pm I've been testing the chd code and simple mode a bit and so far it has been working fine. This is a really handy script, thanks for all the effort you've put into it.

I'm happy to hear you're finding it useful.

I tested the chd sorting code against full PSX and MegaCD libraries, and there were only a handful that had so complicated or irregular file names that the logic can't handle. Exact figures are in git commit log.

(If) Going forward, I should really switch to another language and investigate using frameworks by other people. The game repositories have SQLite files, parsing those with access to more advanced data structures would be the smart thing to build on.

LamerDeluxe wrote: ↑Tue Nov 08, 2022 3:30 pm Would be great if it could be extended to include support for more cores in the future.

If the controller option could be expanded with a buttons for select and page up/down, that would also be really handy.