Reducing the space taken by the HTGDB Gamepacks with symlinks

[toanazma]

Not sure if this is the right place to put it but I figured it might be helpful to someone else.

I love those gamepacks and especially love having the Game Series Collections add on which makes browsing and discovering new games great. However, I don't have much space in my sd card and deduplicating roms was not helping. Instead I figured a slightly convoluted process to optimize space.

I use two tools for this rdfind and symlinks. Both are easy to install in any linux distribution, and with either homebrew or macports on mac.

I download both the normal gamepack and the Game Series Collection add on. I extract both in the same directory. Which gives something like this:

Code: Select all

d--------- 1 root root  15K Apr 28  2021 1 US - A-F
d--------- 1 root root  16K Apr 28  2021 1 US - G-P
d--------- 1 root root  17K Apr 28  2021 1 US - Q-Z
d--------- 1 root root   72 Apr 28  2021 2 Europe - A-Z
d--------- 1 root root   44 Apr 28  2021 2 Japan - A-Z
d--------- 1 root root  118 Apr 28  2021 2 Other Regions - A-Z
d--------- 1 root root  19K Apr 28  2021 2 Unlicensed - A-Z
d--------- 1 root root  154 Apr 28  2021 3 Sega 32X
d--------- 1 root root   62 Apr 28  2021 4 Betas, Prototypes, Revisions
drwxr-xr-x 1 root root  208 Apr 28  2021 4 Game Series Collections
d--------- 1 root root  138 Apr 28  2021 4 Hacks
d--------- 1 root root 9.1K Dec 14 11:08 4 Homebrew
d--------- 1 root root 5.1K Dec 14 11:08 4 Translations
d--------- 1 root root  16K Dec 14 11:08 4 VGM Music Players
d--------- 1 root root 2.8K Dec 14 11:08 5 Tools & Service Test Carts
d--------- 1 root root  968 Apr 28  2021 BIOS
d--------- 1 root root 1.9K Dec 14 11:08 SRAM & TMSS Patched for X3
d--------- 1 root root 2.0K Dec 14 11:08 Unpatched EEPROM-Save Games for X7

Then I run

Code: Select all

rdfind -makesymlinks true *

in that directory
This removes all duplicates and creates absolute symlinks instead.

Since absolute symlinks are impractical if the path change, I convert them to relative symlinks with

Code: Select all

symlinks -cr .

The symlinks are then converted to relative paths. This makes it now easy to copy over to the mister.

To copy to the mister, we can't just copy through samba/cifs unfortunately since symlinks are not supported on network shares.
So the way to preserve the symlinks is to tar the directory

Code: Select all

tar cvf Genesis.tar Genesis

copy that tar file to the mister and then untar it with

Code: Select all

nohup tar xvhf Genesis.tar Genesis

in /media/fat/Games
Do not the h option which tells tar to preserve the symlinks on untarring

I use nohup in case the ssh connection dies while it's decompressing (it takes quite a while). If your connection dies, you can monitor the progress by going to /media/fat/Games and running this

Code: Select all

tail -f nohup.out

This saves 40-60% of the space for the game packs with Game collections. For example with GBA, it went from 52GB to 32GB so saving 20GB.

[kerobaros]

Beautiful! Very fine work. I'll look into doing this myself soon.

[Shaneus]

Thanks for this tip, definitely something I'm going to implement on my MiSTer (even if it is a little convoluted). Nice find!

[Asteld]

I'd recommend mounting the samba share directly on your mister. It saves you the hassle of manually having to transfer files.

An alternative is using hardlinks, which could be more convenient depending on your use case:

Code: Select all

hardlink [GAMEDIR]

HTGDB is a great initiative for organising your roms. See the link.

[FDA]

thanks for the guide, you can also use the command:
jdupes -r -l GAMEDIR
You can install jdupes with pacman, apt or yum (depending of your distro), it will find and remove duplicates creating symbolic links at the same time