MiSTer Downloader Tool

[jvas]

OK, how to configure it in the following way:
- I want Scripts, Core, Config, BIOS, Font, Gamma, linux, Scripts, Saves, Savestates on the SDCARD
- Cheats, Games, Filters, Filters_Audio on the USB0

Thanks!

[Fallon]

Any way to filter out Jotego's arcade alternative cores? I can filter out the mainstream mister arcade cores no problem, but not Jotego's.

In which section of downloader.ini did you put the filter? [mister] or [distribution_mister]?

I have the following filter placed under [distribution_mister] and Jotego's alternatives are downloaded

filter = !cheats !alternatives !acornatom !acornelectron !alicemc10 !altair8800 !apogee !apple-i !aquarius !archie !bk0011m !c16 !coco2 !edsac !galaksija !interact !jupiter !laser310 !macplus !multicomp !ondra_spo186 !orao !oric !pc88 !pdp1 !pet2001 !pmd85 !ql !rx78 !sharpmz !sordm5 !specialist !svi328 !ti994a !trs-80 !tsconf !uk101 !vector-06c !vic20 !zx81 !ay-3-8500 !arcadia !astrocade !channelf !intellivision !atarilynx !vectrex !odyssey2 !vc4000 !wonderswan !other !laser !pc8801

If I change it around and put the filter under Mister, then Jotego's alternative are not downloaded, but the other arcade core alternatives are then downloaded.

[yoshi41]

Finally switched to the new downloader tool. It's much faster. Thanks for your work.

For those who use filters, that's what I had to change.

Old filter /media/fat/Scripts/update_all.ini :

Code: Select all

REPOSITORIES_FILTER="arcade-cores other-cores service-cores AO486 AtariLynx AtariST C64 Minimig Gameboy GBA Genesis Intellivision MegaCD NeoGeo NES SMS SNES TurboGrafx16 WonderSwan"

New filter /media/fat/downloader.ini :

Code: Select all

[mister]
filter = all-filters cheats font scripts arcade-cores other-cores service-cores AO486 AtariLynx AtariST C64 Minimig Gameboy GBA Genesis Intellivision MegaCD NeoGeo NES SMS SNES TurboGrafx16 WonderSwan

The available filter tags can be found here :

https://github.com/MiSTer-devel/Distrib ... rs-feature

[fenderc01]

I currently use the update script and I have the console cores download to the root of the card, with everything else in the default folders. I like my consoles to be the first thing I see, since that's what I use 99% of the time.

Is there an option or will there be an option in the future to change this for the downloader script.

This is how my update.ini is configured:

Code: Select all

CORE_CATEGORY_PATHS["console-cores"]="$BASE_PATH"

Everything else is set to the default paths.

[Sigismond0]

I currently use the update script and I have the console cores download to the root of the card, with everything else in the default folders. I like my consoles to be the first thing I see, since that's what I use 99% of the time.

Is there an option or will there be an option in the future to change this for the downloader script.

This is how my update.ini is configured:

Code: Select all

CORE_CATEGORY_PATHS["console-cores"]="$BASE_PATH"

Everything else is set to the default paths.

Same here. In a recent Twitter thread it was confirmed that custom paths like this are not likely to happen anytime soon. Since the old scripts are now EOL and will stop working at the end of the year (assuming they don't break before then), for the time being you'll just have to get used to the Console folder again. Hopefully, we'll get this functionality back eventually.

If you can't live without it, you could make a custom script that deletes all RBFs in root, moves everything in /_Consoles to root, then deletes /_Consoles. And I'm reasonably certain you could have that run as part of Update_All by adding a custom repository.

[Atohmdiy]

The thread in which I attempted earlier to make the case for package management was summarily closed by sorg, so that's where any effort on my side stopped. I actually had dpkg working. If the project maintainer doesn't like it, then that's his prerogative and I stopped my efforts. I'm now focusing on "consolizing" AO486 instead. I still think these scripts are a serious case of NIH syndrome but I'll leave it at that.

After some times i stop reading my thread about package manager... So i don't even know why the thread was closed.
But let's say i think exactly like you. A shame to put so much effort and energy in something that exist for a very long time with excellent open source tools, especially considering the diversity of these tools in the linux world. Anyway let's not start again...

I tried to download and use the new scripts when sorge announce it, i quickly stop it as it decides to download everything there is. I just try to understand how the last version is working now. With the previous script there is an option to avoid download new core (DOWNLOAD_NEW_CORES="false"). This function is mandatory to me, i manually downloaded what i wanted and i want the script to update only what i have now. What i understand is that the new filter section can doing this, but you can either choose what you want or not with the whole section (arcade-core) or a specific list (arcade-robotron arcade-rshnatk etc). It's so much more complicated than the previous script, and i won't use it until the same function exist.
I hope with this script you won't have to wait ages like the previous script that systematically check all cores, to then understand you don't want them with DOWNLOAD_NEW_CORES="false"

[xFlesk]

hello there
im having a strugle with filtering arcade cores. i want to remove "280Z-ZZAP" game and "280Z-ZZAP (US)" game. so i added this to my downloader ini:

Code: Select all

[mister]
filter = !Arcade-280ZZZAP

for some reason this doesn't work. dont know why? this is pretty simple title and im struggle i wonder what im going to do when i try to remove for example "Action Fighter (World, S16A) [FD1089A 317-0018]" game with all this extra symbols like "(" "[" etc.
any help ?
thank you in advance

[grizzly]

You said remove!
I´m not sure but i would think that the downloader tool do not remove anything only stops downloading whatever you set if it is missing.
So first you have too manually remove the things you do not want but already have downloaded for it too work.

I guess you already have done that, but since you did say remove there is a chance that you have not removed anything manually.

[Sigismond0]

Downloader tool does remove anything that's no on your download list--when I set !computers in my filter, it went through and deleted those RBFs and the _Computers directory once it was empty.

As far as I can tell it syncs your local files/directories to match the repository, both downloading new stuff and deleting anything that no longer matches (e.g, when filters folders were re-organized, it deleted the old structure/files and downloaded the new ones. No duplicates.)

[xFlesk]

downloader does remove the files, which are listed in the ini file. it removed all the computer, console and other cores which i choosed to be filtered.
then i decided to filter all the arcade cores (one by one) which im not going to use and i fall on the very first arcade core
here is my entire filter entry

Code: Select all

[mister]
filter = !Docs !Other-cores !Arcadia !Astrocade !AY-3-8500 !ChannelF !Gameboy2P !GBA2P !Intellivision !Odyssey2 !VC4000 !Vectrex !AcornAtom !AcornElectron !AliceMC10 !Altair8800 !Amstrad-PCW !Apogee !Apple-I !Apple-II !Aquarius !BK0011M !CoCo2 !CoCo3 !EDSAC !Galaksija !Interact !Jupiter !Laser310 !MacPlus !MultiComp !Ondra_SPO186 !ORAO !Oric !PC88 !PDP1 !PET2001 !PMD85 !QL !RX78 !SAMCoupe !SharpMZ !SordM5 !Specialist !Svi328 !TatungEinstein !Ti994a !TRS-80 !TSConf !UK101 !Vector-06C !VIC20 !zx48 !ZX81 !Arcade-280ZZZAP

from this entire list i have problem only with arcade core 280Z-ZZAP. does anyone successfuly filtered a single arcade core ?
depending on the official tutorial it should be easy as this:

Code: Select all

[mister]
filter = !Arcade-pong

but it isn't
any ideas ?

[Sigismond0]

Not sure why !arcade-pong doesn't work for you, looks like it should.

I think the reason 280z-zzap isn't working is because it's part of the Space Invaders core. You'd have to do !arcade-spaceinvaders.

Edit: Can confirm that !arcade-pong removes the RBF for pong, but appears to leave the MRA file out there. So it's still in the list and you can select it, but it doesn't do anything. !arcade-spaceinvaders ensures that 280z Zzap doesn't work, but again the MRA still lives on. Not sure if this is an issue with the updater script, or the Arcade Organizer or what.

I suspect that if you use the Update_All tool to delete the Arcade directory, and then re-build with these filters in place it will do what you want. But that's a longer test than I want to run.

[xFlesk]

yes, *.mra file stays in arcade folder, which means update_all will download a rom file again upon next update. even if i manualy remove *.mra file, update_all will download it anyway and then download rom file as well.
i think its impossible to remove single arcade core.

[theypsilon]

Not sure why !arcade-pong doesn't work for you, looks like it should.

I think the reason 280z-zzap isn't working is because it's part of the Space Invaders core. You'd have to do !arcade-spaceinvaders.

Edit: Can confirm that !arcade-pong removes the RBF for pong, but appears to leave the MRA file out there. So it's still in the list and you can select it, but it doesn't do anything. !arcade-spaceinvaders ensures that 280z Zzap doesn't work, but again the MRA still lives on. Not sure if this is an issue with the updater script, or the Arcade Organizer or what.

I suspect that if you use the Update_All tool to delete the Arcade directory, and then re-build with these filters in place it will do what you want. But that's a longer test than I want to run.

Indeed !arcade-spaceinvaders should work for that. It will remove 280z-zzap MRA (I've just tested it). !arcade-pong removes Pong MRA and RBF only, which is the expected result. What maybe is not removed is the symlink on _Arcade/_Organized folder, which is handled by the Arcade Organizer, not by Downloader. That would be another topic.

Currently there is nothing to target specifically a single MRA without referencing its core and all other MRAs using that core. It does not make much sense to do that for most users anyways. When you choose to download a core, you typically want to download all their games (not counting with hbmame and alternatives here). I could add the possibility to filter single MRAs by tagging them with their setname (which is unique for each MRA), but I'm not convinced yet. It would make the database much heavier.

I currently use the update script and I have the console cores download to the root of the card, with everything else in the default folders. I like my consoles to be the first thing I see, since that's what I use 99% of the time.

Is there an option or will there be an option in the future to change this for the downloader script.

This is how my update.ini is configured:

Code: Select all

CORE_CATEGORY_PATHS["console-cores"]="$BASE_PATH"

Everything else is set to the default paths.

There isn't a simple way to redefine paths like that yet, but for people wanting to have a special folder structure, a possibility would be writing a custom database. For example, you could make your own "console cores database", and then filter out the consoles from the official database by defining 'filter = !consoles', so it doesn't conflict with your definitions. Here is the documentation on how to write a custom database: https://github.com/MiSTer-devel/Downloa ... tabases.md

[SS4Rob]

Hello,
I was using update_all, but I am trying to switch to the new downloader. When I run the downloader.sh, I get a python3 not found error. I am also one of the people that get a black screen whenever I update, so I have to revert to an older version of the MiSTer app. Any assistance would be appreciated, thanks.

[grizzly]

Doooh! Stupid brain
Well looks like my stupid brain was wrong again!

[Atohmdiy]

You right ignore me because i said i prefer a package manager.

So i tried this script and it did exactly what i don't want it to do, i suppose it's a troll really. After all, even if i really don't like the concept, you need to take the train someday. So i run these options in filter :
!arcade-cores arcade-segasys1 !computer-cores console-cores !other-cores service-cores !alternatives filters_video !hbmame !handheld2p essential !readme !docs extra-utilities cheats

I set the arcade-segasys1 filter to see if it still download this core when the "meta" is set with the !.

And of course it keep downloading all the arcade, computer readme etc i don't want and that i specify not to download. So filter option doesn't work or i'm dumb.

And you don't want me to say it but i will keep saying it. You want people to loose their time to learn how to use this script and these options. Options that are overcomplicated and even more annoying than the previous script. Because between set one parameter (here DOWNLOAD_NEW_CORES="false") and have to mess with this big list in the filter option, i think no one could say the second option is easier. This script is indeed easier indeed when you are the guy that want to have everything there is, but not for people that want specific things like me.

And with a package manager just create a package that include the core and the mra (or two package, one for the core and one for the mra if there is a reason to do it). You install the packages you want when you set up your mister, and then let the package manager update everything. It's so much easier than spending hours messing with a braindead script options.

Edit :
I just move the filter options at the beginning of the .ini file, maybe there is a difference. Delete all in the _Computer folder and run the script again, keep downloading it :

o1hv.png (381.62 KiB) Viewed 14544 times

[lunch_box]

Can I ask a potentially stupid question please?

I’m hoping to put together my first mister build this weekend. Do I need to run this downloader as well as update_all? Or can I just run my fusion and then add update_all and run that?

Thanks all

[Sigismond0]

Can I ask a potentially stupid question please?

I’m hoping to put together my first mister build this weekend. Do I need to run this downloader as well as update_all? Or can I just run my fusion and then add update_all and run that?

Thanks all

Update_All already uses this downloader, so no extra steps needed.

[theypsilon]

Hello,
I was using update_all, but I am trying to switch to the new downloader. When I run the downloader.sh, I get a python3 not found error. I am also one of the people that get a black screen whenever I update, so I have to revert to an older version of the MiSTer app. Any assistance would be appreciated, thanks.

It seems that your Linux version is too old, you would have to update Linux with the older updater or start the installation from scratch. Downloader needs python3 in the system. Pretty much all Linux versions have python3 in it for the latest 1.5 years.

Can I ask a potentially stupid question please?

I’m hoping to put together my first mister build this weekend. Do I need to run this downloader as well as update_all? Or can I just run my fusion and then add update_all and run that?

Thanks all

Update_All already uses this downloader, so no extra steps needed.

Correct.

You are very welcomed to send a PR with a "download_new_files=false" feature.

[SS4Rob]

Hello,
I was using update_all, but I am trying to switch to the new downloader. When I run the downloader.sh, I get a python3 not found error. I am also one of the people that get a black screen whenever I update, so I have to revert to an older version of the MiSTer app. Any assistance would be appreciated, thanks.

It seems that your Linux version is too old, you would have to update Linux with the older updater or start the installation from scratch. Downloader needs python3 in the system. Pretty much all Linux versions have python3 in it for the latest 1.5 years.

Bingo! I've been on this same MiSTer build for years, always updating with "update.sh" and "update_all.sh", so I assumed it was handling everything, including Linux. With your suggestion, I peeked in update.ini and saw this:

Code: Select all

#EXPERIMENTAL: specifies if the Kernel, the Linux filesystem and the bootloader will be updated; use it at your own risk!
UPDATE_LINUX="false"

#EXPERIMENTAL: specifies if the update process must be done with parallel processing; use it at your own risk!
PARALLEL_UPDATE="false"

Flipping that to "true" updated my Linux and *poof*, 'python3' was there, with "downloader.sh" working perfectly! Also, the black screen after update problem is gone as well - I always had to revert to MiSTer_20210917 to get video back, but now, the latest version is working correctly. I can't imagine me manually not allowing Linux to update, so maybe this was an old default at some point that I never noticed? Anyway, I'm good to go!

Thank you @theypsilon!

[cursedverses]

I can see a lot of people having trouble in here, so I'm going to drop a file that works for me that people should be able to edit to their preference.

Code: Select all

[MiSTer]
base_path = '/media/fat/'
base_system_path = '/media/fat/'
allow_delete = 2
allow_reboot = 2
update_linux = true
parallel_update = true

[distribution_mister]
filter = essential all_filters font scripts amiga ao486 amstrad atarist mastersystem megadrive nintendo playstation supernintendo

Now I have a USB drive (fresh formatted to EXFAT), so once I ran this the first time, I modified base_path to '/media/usb0/' which triggered a migration of files to the drive on the second run. Running it before led to issues with the mister binary (cannot link cross-device, I think).

Also, playstation in my filter list won't work until it's supported (it's ignored for now).

[theypsilon]

I can see a lot of people having trouble in here, so I'm going to drop a file that works for me that people should be able to edit to their preference.

Code: Select all

[MiSTer]
base_path = '/media/fat/'
base_system_path = '/media/fat/'
allow_delete = 2
allow_reboot = 2
update_linux = true
parallel_update = true

[distribution_mister]
filter = essential all_filters font scripts amiga ao486 amstrad atarist mastersystem megadrive nintendo playstation supernintendo

Now I have a USB drive (fresh formatted to EXFAT), so once I ran this the first time, I modified base_path to '/media/usb0/' which triggered a migration of files to the drive on the second run. Running it before led to issues with the mister binary (cannot link cross-device, I think).

Also, playstation in my filter list won't work until it's supported (it's ignored for now).

This one should work fine, indeed.

But if you don't mind, I would suggest these changes to your configuration:

Code: Select all

[MiSTer]
base_path = '/media/fat/'
allow_reboot = 2
filter = essential all_filters font scripts amiga ao486 amstrad atarist mastersystem megadrive nintendo playstation supernintendo

I removed allow_delete cause the default value seems to be better nowadays. We are currently renaming a huge amount of files that are not cores (video processing filters, shadow masks, and so on), and with allow_delete=2 you will end up with duplicates.

Removed base_system_path too cause that one is by far the least likely to change of all options. Only makes sense to change it for downloader testers.

Removed update_linux and parallel_update because they have the same value as the default, and there shouldn't be a need to change them.

Moved filter from [distribution_mister] to [MiSTer] because it's a bit more tidy this way and should work the same.

Removed [distribution_mister] because now it doesn't contain any configuration parameter after moving filter.

[cursedverses]

Wow, that's a lot of changes, but they all make sense to me. Thanks so much!

[theypsilon]

Downloader 1.5 has just been released: https://github.com/MiSTer-devel/Downloa ... ANGELOG.md

[Alkadian]

Downloader 1.5 has just been released: https://github.com/MiSTer-devel/Downloa ... ANGELOG.md

Thanks!

[lsnderick]

Hi, first I'd like to say big thanks for all the work by theypsilon for this great tool, and his continuous development of it. Currently my favorite feature is 'Download Filters', really helpful for keeping the setup clean. But I have a question, regarding the 'palettes' filter term. I don't have it in my downloader.ini but every time it runs it keeps downloading the Palettes into games/NES folder. Are they mandatory? Or is it some kind of a bug currently?

[theypsilon]

Palettes are in by default, but you can remove them by adding the term "!palettes" to your filter. Thanks for the support!

[MadDog]

The downloader.ini is disobeying me. I've added MysticMarathon and TurkeyShoot to my 'exemptions' in the ini by using !arcade-mysticmarathon and !arcade-turkeyshoot, and it keeps downloading these two. I have all of my exemptions and inclusions in alphabetical order on my list, to keep track. Not sure why it skips over these two, when it properly excludes all others just fine. Curious...

[theypsilon]

You can consult the terms for the filters here: https://github.com/MiSTer-devel/Distribution_MiSTer
Make sure it matches first with that list.

[akeley]

I've just tried using filters for the first time, but after editing the config update_all fails with a lot of errors. All I have done is put these lines in downloader.ini. It looks like this atm:

Code: Select all

[mister]
base_path = '/media/fat/'
allow_reboot = 2
filter = !arduboy !gba2p !gameboy2p !cheats !alicemc10 !altair8800 !apogee !apple-i !aquarius !arcadia !archie !ay-3-8500 !bk0011m !edsac !galaksija !gamenwatch !jupiter
!laser !laser310 !lynx48 !multicomp !ondra_spo186 !orao !pdp1 !pmd85 !rx78 !sordm5 !specialist !sufami !tsconf !uk101 !vector06

[distribution_mister]
db_url = https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip

[jtcores]
db_url = https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip

[bios_db]
db_url = https://raw.githubusercontent.com/theypsilon/BiosDB_MiSTer/db/bios_db.json

Before that using update_all has always worked ok. Well, at least I think so, have never really read all the messages it displays, but they mostly looked okay.