MiSTer Downloader Tool

[theypsilon]

This is a new updater designed with robustness, speed and safety in mind.

URL: https://github.com/MiSTer-devel/Downloader_MiSTer

Robustness

- It validates every downloaded file against its hash, eliminating the possibility of corrupted downloads.
- It removes old invalid files and avoids duplications. That means, the downloader will delete files that have been renamed on GitHub, and install them again with the new names.
- Can't miss an update because of some fluke with the GitHub API.
- In fact, it doesn't have to query the GitHub API at all, which has severe usage restrictions.
- It doesn't have to read brittle GitHub HTML code to decide what has to download neither.
- It has extensive automated test coverage, through unit, integration and end-to-end system tests.

Speed

- Fresh installs are around 3x faster than with the MiSTer Updater.
- Incremental updates are nearly instant.
- It can consistently download many files in parallel without issues.
- Lots of opportunities to make it even faster in next iterations.

Safety

- It uses secure connections. If you don't have the proper certificates, it will help you to install them. Most users will have some certificates already, since the system provides them by default.
- It doesn't run INI files that can contain arbitrary code under the root user. That could lead to hackers taking total control of your system through a malicious INI file. Instead, it uses a very safe INI parser (ConfigParser from Python).
- Hash validation will guarantee that downloaded files are trusted.

In short, this tool will solve long-standing issues that users were facing with the old Updater script, while providing other nice benefits. Many problems previously experienced by the users were related to corrupted downloads, and these should go away.

All files installed through the Downloader by default are coming from this repository: https://github.com/MiSTer-devel/Distribution_MiSTer . In fact, it will be a 1:1 copy of that structure into your system.

From a more technical point of view, the whole thing has been implemented with Python, which makes it quite maintainable, modular and efficient. So it should be more approachable to new contributors. Internally, it uses a database that's being auto-generated in the server, and this DB just contains the links of what it has to fetch. This DB format will be documented soon. And that might be useful to make this solution much more expandable, by just modifying the downloader.ini file. It's not officially supported yet, though. There are a few other things in the code that are not documented yet, so keep in mind, if you find a feature that is not documented, it's not officially supported, and that means it could go away in the future without notice (in reality, it's unlikely, but still, we should keep this as a principle).

The current Roadmap is on the README.md from the GitHub repository. If you miss a feature from old MiSTer Updater script, you probably should keep using it for the mean time.

Any feedback is very welcomed.


PD. Thanks to all the people that has helped me to test this project. Specially, thanks to Pig_Saint, atrac17, Sorgelig, amoore2600, Kitrix, Fed, jotego, Antonio, ItalianGrandma, Guillermo, and probably others that I might have forgotten along the long way (sorry!). That kind of feedback is always invaluable.

[akeley]

Hi, just to clarify, is this a replacement for the old official updater? Or another alternative? And it's not the update_all thingy, right?

Sorry if these are obvious questions, but I do get a bit confused by all these updaters

[theypsilon]

Hi, just to clarify, is this a replacement for the old official updater? Or another alternative? And it's not the update_all thingy, right?

Sorry if these are obvious questions, but I do get a bit confused by all these updaters

Yes, this is meant to replace the old official updater. No, this is not meant to be the continuation of update_all

[theypsilon]

I published some changes for tweaking the move operation and increasing the waiting time before reboot.

The goal is trying to minimize the MiSTer and menu.rbf file corruption that has been rarely happening during the Linux & Main update process since the start of the old MiSTer Updater.

Keep in mind, although the problem happens in all available updaters, the incidence is very low. And with the new Downloader the occurence should be even lower since corrupted downloads are not happening anymore, as files are being validated against their hash. I'm being very transparent about these findings on social media, but I realize now that I might have created unnecessary alarm along the way. My apologies.

The fix when corruption happens is the following one:
Copy the MiSTer and menu.rbf files from here (MiSTer) and here (menu) to the root of your SD.

As of now, the recommended procedure for removing all possibilities of having this issue, is not updating Linux through any updaters. With the Downloader you can specify disabling Linux Updates by adding the following line at /media/fat/downloader.ini below the [MiSTer] section:

Code: Select all

[MiSTer]
update_linux = 'false'

Further investigation is ongoing in order to fix the root cause of the problem once and for all. One of the current hypothesis would be that there is a race condition when there is a big amount of data being written to the SD before a reboot.

[jca]

Thanks for the new script and finally a readme file with some explanations regarding the downloaded.ini file.
As per the readme: "This tool installs and updates all the cores and other extra files for your MiSTer."
Is there any possibility to add a line to disable the download and install of these other extra files?
When the script was announced I tried it and it downloaded a bunch of things like a gazillion fonts, palettes, bioses and such. I do not like very much this idea and would rather manually download what I need and not everything in existence.
Regarding symlinks like GBA and GBA2P: I already have these 2 directories, will the downloader modify anything in this case?
Not all GBA games support the linking of 2 GBAs to play in 2P mode, until now I have 2 directories and created symlinks in GBA2 for the game supporting linking. It is difficult to find an exhaustive list of such games and require some work to create the symlinks. I suppose that running a game which does not support linking will work in GBA2 but in this case it won't be multiplayer but each player will run its own game. Is this correct?

[theypsilon]

Thanks for the new script and finally a readme file with some explanations regarding the downloaded.ini file.
As per the readme: "This tool installs and updates all the cores and other extra files for your MiSTer."
Is there any possibility to add a line to disable the download and install of these other extra files?
When the script was announced I tried it and it downloaded a bunch of things like a gazillion fonts, palettes, bioses and such. I do not like very much this idea and would rather manually download what I need and not everything in existence.
Regarding symlinks like GBA and GBA2P: I already have these 2 directories, will the downloader modify anything in this case?
Not all GBA games support the linking of 2 GBAs to play in 2P mode, until now I have 2 directories and created symlinks in GBA2 for the game supporting linking. It is difficult to find an exhaustive list of such games and require some work to create the symlinks. I suppose that running a game which does not support linking will work in GBA2 but in this case it won't be multiplayer but each player will run its own game. Is this correct?

Filtering downloads is in the roadmap:

image.png (50.89 KiB) Viewed 43298 times

When that's available, there will be ways to have fine-grained control over what is getting downloaded. This was actually shared since the release day, so I expected there wouldn't be many surprises.

About the symlinks on GBA and GBA2P, the idea would be having it opt-in and would just link games/GBA2P to games/GBA. There won't be any rom manipulation made by this tool there. Maybe there can be another solution for that that does not need to be included in the downloader.

[jca]

On release day there was no mention of the downloader downloading a bunch of stuff not previously downloaded with the update script, there was not even a readme file. So yes I was surprised when it downloaded a gazillion fonts, bioses and what not.

Regarding the symlinks it was just an observation, not a request for anything.

I must say that the downloader is really fast, especially on the second run.

I also read a tweet which seems to imply that the update all script uses the downloader. Is this correct?

[theypsilon]

Yes, update_all is now using the Downloader by default (except for users that have custom update.ini files). If you see the word "Downloader" at the start of the console output when it shows the sequence, it is using it, but you can always deactivate it from the Misc menu.

The README.md has always been there with the same content since one day before the public announcement, the git history doesn't lie.

[jca]

The README.md has always been there with the same content since one day before the public announcement, the git history doesn't lie.

I see, I have been tricked by the announcement made by Sorge which pointed to:
https://github.com/MiSTer-devel/Downloa ... r/releases
I guess I was too excited by the new update and did not see the link ended with releases and is not apparent once you click on the link.

[ash2fpga]

Further investigation is ongoing in order to fix the root cause of the problem once and for all. One of the current hypothesis would be that there is a race condition when there is a big amount of data being written to the SD before a reboot.

Interesting that even with the sync calls it is an issue.

When using iomega zip disks back in the day, running sysinternals sync became a must, before ejecting a disk. No fun going to a customer site for a deploy only to find out your files are corrupted.

I still run sync to this day, before "ejecting" any usb storage.

[Bas]

Reinventing the wheel seems kind of rampant here these days. What you have here is dpkg without the packages.

[theypsilon]

I published some changes for tweaking the move operation and increasing the waiting time before reboot.

The goal is trying to minimize the MiSTer and menu.rbf file corruption that has been rarely happening during the Linux & Main update process since the start of the old MiSTer Updater.

Keep in mind, although the problem happens in all available updaters, the incidence is very low. And with the new Downloader the occurence should be even lower since corrupted downloads are not happening anymore, as files are being validated against their hash. I'm being very transparent about these findings on social media, but I realize now that I might have created unnecessary alarm along the way. My apologies.

The fix when corruption happens is the following one:
Copy the MiSTer and menu.rbf files from here (MiSTer) and here (menu) to the root of your SD.

As of now, the recommended procedure for removing all possibilities of having this issue, is not updating Linux through any updaters. With the Downloader you can specify disabling Linux Updates by adding the following line at /media/fat/downloader.ini below the [MiSTer] section:

Code: Select all

[MiSTer]
update_linux = 'false'

Further investigation is ongoing in order to fix the root cause of the problem once and for all. One of the current hypothesis would be that there is a race condition when there is a big amount of data being written to the SD before a reboot.

Coming back to this topic. I've created a stress test for trying to replicate this issue.

https://github.com/MiSTer-devel/Downloa ... corruption

So far, I've been running it for over a day and I couldn't reproduce the issue myself yet. It might be because the issue is solved with the latest mitigation changes, or because my SDs are not prone to this error at all. In any case I would say that this already proves that the issue is very unlikely to happen if it still happens, so that's good news.

If some of you would like to help by giving this stress test a try, please read carefully the README.md file linked above and go ahead. It would be very valuable to have a few reports on this. That being said, running a script like this for a long time can not come without risks. Things should not explode, but I have to remind you we are testing whether files can get corrupted here, so yeah, your files could get corrupted due to this stress test. Also, I wouldn't run this if your setup doesn't dissipate heat properly (if you have a fan, you should be good).

[aberu]

Reinventing the wheel seems kind of rampant here these days. What you have here is dpkg without the packages.

Theypsilon's new method is excellent. It's faster, more reliable, and still very minimal (a small filesize script) since it has basically zero impact on the size of the linux limage. There are good reasons for doing it the way he did it. He put a lot of work into the script so I think we should not be making short and dismissive comments that undermine all the work he did here.

If you find a way to implement traditional packages instead, while keeping the linux image under the maximum file size in github repos, you are free to make the changes and submit them as a pull request for review. This isn't meant to be snarky (as this kind of reply sometimes is), I am being serious because if there is a benefit we should be open-minded and encourage everyone to try and contribute and improve the project together.

If you'd like to manage those packages or configure the automation for them too, that's great.

[Bas]

The thread in which I attempted earlier to make the case for package management was summarily closed by sorg, so that's where any effort on my side stopped. I actually had dpkg working. If the project maintainer doesn't like it, then that's his prerogative and I stopped my efforts. I'm now focusing on "consolizing" AO486 instead. I still think these scripts are a serious case of NIH syndrome but I'll leave it at that.

[ash2fpga]

The thread in which I attempted earlier to make the case for package management was summarily closed by sorg, so that's where any effort on my side stopped. I actually had dpkg working. If the project maintainer doesn't like it, then that's his prerogative and I stopped my efforts. I'm now focusing on "consolizing" AO486 instead. I still think these scripts are a serious case of NIH syndrome but I'll leave it at that.

Understandable, I think, from both sides (project maintainer / contributor). I have tried submitting a PR on a script, which was never accepted (afaicr).

I would like to see, perhaps, a directory of Community (Unofficial) scripts / tools. If we can acknowledge unofficial cores, I think we could acknowledge unofficial scripts, too.

[Moondandy]

I was thinking something similar, that it could be useful to have a scripts_getter script that would grab the latest version of various useful unofficial scripts, like the Top 300 updater, and have that run as part of update_all. If desirable first step would presumably be to collate a list of scripts people use that aren't included in Main.

I am really hoping that the 240p Test Suit getter script, and the tty2oled updater script can at some point be included in (ideally) the main updater and there to be options to run those as part of the standard update.

[morgoth]

Yes, update_all is now using the Downloader by default (except for users that have custom update.ini files). If you see the word "Downloader" at the start of the console output when it shows the sequence, it is using it, but you can always deactivate it from the Misc menu.

The README.md has always been there with the same content since one day before the public announcement, the git history doesn't lie.

How do I know if I have a custom update.ini? When I ran update_all, I don't believe I saw "Downloader" displayed anywhere. I can tell you that I do have "update.ini"...should I simply erase it to be sure update_all uses this new and improved Downloader?
Thanks much,

[jca]

You don't have to do anything with the ini file. Start update_all.sh and press the up arrow to go to the menu. In the Misc section you can select the old update.sh or new downloader. Do not forget to save the changes.

[morgoth]

Worked like a charm. Thanks jca

[Solskogen]

As I've reported in github, the new downloader tool doesn't fetch the bluetooth script

[theypsilon]

The Downloader 1.2 version was released last weekend.

This version includes support for custom databases. This feature enables a way to integrate all kinds of file downloads inside the downloader.

In order to take advantage of it, contributors will have to publish their databases. Once databases pop up in the community, users will be able to integrate it by adding a couple of lines to the file /media/fat/downloader.ini

One database that somebody could do, for example, would be a compilation of additional useful scripts that maybe some people miss.

Here is documentation for contributors about how to create custom databases: https://github.com/MiSTer-devel/Downloa ... tabases.md

[theypsilon]

I published some changes for tweaking the move operation and increasing the waiting time before reboot.

The goal is trying to minimize the MiSTer and menu.rbf file corruption that has been rarely happening during the Linux & Main update process since the start of the old MiSTer Updater.

Keep in mind, although the problem happens in all available updaters, the incidence is very low. And with the new Downloader the occurence should be even lower since corrupted downloads are not happening anymore, as files are being validated against their hash. I'm being very transparent about these findings on social media, but I realize now that I might have created unnecessary alarm along the way. My apologies.

The fix when corruption happens is the following one:
Copy the MiSTer and menu.rbf files from here (MiSTer) and here (menu) to the root of your SD.

As of now, the recommended procedure for removing all possibilities of having this issue, is not updating Linux through any updaters. With the Downloader you can specify disabling Linux Updates by adding the following line at /media/fat/downloader.ini below the [MiSTer] section:

Code: Select all

[MiSTer]
update_linux = 'false'

Further investigation is ongoing in order to fix the root cause of the problem once and for all. One of the current hypothesis would be that there is a race condition when there is a big amount of data being written to the SD before a reboot.

After a few months, and a couple of linux & main updates, it seems that the fix I did worked. I haven't found any occurrence of this problem anymore after extensive testing.

[Solskogen]

To ensure that it won't happen again, I think adding

Code: Select all

sync 

before reboot should be enough.

[theypsilon]

To ensure that it won't happen again, I think adding

Code: Select all

sync 

before reboot should be enough.

That's not it, we had a couple of syncs before, after changing all the files. The old Updater has them too, and is still vulnerable to this problem.

[theypsilon]

You now can filter what and what not to install thanks to a new feature in Downloader 1.4

Here is the documentation: https://github.com/MiSTer-devel/Downloa ... filters.md

[Fallon]

Any way to filter out Jotego's arcade alternative cores? I can filter out the mainstream mister arcade cores no problem, but not Jotego's.

[jvas]

I tried to use the new downloader:
"
Running MiSTer Downloader

START!

Reading file: /media/fat/downloader.ini
Reading 'distribution_mister' db section
Reading 'jtcores' db section
Reading 'theypsilon_unofficial_distribution' db section
Reading 'names_txt' db section
Could not download file from db_url: "https://raw.githubusercontent.com/joteg ... b.json.zip"
Could not download file from db_url: "https://raw.githubusercontent.com/Three ... on_US.json"
Could not download file from db_url: "https://raw.githubusercontent.com/theyp ... b.json.zip"
Could not download file from db_url: "https://raw.githubusercontent.com/MiSTe ... b.json.zip"
"

Which script is running? I ccan't find the "downloader.sh" in the Scripts directory.
Thanks for your help in advance!

Edit: the log file is attached now:

downloader.zip

(1.23 KiB) Downloaded 352 times

[theypsilon]

Any way to filter out Jotego's arcade alternative cores? I can filter out the mainstream mister arcade cores no problem, but not Jotego's.

In which section of downloader.ini did you put the filter? [mister] or [distribution_mister]?

I tried to use the new downloader:
"
Running MiSTer Downloader

START!

Reading file: /media/fat/downloader.ini
Reading 'distribution_mister' db section
Reading 'jtcores' db section
Reading 'theypsilon_unofficial_distribution' db section
Reading 'names_txt' db section
Could not download file from db_url: "https://raw.githubusercontent.com/joteg ... b.json.zip"
Could not download file from db_url: "https://raw.githubusercontent.com/Three ... on_US.json"
Could not download file from db_url: "https://raw.githubusercontent.com/theyp ... b.json.zip"
Could not download file from db_url: "https://raw.githubusercontent.com/MiSTe ... b.json.zip"
"

Which script is running? I ccan't find the "downloader.sh" in the Scripts directory.
Thanks for your help in advance!

Edit: the log file is attached now:

downloader.zip

There seems to be a problem with your ssl certificates. You could delete what you have at /etc/ssl/certs/* so it tries to grab new ones. You could also install cacert manually by downloading this file and putting it in that folder: https://curl.se/ca/cacert.pem

[jvas]

There seems to be a problem with your ssl certificates. You could delete what you have at /etc/ssl/certs/* so it tries to grab new ones. You could also install cacert manually by downloading this file and putting it in that folder: https://curl.se/ca/cacert.pem

Deleted everything from /etc/ssl/certs/. Stills the same

[jvas]

There seems to be a problem with your ssl certificates. You could delete what you have at /etc/ssl/certs/* so it tries to grab new ones. You could also install cacert manually by downloading this file and putting it in that folder: https://curl.se/ca/cacert.pem

Putting "cacert.pem" solved the problem. Thanks !!!