MiSTer Downloader Tool

[akeley]

So, is nobody using these filters? I'd really like to get it sorted out, seeing as I have cores organised a little bit differently from normal on my SD card, and everytime I run update all I have to move stuff around.

[sawf]

Hi, I'm getting an error when running the downloader script of "cacert file at "/etc/ssl/certs/cacert.pem" seems to be wrong! Return code 6. It asks me to reboot and try again.

I've tried a fresh install of both the regular sd installer from github and the mr fusion image and it still returns the same error.

Is there anything I can do to fix this problem?

[SegaMan]

Hi, I'm getting an error when running the downloader script of "cacert file at "/etc/ssl/certs/cacert.pem" seems to be wrong!

Same message.
Any suggestions?

[PistolsAtDawn]

Hi, I'm getting an error when running the downloader script of "cacert file at "/etc/ssl/certs/cacert.pem" seems to be wrong!

Same message.
Any suggestions?

Are you running the Downloader directly instead of using the wrapper update_all.sh script?

https://github.com/theypsilon/Update_All_MiSTer

[SegaMan]

i am using update_all.sh

[prenetic]

Might be worth checking if the volume is somehow full -- I know it seems silly and unintuitive but I've run into this exact problem in the past and it was because I accidentally clogged up the volume with a bunch of data in a hidden folder.

[sawf]

Hi, I'm getting an error when running the downloader script of "cacert file at "/etc/ssl/certs/cacert.pem" seems to be wrong!

Same message.
Any suggestions?

Sorry I've only just seen this. In my case it doesn't like my wifi dongle anymore for some reason so I have to update it via ethernet which works fine. Bit annoying but ya know.

[Dafoof]

Getting a similar "cacert.pem" message, but with an error 60 instead, when trying to run udpate_all this evening (latest update I did was yesterday or the day before, and it ran perfectly fine).
It apparently tries to remove the file (3 attempts), but SFTPing into MiSTer, I can tell it's still there, and has a modification time from 7/7/2021.

EDIT: Nevermind, after another update attempt, it went all back to normal, without doing anything really... Not sure what happened here.

[Flandango]

Most of the cert errors that come up are caused by the clock being off.
If the Mister was just powered on or fairly recently connected to the internet, may want to wait till you get the date/time on the Mister Bar on the OSD before attempting to do an update. Of course make sure the date/time is fairly accurate and not days or years off.

[Dafoof]

Thanks Fandango for the explanation, that might be it. This being said, I have a RTC in my build, but maybe is it only used in the cores themselves?

[jca]

The RTC is also used in Linux. Did you set it up properly, did you set your time zone?

[Dafoof]

I'll double-check, it's been a while... But I think so, yeah.

[jvas]

With the new downloader I still have the "OSError: [Errno 18] Invalid cross-device link". Please find the log file attached. Any help appreciated!

[jca]

This is a known problem when using "base_path": "/media/usb0". It happened to me and others a long time ago.
The solution I found was to use symlinks from SD to everything which is on USB and use" base_path": "/media/fat".
Someone found the reason for the problem: it is a command used to transfer the file and this command does not work across file systems.
See:
https://www.geeksforgeeks.org/python-os-replace-method/

[jca]

You should be able to recover manually from the error. When it happened to me I got scared and did not reboot or turn off MISTer. I sshed into MISTer and copied the file manually followed by the command sync;sync and I rebooted.
Even if you rebooted or turned off MISTer you should still be able to boot MISTer and ssh into it.

[jca]

Question: why do you use base path to usb?
If your MISTer is set to "Switch to USB" don't, use symlinks instead.

[jvas]

Thanks for your answers.
I was able to update MiSTER and MAIN.rbf manually via ssh, so the problem is not critical.
With the old update script everything was working fine. This is the behaviour of the new script, where the author tried to update the main file in an atomic statement. For some unkown reason, it tires to back up the main mister files to USB0 instead of its original location.
Please advise me to a configuration, where
- every game roms, bioses, virtual HDDs are on the USB
- everything else is on the SDCARD (cores, configurations, scripts)

[jca]

This is exactly what happened to me: I used the old updater for a long time as initially I did not like the new downloader. I switched to the new downlaoder when many features of the old updater started to be removed and on the first update to main I got the error as others did.
Please confirm that the only directory you have is the games directory, if you have more please post the list.

[theypsilon]

Hey jvas, I'm the main maintainer of this tool. Sorry that you are facing this bug, but it's a hard one to reproduce. It's at this moment not fixed but still on my radar. GitHub issue is here: https://github.com/MiSTer-devel/Downloa ... /issues/24

For the setup you describe, you don't need base_path set to usb0, you ideally should just use the "storage priority" feature, which has been new since May, it's on by default, and it's described in the README: https://github.com/MiSTer-devel/Downloader_MiSTer and in my public Patreon post: https://www.patreon.com/posts/external-storage-66092738

Thanks to "storage priority," the downloader uses the same logic the firmware uses to load any file under games, by checking on external folders first. Thus, you can just move your games and docs folders to your USB folder, remove the config you had for your base_path, and all will work fine. No need to use cross-device symlinks, which are a source of problems in their own right.

[jvas]

Hey jvas, I'm the main maintainer of this tool....

Thanks. Will check that if there is another update of main and will let you know...
So base_path entry can be completely deleted from the ini file?

[theypsilon]

Hey jvas, I'm the main maintainer of this tool....

Thanks. Will check that if there is another update of main and will let you know...
So base_path entry can be completely deleted from the ini file?

Exactly. Yeah, let me know in the next main update

[jca]

...
No need to use cross-device symlinks, which are a source of problems in their own right.

This is totally false.

[theypsilon]

It's indeed a fact that cross-device symlinks are tricky, and your statement could be seen as a source of misinformation.

They are tricky because they don't have a warranted valid path since the external devices could be unmounted or mounted in different locations. They also violate assumptions on software that are taken to prevent corrupting files via atomic operations, as the other end of the symlink it's a different filesystem. Furthermore, they are not officially supported in ExFAT systems, and many utilities outside the MiSTer ecosystem could break them without notice. And I could continue with more reasons why they do bring trouble.

I've seen many users experiencing errors because of cross-device symlinks, so your comment is not very helpful. As I said previously, they can be used by power users that know how to troubleshoot all of this but it is not a recommended general practice.

[jca]

I must admit that my post was somewhat extreme, mainly due to the fact that recently you posted in different threads that symlinks were bad without giving any reason.
Now, after reading your post I realize that things are not so simple.
Following are some comments, my experience with symlinks. IT IS IN NO WAY A RECOMMENDATION FOR THE USE OF SYMLINKS (see theypsilon post above).
I am in no way what I would call a power usuer, it has been many, many years since I did some real work with Unix/Linux. I know more than enough to figure out things and if not I know where to look.
I was using "Switch to USB" until Sorge said it was kind of obsolete and one should use symlinks. I like to experiment so it is what I did.
If the drive is unmounted you also have the problem when just the games directory is on USB and no symlinks are used.
The drive is mounted differently, for example on usb1, I doubt it would happen in normal use. It never happened to me. I suppose it could happen if one connects a 2nd USB drive and on MISTer boot the new drive shows up as USB0 and the MISTer data drive shows up as USB1. I suppose if it happened the user would realize the problems he experiences has something to do with this new USB drive. Who would do such a thing? I did: I wanted to know how slow it would be to make a backup of my USB drive using only MISTer (I like to experiment). I connected the new drive while MISTer was up so it showed up as USB1, after the backup I unmounted the drive and disconnected it so I do not know what would have happened if I booted MISTer with both drives connected. If a switch had happened I wouldn't have noticed it as it was a backup. Note: don't try this at home as it took almost 2 days to do the backup.
Regarding atomic operations I did not think about that and I wonder how often such operations happen on MISTer and if they would cause problems. But as I said I am quite ignorant on the subject. I know they are used by the downloader when updating MISTer or Linux but everything happens on the SD card and symlinks are not involved.
EXTFAT and symlinks: MISTer supports them and it seems to work fairly well. But I experienced a problem which may have something to do with that:
Mame Getter and HBMame Getter did not like them for some reason and always reported 0 mra found. I "fixed" the problem by creating ini files pointing to the real location of /media/udb0/_Arcade, /media/usb0/games/mame and /media/usb0/games/hbmame.
I also must admit that I created a symlink for the games directory which seems useless and an heresy but I had a good reason.
The Minimig core has some kind of bug: when the games directory is on USB it only want its games directory in /media/usb0/Amiga, not /media/uisb0/games/Amiga which is not a big deal by itself but it also accepts only one KickStart which must be /media/usb0/KICK.ROM. This is a big deal as you can only have one configuration or several configurations using the same KickStart. With a symlink /media/fat/games -> /media/usb0/games Minimig is fooled and one can use /media/usb0/games/Amiga and any KickStart which allows different configurations with different KickStart.

Once again this is not a recommendation, just my experience and it has worked very well for me for more than a year.
Thanks to theypsilon for your downloader. I mentioned earlier that I initially I did not like it and kept using the old updater for a while, this was because it started to download things which the old updater didn't. When you are used to something it can be hard to switch to something new. In the end the downloader is way better, faster, offers more features. The end of the year is coming, R.I.P. the old updater which did it job and was way better than manually downloading core updates. Thanks also for the old updater, it made my life easier as did the introduction of the mras for the arcade cores (I am not sure who introduced them but thanks to him/them).

[segamemister]

@jca

Relating the minimig kickstart issue that should be fixed not sure about the games issue though

viewtopic.php?t=5038

[jca]

It does not look like it had been fixed:
My default configuration is MegaAGS, with the symlink /media/fat/games -> /media/usb0/games it starts properly. When I delete the symlink it fails with the error No KickStart found, system halted. If I use again the symlink it works.
This happens only when using the USB drive for the Amiga directory: it must be /media/usb0/Amiga, it does not allow /media/usb0/games/Amiga and the only KickStart which can be used is /media/usb0/KICK.ROM, nothing else works.
I read the thread but it does not mention where the Amiga directory is located.
I will make some more tests at some point instead of just deleting the symlink.

This is the link where this was discussed at the time:
viewtopic.php?p=29483#p29483

[Peredonov]

What's the database for mame and hbmame games to add to downloader.ini for the PC launcher of Downloader? My MiSTer cannot connect to the internet, so I need to keep the mame and hmame ROM folders updated via PC. Thanks for any tips.

[jvas]

Hey jvas, I'm the main maintainer of this tool....

Thanks. Will check that if there is another update of main and will let you know...
So base_path entry can be completely deleted from the ini file?

Exactly. Yeah, let me know in the next main update

Seems to be working. Needed to cleanup after the first run: the same folders appear both on usb0 and fat. Still not sure exactly what folders are affected by the "storage_priority" settings...

[Ganondork]

Hey there! This thread greatly helped me cleaning up my folders, removing unwanted cores. However, I just can't remove zx48, there seems to be no filter-tag for it. Any experience removing /filtering this core?

[jca]

This is an unofficial core, just manually remove it.