MiSTer FPGA Forum

Posted: **Sun Jun 12, 2022 1:46 am**

Hello! I'm having some very odd issues with using the two downloader scripts on my mister.
This is the mister downloader script

: 79yTuFL.png (19.48 KiB) Viewed 8669 times

This is the download all script

: HRaO90b.png (59.62 KiB) Viewed 8669 times

Alright, so a few things to note

1. everything else works perfect. samba, pinging to external addresses. I get an IP and everything looks good on my router

2. does not change with ethernet, wifi, ethernet straight into the wall (my isp is fiber with the ethernet port running to their switch stack)

3. the download script works when i use wifi connected my phone's hotspot, connected to cellular data. this does not work when my phone hotspot is connected to my router, leading me to believe it's an isp issue.

4. another one time workaround i first used... at first i could not use the download script to get anything. but after changing my mac from the default 01:02:03:04:05:06:07 using this https://pastebin.com/raw/8L6kg99Q, it worked! until... i rebooted. then it did not work again connect to my isp internet, even after changing the mac again. thinking this was a fluke, i reinstalled and started from scratch. and the same behavior exhibited itself again.

I'm at a bit of a loss! I'm guessing it's something with my ISP and not the mister, but I'm not sure exactly what I should say to my ISP. any extra info I could give them?

Posted: **Sun Jun 12, 2022 8:31 am**

Does using a VPN have any difference?

Posted: **Sun Jun 12, 2022 10:34 am**

I'm guessing an IP conflict in your LAN.

Posted: **Sun Jun 12, 2022 2:05 pm**

Bas wrote: ↑Sun Jun 12, 2022 10:34 am I'm guessing an IP conflict in your LAN.

nah I don't think so. I've double checked my dhcp leases, no conflict. and samba and external pings still work!

Bunker wrote: ↑Sun Jun 12, 2022 8:31 am Does using a VPN have any difference?

alright! this does work! I put a vpn on my phone and connected my mister to the phone with a wifi hotspot. I made sure to turn off mobile data, only going through my router. I don't know much of anything about external networking... does anyone know why it might work on a VPN?

Posted: **Sun Jun 12, 2022 2:44 pm**

Ah I see the screenshot now. Check the actual error first. Is cacert.pem ok on your system?

Posted: **Sun Jun 12, 2022 4:47 pm**

Bas wrote: ↑Sun Jun 12, 2022 2:44 pm Ah I see the screenshot now. Check the actual error first. Is cacert.pem ok on your system?

seems to be! I looked at it in vim via ssh and... that seems? like a cert file lol. so what I did was delete it and grab a new one using

Code: Select all

cd /etc/ssl/certs
wget --no-check-certificate https://curl.haxx.se/ca/cacert.pem

Same behavior. Works using cellular and using a vpn through my router. But just straight ethernet to my router, wifi to my router, or phone hotspot hooked up to my router, or ethernet straight into the wall, no dice.

Posted: **Sun Jun 12, 2022 5:36 pm**

does DNS work properly?

is there a firewall in your path to the internet ?

Posted: **Sun Jun 12, 2022 6:08 pm**

held wrote: ↑Sun Jun 12, 2022 5:36 pm does DNS work properly?

is there a firewall in your path to the internet?

If I can ping external addresses from the mister like google.com, and github.com, I assume DNS is working properly?

Just now I tried disabling something called SPI firewall on my router, didn't seem to change anything. And I think when I plug my mister straight into the wall and get an external IP... there isn't a firewall on that is there? and would a VPN somehow bypass it? (I'm pretty shaky on network stuff, once it leaves my router, I really don't know what happens lol)

Posted: **Sun Jun 12, 2022 6:09 pm**

I just wanna say too, I appreciate everyone's help and suggestions so far! :)

Posted: **Sun Jun 12, 2022 6:15 pm**

You plug straight into the wall and get an external IP!? That's weird, but also bad in this case. That means you're doing two things:

1. Hooking up your MiSTer to the internet, fir which it's thoroughly unsuited with its default SSH root login and all.

2. Messing with your ISP's upstream router. That device won't appreciate you changing MAC addresses around all the time.

Get a NAT router in between there, that'll help a lot.

But just to check, what external IP do you get? Just the first octet would be helpful, leave the rest out for now for security.

Posted: **Sun Jun 12, 2022 6:25 pm**

So I live in apartment with fiber (after 10 years of DSL I earned it lol), so you plug a router into the wall via ethernet, bypassing the need for a modem. I do need to stress, I only plug my mister into the wall for testing this issue, a router sits between my devices 99% of the time.

Normally my external facing IP first octet is 72 (just googling 'my ip'). When I plug my mister into the wall, the first octet is 208

Posted: **Sun Jun 12, 2022 8:21 pm**

Both of those are publicly routable. This may be indicative of the upstream device putting your port on a different VLAN based on the MAC address it reports. Seeing as your uplink is very non-standard and the MiSTer behaving somewhat poorly on layer 2 of the OSI model, a NAT box between at least it and your uplink would probably bring a lot more stability.

More generally speaking, a fast enough NAT router for the entire network would probably be best. You'd completely insulate your local LAN from whatever is going on upstream. Unless you're seriously geeking out in the networking department, that would be my advice. Things would then work like they do in 99.9% of homes and MiSTer would just naturally fit in there.

Now this *may* not be possible due to the way the network is configured and you might lose things like VOIP or IPTV if you use those services. Get in touch with your ISP if you do.

I'd be happy to help out with this further (I used to do datacenter networking for a university), but there's quite a few unknowns here still.

Posted: **Sun Jun 12, 2022 9:22 pm**

Don't I have a NAT router already? I should clarify, the IP my mister gets when I plug into my router, normal operation, is 192.x.x.x, an internal IP like normal. The 72 I was talking about is my external address that my router receives, and what my computer is seen as from the perspective of a website (ie googling 'whats my ip'). This really doesn't have too much to do with the issue, but I'm curious why my ISP consistently gives my router 72 and the mister 208. I suppose it's going off of the device type via the mac address?

Let me know if I'm asking a silly question or misunderstanding something, I really appreciate the help!

Posted: **Mon Jun 13, 2022 11:02 am**

Ah yes, then you do have a NAT router. But the MiSTer should also "just work" from behind that device, as long as it gets a locally unique 192.168.xxx.xxx address. This could only really go wrong if another device on the same segment uses the MiSTer's MAC address, like a second MiSTer on the same network could do.

Plugging the MiSTer into the wall directly will only work if you set its MAC address to be the same as your router's public interface. In that case the MiSTer device will simply impersonate the router to the upstream switch and you'll be put into the same upstream VLAN directly. As long as you're not using PPPoE or Radius/802.1x authentication before you get a connection, chances are quite high that the ISP determines your device from its MAC address and puts it in a VLAN based on that. But let's not go there. It's unsafe to give a MiSTer a publically routable IP with no firewalling in front of it. Stick with your router.

From behind your NAT router, given that you have a working 192.168.xxx.xxx address on the MiSTer, you should be able to at least connect locally. You confirmed that as connectivity to your CIFS shares apparently does work and remote connectivity is also there. So what remains for me as a logical culprit would be DNS. Can the MiSTer, once connected behind the NAT router, resolve a hostname that it doesn't know about yet? You can test that with ping. It may fail the actual ping because ICMP tends to get blocked a lot on the internet, but at least it should give you an IP address for the other side that it's sending packets to. If it gets to that stage, DNS works.

Looking into the script itself.. What I'm seeing the downloader script do, is pull the cacert.pem file from a hard-coded location: https://curl.se/ca/cacert.pem. That URL is behind a CDN from Fastly so the target IP changes all the time but it should be nothing special, just a plain HTTPS-request. I can even download that file from work, with their strict firewall/security crud going on. Do make sure that the file you pull from there is unmangled by things like Windows line endings, character encoding fudgery etc. if you download it on another computer than the MiSTer itself. The easiest way to prevent this is to cd into /etc/ssl/certs on your MiSter itself and then wget the file directly.

The script uses that file to validate the upstream URL's it connects to and the errors you're seeing are the script complaining that A) the cacert.pem file is somehow corrupt and B) it's unable to correct the situation.

What it tries to do at step B, is simply delete the file. Since you're running as root, permissions can't be the issue here. The script, however, is not smart enough to deal with missing directories. Are you 100% positive that /etc/ssl/certs is an existing directory on your MiSTer and that the cacert.pem file is in there? Couldn't hurt to limit permissions to 755 on the file itself and change the owner to root since some tools tend to check on that (even if redundant on MiSter).

I don't have my MiSTer handy right now (at work.. lunch break) but I'll see if I can reproduce the issue on mine tonight and what's needed to fix it.

Posted: **Mon Jun 13, 2022 7:11 pm**

so on the dns front, everything seems to be working as far as I can tell. I pinged all these addresses and the pings returned fine
8.8.8.8
google.com
github.com
curl.se

But! I think I'm making some progress on the culprit.
So I decided to just delete the cacert file and run the download script (not downloader). And I get this:

: LMLki7n.png (39.45 KiB) Viewed 2031 times

Now I can say no and run the downloader without the cert and it works. I'm not knowledgeable enough to know why, but I'm knowledgeable enough to know I should avoid that as much as I can.
If I say yes, it downloads and installs cacert and... then the message pops up again. and if i say yes, it downloads and... pops up again. repeat
so I type ls again in /etc/ssl/certs to see whats up and oh wow. way more files than there was before! before cacert.pem was the only file in that directory.

: ZOFwz6I.png (77.07 KiB) Viewed 2031 times

(it goes down for a while, couldn't capture it all in a single screenshot)

I decide to delete all the files in the directory and run the separate download all script. And it ran fine, without creating any new cert files. This is outside my expertise, but that seems concerning? That the update all script either a. doesn't check to see if the certs are there or b. doesn't use certs at all? I also ran the download all script after letting the normal download script create certs, and it ran the same (seemingly perfectly).

With the certs that the download script created, I tried using it on a VPN. Same behavior! Thought the certs it just installed were corrupt and immediately wanted to try again.

So I decided delete all the files again and just run wget --no-check-certificate https://curl.haxx.se/ca/cacert.pem and same behavior as before, internet connection failed.

So... I delete the file again, run JUST wget https://curl.haxx.se/ca/cacert.pem. It says the cert isn't activated, and the download script treats it as if it wasn't there and tries to create its own.

So it's something with these certs. I'm just at a loss why:

getting a cert via wget --no-check-certificate https://curl.haxx.se/ca/cacert.pem works on cellular and a vpn, but not my own network
why the download script installs many many files instead of just one cacert.pem, and why it still thinks its corrupt after install
why the download all script runs seemingly perfectly after deleting the cert.

Let me know if you have any questions about what I wrote! I was typing it up while troubleshooting and may not have made everything clear. again, thank you very much for helping me out with this

Posted: **Mon Jun 13, 2022 8:38 pm**

What does running "ip addr show" display? Do you have two IP addresses on one interface, and is it an APIPA address? (e.g. 169.254.xxx.xxx etc...)

If you have more than one then you need to flush the interfaces with "ip addr flush up".

Posted: **Mon Jun 13, 2022 8:51 pm**

aberu wrote: ↑Mon Jun 13, 2022 8:38 pm
What does running "ip addr show" display? Do you have two IP addresses on one interface, and is it an APIPA address? (e.g. 169.254.xxx.xxx etc...)

If you have more than one then you need to flush the interfaces with "ip addr flush up".

: MiHfylj.png (17.52 KiB) Viewed 2030 times

I think that's normal? the 0.150 address is the permanent lease I set in my router's DHCP server.

Posted: **Mon Jun 13, 2022 10:53 pm**

Yup looks normal. Just wanted to make sure.

Posted: **Tue Jun 14, 2022 12:36 am**

If it's the cert that's giving you trouble, maybe your system date is wrong? The Mister will set its own date about thirty seconds after coming up, but if you reboot and then *immediately* run the update script, any date checks on the cert would probably fail.

Basically, after rebooting, wait until the time appears in the title bar, and see if the behavior changes.

Posted: **Tue Jun 14, 2022 7:33 am**

That whole bunch of files in /etc/ssl/certs is by design. Those are symlinks to trusted CA certificates elsewhere on your system. I never needed to touch them on my MiSTer yet, but some background is here: https://www.openssl.org/docs/man1.1.1/man1/rehash.html

Malor's point is also a great one: the system date must be correct(-ish) for certificates to work. MiSTer doesn't have a battery-backed clock so it'll always be January 1970 right after you turn it on. Give it some time to do an NTP sync and skip back to the present day before you attempt an update.

Posted: **Tue Jun 14, 2022 7:55 am**

Is your MAC address different or the same each to time you reboot?
If its different, you might want to set it via the "scripts" menu.

Posted: **Wed Jun 15, 2022 3:09 pm**

Here's a thought... Undo the static DHCP lease for your MiSTer in your router. then reboot your router. After the Router comes all the way up, reboot your MiSTer. Wait about a minute after it comes up, and try again.

Posted: **Tue Jul 11, 2023 3:03 pm**

Hey milkdrinker, did you ever find why this was happening? I am going through the same problem and deleting the file also seems to have helped. I did not have this problem when I had my MiSTer connected over ethernet, but I do now over Wi-Fi.

My money is on some sort of MTU issue, hence why the different results with different connections, but I am yet to run more testing and first I have to wait for the Downloader to finish

Posted: **Thu Jul 13, 2023 1:05 am**

so unfortunately... i never found the cause.

what i ended up doing is starting over from scratch on a new sd card. its been a year so im fuzzy on the details... but the download all script worked great and i just used that for everything. i tested it this morning and the regular download script still doesn't work. let me know if you want me to test anything for you!

MiSTer FPGA Forum

Odd Issue with my MiSTer and ISP?

Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?

Re: Odd Issue with my MiSTer and ISP?