NTP Not Updating Automatically

[jlancaster86]

I've noticed recently that "update_all.sh" has been failing for me, and I realized today that it's because the clock is resetting to Unix epoch, breaking HTTPS.

My MiSTer is refusing to update the time via NTP unless I run rtc.sh manually every time I boot. I've tried automating it with startup scripts (in "/etc/network/if-up.d/") but it just never works (and yet the scripts will work if run manually). Am I missing something?

On a related note, rtc.sh appears to be broken now. The default NTP server ("0.pool.ntp.org") no longer responds to ping, so the script will always fail. Either the NTP server has to be set to something that will respond to ping (such as "2.pool.ntp.org"), or the ping check needs to be removed/commented out.

[blacklistedcard]

I've noticed recently that "update_all.sh" has been failing for me, and I realized today that it's because the clock is resetting to Unix epoch, breaking HTTPS.

My MiSTer is refusing to update the time via NTP unless I run rtc.sh manually every time I boot. I've tried automating it with startup scripts (in "/etc/network/if-up.d/") but it just never works (and yet the scripts will work if run manually). Am I missing something?

On a related note, rtc.sh appears to be broken now. The default NTP server ("0.pool.ntp.org") no longer responds to ping, so the script will always fail. Either the NTP server has to be set to something that will respond to ping (such as "2.pool.ntp.org"), or the ping check needs to be removed/commented out.

I can ping that ntp server fine?

Pinging 0.pool.ntp.org [162.159.200.123] with 32 bytes of data:
Reply from 162.159.200.123: bytes=32 time=35ms TTL=54
Reply from 162.159.200.123: bytes=32 time=36ms TTL=54
Reply from 162.159.200.123: bytes=32 time=36ms TTL=54
Reply from 162.159.200.123: bytes=32 time=37ms TTL=54

Ping statistics for 162.159.200.123:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 35ms, Maximum = 37ms, Average = 36ms
PS C:\WINDOWS\system32>

[Bas]

Those pool.org servers change all the time, they're an aggregation of many different servers living behind a common DNS name. Ping as a connectivity check is sadly quite broken on the internet these days. With NTP being a UDP-based service you generally just hope for the best as a client so you attempt to get a response from more than 1 server.

Many routers permit you to set up the IP addresses of NTP servers to pass on to clients. That way the MiSTer could be passed a good set that works for you without disappearing again after every Linux update to the base system.

[blacklistedcard]

I just checked. The rtc.sh script is running fine for me.

/media/fat/Scripts# ./rtc.sh
Syncing date and time with
0.ca.pool.ntp.org
Date and time is:
Sun 19 Dec 2021 02:18:09 PM EST
RTC set.
/media/fat/Scripts#

[blacklistedcard]

I just checked. The rtc.sh script is running fine for me.

/media/fat/Scripts# ./rtc.sh
Syncing date and time with
0.ca.pool.ntp.org
Date and time is:
Sun 19 Dec 2021 02:18:09 PM EST
RTC set.
/media/fat/Scripts#

oppsss. I ran it with the other server reference. Using 0.pool.ntp.org causes the script to sometimes fail. So I used another server reference.

/media/fat/Scripts# ./rtc.sh
No Internet connection
/media/fat/Scripts#

/media/fat/Scripts# ./rtc.sh
Syncing date and time with
0.pool.ntp.org
Date and time is:
Sun 19 Dec 2021 02:22:24 PM EST
RTC set.
/media/fat/Scripts#

[jlancaster86]

To get things back on track: The main point of my initial post is that the clock will not update automatically at startup, no matter what I do with startup scripts. The issue with rtc.sh failing to ping "0.pool.ntp.org" is just a minor bug, and I already mentioned two ways to fix that.

It seems that ntpdate just won't work from a startup script now, and for no apparent reason. It works just fine when run via a script manually (and via SSH), but apparently does nothing when run via a startup script. I've verified the startup scripts are actually firing by calling reboot, which predictably causes a reboot loop.

[Bas]

It'll fail if it runs before the MiSTer has an IP.

[jlancaster86]

It'll fail if it runs before the MiSTer has an IP.

I thought I'd already accounted for that, but it turns out that's exactly what it was in the end. I had been delaying the script by 5 seconds, thinking that was plenty of time for the Ethernet interface to fully come up (it wasn't, obviously). Delaying it by 10 seconds was all it took to get it working.

[mightor]

Can you please give us some more details about how you automated this ?
By just copying the rtc.sh script in /etc/network/if-up.d/ ?
How did you manage to delay for 10 seconds ? By changing the content in the script ?

[jlancaster86]

Can you please give us some more details about how you automated this ?
By just copying the rtc.sh script in /etc/network/if-up.d/ ?
How did you manage to delay for 10 seconds ? By changing the content in the script ?

First, create a script in the Scripts folder. I named mine ntp.sh:

Code: Select all

#!/bin/bash

NTP_SERVER="0.pool.ntp.org"
i=0

while [ $i -lt 3 ]
do
	echo "Waiting 10 seconds..."
	sleep 10s
	if ! ntpdate -bsu $NTP_SERVER
	then
		if $i == 2
		then
			echo echo "Unable to sync after 3 attempts."
		fi
		((i++))
	else
		echo "Date and time is:"
		echo "$(date)"
		i=3
	fi
done

Then, add a line to /media/fat/linux/user-startup.sh to run it at startup:

Code: Select all

sh /media/fat/Scripts/ntp.sh &

[mightor]

Thank you !

[mightor]

The ntp.sh script works well when executed manually through the interface menu of the MiSTer.
But it does not execute automatically. To be sure that the boot process has enough time to enable the ethernet connexion, I increased the pause from 10 to 20 seconds.
The ntp.sh script also has execute permissions.
Please note that the name of the startup script is "_user-startup.sh"

[jlancaster86]

The ntp.sh script works well when executed manually through the interface menu of the MiSTer.
But it does not execute automatically. To be sure that the boot process has enough time to enable the ethernet connexion, I increased the pause from 10 to 20 seconds.
The ntp.sh script also has execute permissions.
Please note that the name of the startup script is "_user-startup.sh"

You need to rename _user-startup.sh to user-startup.sh for it to run at startup.

I forgot about the execute permission–good point!

[mightor]

Perfect, it works perfectly.
Like you, I've noticed that "update_all.sh" has been failing for me, and I realized after a lot of research that it's because the clock is resetting to Unix epoch, breaking HTTPS.
After some time spent on lauching "rtc.sh" manually I wanted to automate this and found your thread. Thanks for sharing.

[Bas]

The updaters should actually guard against this by refusing to run when a date prior to the script's own release date is detected, maybe overridable with a flag for those who they know what they're doing. Currently things are very fragile.

[jlancaster86]

It's not just the update scripts that need the correct time, though. The microcomputer cores and certain cartridge games will be screwed up if the system time is wrong. And savegame files will have the wrong timestamps.

I find it odd that so few users seem to have run into this problem. Do most users have physical RTC modules, or is there something else going on?

[rhester72]

No physical RTC here and very literally never had the problem described after nearly 2 years.

[mightor]

No physical RTC here and very literally never had the problem described after nearly 2 years.

Know what ? I bought my MiSTer from Manuferhi, which came with a pre-configured SD card. With this one, no problem: as soon as the ethernet connexion is up (logo displayed in menu bar), date and time are set.

I only have problem with my two SD card which were configured using Mr Fusion setup (the recommended method). This may be the begenning of an explanation...

[jlancaster86]

Mine was working fine for almost two years. I may have reinstalled with Mr Fusion prior to it breaking, but I can't be sure. I honestly don't know how long it was broken for before I noticed.

I might try some older Mr Fusion images over the weekend and see if I can narrow things down a bit.

[Bas]

Running ntpdate as a one-shot operation sometime during the bootup is hardly resilient on a device that won't be guaranteed to have connectivity at that point in time. A service like Chrony may work better as it'll be persistent and correct the clock whenever connectivity recovers. Setting the makestep parameter in the config to 1 -1 should allow for an instant leap from epoch to the current time. If scripts guard themselves against an obviously wrong clock that would fix this whole issue. There are far too many external dependencies for ntpdate to work reliably in this situation. If it does, you're mostly just lucky.

[jlancaster86]

The Main core throws an icon up on the screen once the network connection goes live. Wouldn't it make sense for it to trigger ntpdate at the same time?

[mightor]

If scripts guard themselves against an obviously wrong clock that would fix this whole issue. There are far too many external dependencies for ntpdate to work reliably in this situation. If it does, you're mostly just lucky.

I agree. The working MiSTer config I have on my first SD card is the proof that it must just work out of the box. Now we have to find out what changed in more recent releases.

RSetting the makestep parameter in the config to 1 -1 should allow for an instant leap from epoch to the current time.

In which config file do you set makestep to 1 -1 ?

I tried a fresh install using the manual method with MiSTer SD card installer. Same problem with date/time.

[Bas]

Nothing needs to change between releases for this to intermittently fail. Running ntpdate like this depends on many eventualities on the local network like DHCP, DNS, any upstream firewalls, the specific NTP server that the pool throws at you etc.

On the side of the MiSTer things are currently not set up to deal with any of these. Integrating Chrony would help with that, but that's not the current state of things.

Ntpdate as a one-shot with no error handling or fallback is bad engineering.

[mightor]

I'm not speaking about something that intermittently fail: the MiSTer setup I have received from Manuferhi allways set the date/time correctly at boot time.
The MiSTer setup I have done myself with more recent releases using mr fusion or manual method give me the problem with date/time not beeing initialized, at every boot.

Here are some some useful informations:
I received my MiSTer in october 2021. I immediately set up a new SD card using mr fusion 2.3 and starting from there I regularly updated with the update_all script. This is the version in which I found the problem with rtc/ntp.
I recently set up a new SD card mith mr fusion 2.4 (latest at this time): same problem.
Today I've just set up a new SD card with mr fusion 2.2. And guess what: no problem with rtc/ntp !

Next step: update the MiSTer starting from this version of mr fusion 2.2 and see what happens with rtc/ntp.

[jlancaster86]

I'm in the same boat as mightor. Till around December of last year, the time had always updated correctly at startup. But at some point that broke, and it would always fail. It was never an intermittent problem.

It appears mightor and I are both using Ethernet too, so it can't be blamed on a dodgy Wi-Fi adaptor/connection.

Whatever built-in mechanism there is for updating the time at startup just plain isn't working any longer, and that's why we're having to use our own startup scripts to compensate.

[mightor]

Well well well. After an "update_all" from the fresh install of mr fusion 2.2, date/time is broken. Again.

[jlancaster86]

Well well well. After an "update_all" from the fresh install of mr fusion 2.2, date/time is broken. Again.

Are you able to roll back just the menu.rbf to the one in Mr. Fusion 2.2? That'll help to narrow it down to either the Menu core or the Linux installation.

[mightor]

Are you able to roll back just the Menu .rbf to the one in Mr. Fusion 2.2? That'll help to narrow it down to either the Menu core or the Linux installation.

I can do that in a next troubleshoot step but I'm pretty sure it's the Linux installation and not the menu.rbf nor MiSTer.rbf.
I'm currently testing an update using the included update.sh script instead or "update_all", starting from a fresh install of mr fusion 2.2. Everything out of the box.

[jlancaster86]

Are you able to roll back just the Menu .rbf to the one in Mr. Fusion 2.2? That'll help to narrow it down to either the Menu core or the Linux installation.

I can do that in a next troubleshoot step but I'm pretty sure it's the Linux installation and not the menu.rbf nor MiSTer.rbf.
I'm currently testing an update using the included update.sh script instead or "update_all", starting from a fresh install of mr fusion 2.2. Everything out of the box.

Yeah, I just tried rolling back to menu_20210315.rbf, and the clock isn't updating. Looks like it's something in the Linux installation.

[mightor]

I'm currently testing an update using the included update.sh script instead or "update_all", starting from a fresh install of mr fusion 2.2. Everything out of the box.

Date/Time broken after running update.sh over a fresh installation of mr fusion 2.2.
menu core hasn't been updated since march 2021. It has nothing to deal with this problem.
MiSTer Core rolled back to may 2021: does not restore date/time problem.

Starting back from a fresh mr fusion 2.2 install, I tried to only update the MiSTer.rbf core. The most recent which is compatibe with OSv200908 is 20211014. No problem at all, date/time still good with this version.

So the problem may have been introduced in the latest Linux versions. Keeping old Linux version is not an option as modern cores needs up to date Linux version. So while waiting for a fix, there are only two ways to get date/time at each boot: by manually running rtc.sh or by using your automated method to run a ntp.sh script at boot time.