archive.org folder synchronization script [0mhz-dos]

[blktiger]

Hey folks, I hacked this up today for my own use. It will synchronize any item folder @ archive.org (default is 0mhz-dos) with a local directory. Unlike the other download script I saw mentioned here, it doesn't need to parse XML, works for any archive.org folder not just 0mhz, does not use the 0mhz git repository, but instead fetches the current state of the archive.org folder -through the JSON API- then calculates what has changed (file size, mtime, md5 hash) and proceeds to synchronize. It can also hash all the files and check for corruption.

https://github.com/atomontage/aorg-sync

[HerrBerzerk]

Is this to execute on the Mister or on a desktop? And does it synchronize to the local folder where the script is or can I configure a folder?
And what does "# Change these to GNU on macOS as BSD versions will not do" mean?

Sorry, might be very basic questions, but I've never written a script.

[blktiger]

Is this to execute on the Mister or on a desktop?

You can do both, the Mister Linux distribution comes with all the utilities the script needs already installed.

And does it synchronize to the local folder where the script is or can I configure a folder?

It will synchronize to the local directory you execute the script from. This doesn't need to be the same directory where you have the script installed. Example:

Code: Select all

cp aorg-sync.sh /media/fat/Scripts
mkdir -p /media/fat/games/AO486/0mhz && cd /media/fat/games/AO486/0mhz
/media/fat/Scripts/aorg-sync.sh # Will proceed to sync into /media/fat/games/AO486/0mhz

And what does "# Change these to GNU on macOS as BSD versions will not do" mean?

If you're running the script on macOS (or another BSD distribution), you'll need to install GNU coreutils (e.g. from macports or homebrew) as the BSD versions of these utilities do not support the same features. This does no affect executing on Mister or Linux.

[SirBeers]

Thanks for this script! I am getting this error though:

Code: Select all

curl: (60) SSL certificate problem: self signed certificate in certificate chain

[blktiger]

Thanks for this script! I am getting this error though:

Code: Select all

curl: (60) SSL certificate problem: self signed certificate in certificate chain

Follow the directions here to install missing certificates on your Mister, or run the script (I just updated it) with --no-check-cert.

If you are running it on a Linux distribution other than Mister, there's usually a package you can install with common CA certificates e.g. on Debian it's ca-certificates.

[HerrBerzerk]

You can do both, the Mister Linux distribution comes with all the utilities the script needs already installed.

And does it synchronize to the local folder where the script is or can I configure a folder?

It will synchronize to the local directory you execute the script from. This doesn't need to be the same directory where you have the script installed. Example:

Code: Select all

cp aorg-sync.sh /media/fat/Scripts
mkdir -p /media/fat/games/AO486/0mhz && cd /media/fat/games/AO486/0mhz
/media/fat/Scripts/aorg-sync.sh # Will proceed to sync into /media/fat/games/AO486/0mhz

Thank you very much, I understand now.

[SirBeers]

Sorry I'm having trouble with the REMOTE= option. I tried:

Code: Select all

/media/fat/Scripts/aorg-sync.sh --no-check-cert REMOTE=TGODFloppyCollection

And the script still downloaded everything from the 0mhz-dos link.

[docbobo]

If you are overriding environment variables when starting a script, they should go before the script. That's also how it's document on the GitHub page, as far as I can tell.

So the following should theoretically work. Haven't tested it though.

Code: Select all

REMOTE=TGODFloppyCollection /media/fat/Scripts/aorg-sync.sh --no-check-cert

[SirBeers]

OHHH so that's how that works. Thanks!

[SirBeers]

To get it to work with a different archive I had to change this line in the script to allow curl to create directories.

Code: Select all

CURL_ARGS=(--create-dirs -RSLf#)