DOSContainer Alpha Release

[Bas]

I'm taking a small detour now. The current state of DOSContainer allows me to package just about anything from the dawn of time until about 1985. That's where I'll start, and I want the process be friendly to outside contribution. So in short:

1. I'm completing the zipfile downloader feature on DOSContainer to enable the layered build architecture that I envisioned from the start.
2. Set up robust and fast hosting for everything. (done)
3. Create a friendly webform for contributed games and YAML's.
4. Publish tested and validated games on a new DOSContainer website.
5. Put DOSContainer itself behind a web form. Upload your YAML, it spits out a VHD just for you. (stretch goal)

Once that flow is up, I'm starting in 1981 and working my way up through history. Implementing DOS versions and FAT variations follows as I reach the games for which they are period appropriate. Starting the packaging early helps me learn and figure out optimal workflow.

Another feature that I feel is a hard requirement when games become configurable themselves (1985-ish), is a "pull" option. This means you run a VHD on MiSTer and have a shell open at the same time. As you make the needed changes in the DOS environment to set up all the different hardware settings, you run a simple "pull" command on the VHD every time to extract a delta that captures only the config changes you made in DOS (yes, I think I can detect open/uncommitted files with reasonable confidence). Those changes then become a layer in the YAML. Once done, the tool wraps the whole lot up in a zipfile and a YAML manifest. The goal: instantly allow custom builds that mix and match all settings for the game itself where it makes sense. Just tweak the YAML to suit your wishes, or just build VHD's for every permutation you can think of in the blink of an eye. This should fix the huge amount of manual toil to configure each individual game.

By the way: this is all a hobby and I only have a few hours each week for it. I do welcome any help I can get in this! If you know Rust, awesome! If you're good with AWS serverless code: please raise your hand. If you want to design a website, logo or anything else: yay! I am very partial to static hosting with things like Hugo due to the low maintenance and no security/maintenance headaches like WordPress would bring.

Another option that I'm very open to, is to join with the 0Mhz effort and provide the plumbing for that. My own scope is wider than MiSTer itself.

[AmintaMister]

I'm taking a small detour now. The current state of DOSContainer allows me to package just about anything from the dawn of time until about 1985. That's where I'll start, and I want the process be friendly to outside contribution. So in short:

1. I'm completing the zipfile downloader feature on DOSContainer to enable the layered build architecture that I envisioned from the start.
2. Set up robust and fast hosting for everything. (done)
3. Create a friendly webform for contributed games and YAML's.
4. Publish tested and validated games on a new DOSContainer website.
5. Put DOSContainer itself behind a web form. Upload your YAML, it spits out a VHD just for you. (stretch goal)

Once that flow is up, I'm starting in 1981 and working my way up through history. Implementing DOS versions and FAT variations follows as I reach the games for which they are period appropriate. Starting the packaging early helps me learn and figure out optimal workflow.

Another feature that I feel is a hard requirement when games become configurable themselves (1985-ish), is a "pull" option. This means you run a VHD on MiSTer and have a shell open at the same time. As you make the needed changes in the DOS environment to set up all the different hardware settings, you run a simple "pull" command on the VHD every time to extract a delta that captures only the config changes you made in DOS (yes, I think I can detect open/uncommitted files with reasonable confidence). Those changes then become a layer in the YAML. Once done, the tool wraps the whole lot up in a zipfile and a YAML manifest. The goal: instantly allow custom builds that mix and match all settings for the game itself where it makes sense. Just tweak the YAML to suit your wishes, or just build VHD's for every permutation you can think of in the blink of an eye. This should fix the huge amount of manual toil to configure each individual game.

By the way: this is all a hobby and I only have a few hours each week for it. I do welcome any help I can get in this! If you know Rust, awesome! If you're good with AWS serverless code: please raise your hand. If you want to design a website, logo or anything else: yay! I am very partial to static hosting with things like Hugo due to the low maintenance and no security/maintenance headaches like WordPress would bring.

Another option that I'm very open to, is to join with the 0Mhz effort and provide the plumbing for that. My own scope is wider than MiSTer itself.

This is an extraordinary effort, a perfect strategy, and a fundamental openess to collaboration, THANKS!

[Bas]

Build 64 actually builds a working (in PCem, MiSTer untested) bootable Alley Cat VHD based on IBM PC DOS 2.00.

This build is a partly hardcoded proof of concept that should prove a number of items. I would VERY MUCH appreciate some testing of the below functionality:

• Does this even run properly on Windows at all? I don't know, I don't have Windows.
• Does the ARMv7 build run properly on the MiSTer? (stretch goal: 32-bit Raspberry Pi's should work in theory.. test please?)
• This is the first build with the ZIP downloader/extraction code hooked into the build phase.
• The downloader pulls files in over HTTPS from my under heavy construction AWS-based hosting location.

So please, anyone adventurous, have a look at build 64 and see how it goes. Use the YAML I posted above as an input file. The result should be alleycat.vhd which should work as advertised.

WARNING: This build contains a command to delete a single file. What it does is it downloads the zip file for Alley Cat, extracts that to a platform-appropriate temp location and then deletes the zip file. While this should all be very safe (I'm using standard-fare Rust crates for the handling of tempfiles/dirs, my own tests on Linux turned out fine) I can not guarantee anything here. Do NOT test this build on the master control station of your in-house nuclear reactor please, or on any other system you are particularly attached to. Chances of anything going badly wrong are practically non-existent, but I do feel I should warn you about being a guinea pig.

To satisfy some people's curiosity. The binaries for IBM DOS are currently still compiled as raw bytes into the executable itself. It was done this way because I didn't have a working downloader yet. When this build proves stable, my next step will be to remove these embedded files and make them into downloads just like the game files. When that's done you'll start seeing multiple progress bars as a VHD-build proceeds.

I do like how going from zero to working bootable VHD including this tiny game's download is done in 0.16 seconds on my laptop. This opens the possiblity of stuffing DOSContainer into an AWS Lambda function eventually.

[rcblanke]

Hi Bas,
Appears to be working fine on Win11:

win11 output

doscontainer.png (108.95 KiB) Viewed 12520 times

though as you can see leaving out the .yml extension results in a funky message. Please note that I haven't tried to actually run the vhd on PCem or MiSTer.

output alleycat.vhd:
MD5 9cb6b2bb9d315726a4b6ab7de5a401a1
SHA 0fffca953b3243063b4059c0cd09a35ad4ba173f

Regards,
Ronald

[NML32]

Confirmed working on Windows 10
Used doscontainer.exe windows_x86_64

[Bas]

Thanks for your testing! The error messages are funky because you're looking at Rust passing the error down the chain and giving up because I didn't yet write a nice formatter for it yet. That's my bad, but mostly cosmetic. You giving me checksums makes a huge amount of sense, but I just realized they aren't meaningful yet. Files are dropped into the VHD with the current system date/time stamped on them so every VHD will in fact be slightly different. I hadn't thought of that yet (just needed some kind of date/time to work and even that is wrong right now). I'll pick that up as an issue and use the source files' timestamps so builds become reproducible and actually correct.

[NML32]

I get this error when running from MiSTer

root@MiSTer:/media/fat/doscontainer # ./doscontainer build allycat.yaml
./doscontainer: /lib/libc.so.6: version GLIBC_2.32' not found (required by ./doscontainer)
./doscontainer: /lib/libc.so.6: versionGLIBC_2.33' not found (required by ./doscontainer)
./doscontainer: /lib/libc.so.6: version `GLIBC_2.34' not found (required by ./doscontainer)

[NML32]

Success with build #68 creating allycat.vhd from MiSTer FPGA.

https://youtu.be/4ghmIWLhk_M

[Bas]

Wow, that was fast! I changed the build target over from glibc to musl, that helps apparently. Impressed with the speed of this thing on dinky MiSTer ARM!!

Last build for today will probably be 72 if it succeeds in about 15 minutes or so, after that it's back to work for a whole week again so not sure how much time I'll have to keep going. The biggest hurdle now is to support subdirectory's on the VHD. I did clean up and expand the documentation, added a section on the updated YAML format. I also do fully intend to support the Bash version's YAML format as well, but things may break there for a while. I will port over any games from the old version to the new though.

Change log lives here: https://code.area536.com/DOSContainers/ ... ranch/main

[Bas]

Just a quick update on progress. Lots of regression as I'm generalizing the method of writing files/dirs to the file system. So while it seemed that build 64 was really close, I had to rip out ever growing parts of the file system code to allow for the general pattern of: extract this ZIP file to a temp location, then copy everything from that location over to the VHD recursively.

Why is this difficult? Mainly because ancient DOS versions have divergent ideas on how to write the FAT's root directory versus regular subdirectories. Flexibility in the core code is required but that runs the risk of creating a ball of spaghetti.

Writing a file on FAT entails allocating clusters for it, checking and creating the correct directory entry for it, and then writing the bytes to the previously allocated clusters in the data area of the partition/volume. What I'm seeing now is slight fluctuations on where the data area is supposed to actually begin.

Now I remember the rumors back in the day of how Commodore and Olivetti supposedly weren't exactly IBM compatible but I never got more than amecdotal hearsay on those. As this project progresses I get to see a lot of the icky underbelly of how MS-DOS was marketed before MS released its own builds directly. Great stuff for a blog series, but very annoying right now.

[limi]

[Bas]

Interesting. This looks a lot more complete than the previous code, which was a steaming mess of versions all mixed together.

[RealLarry]

If anyone is interested: MS published their MS-DOS v4.0 on GitHub.

[Bas]

Build 92 just came down the pipes. It's functionally almost identical to build 64 from earlier. However:

• The YAML format changed and is now far more sensible and more in line with what is in the enclosed manual.
• The OS files now also get downloaded instead of being hardcoded into the binary.
• The downloader fully works and is now wired into the application properly. Use your own URL's if you want, any ZIP over http(s) is fair game. No more depending on my infra to package your games*.
• Lots of cleanups and restructuring that enables the next steps.

The path is now clear for me to do two absolutely crucial things:

• I̶m̶p̶l̶e̶m̶e̶n̶t̶ s̶u̶p̶p̶o̶r̶t̶ f̶o̶r̶ s̶u̶b̶d̶i̶r̶e̶c̶t̶o̶r̶i̶e̶s̶. Done!
• Implement support for FAT16 and FAT16B

So what can you do with this already? Well.. package really simple games like Alley Cat, that do not require subdirs to work. Everything now just gets plonked into C:\ and can run from there if you give it your own AUTOEXEC.BAT inside the ZIP. This is very limited and looks like a mess, but functionally should be just fine for the early 1980's when games didn't have many config options yet anyway.

A new test YAML for Alley Cat:

Code: Select all

---
metadata:
  title: Alley Cat
  publisher: IBM
  year: 1984
  comment: Testing DOSContainer, not functional yet.
  filename: alleycat.zip
diskname: alleycat.vhd
disktype: pcxt
disksize: 10
application:
  - url: https://dosk8s-dist.area536.com/alleycat.zip
    label: Alley Cat
os: IBMDOS200

And for some more proof of concept ancient history:
https://github.com/doscontainer/games/

Yes, I will eventually port all the games that were already packaged for the Bash version of DOSContainer.

*Yes, for OS downloads these builds still do depend on my infra. Just you go an try beat AWS S3+CloudFlare for serving out a tiny ZIP file though.. it'll be there, trust me. And for the future, yes, you get the freedom of overriding the OS zips as well.

[Bas]

I just cracked the puzzle on how to recursively create subdirs. Filling them with files shouldn't be rocket surgery, just a lot of grunt work. To be continued. Excited!

..learning a lot about memory allocation, memory safety and concurrency. I'm actually glad I picked Rust over other languages. It's one thing to write up a tree structure in a language like Python. It's a whole different matter to do so correctly while working with Rust's borrow checker breathing down your neck. Really appreciating the elegance of an Arena allocator even though it means scrapping and reworking half my FAT code. Things are working out for the better.

[Bas]

Puzzle cracked I said last week.. right? Well, yes, theoretically. But just a minute ago: I created my first dir within a dir completely programmatically and scandisk does not complain anymore for any hdd size currently in scope. Now I can finally start writing functions that copy files over from the host to the VHD recursively into their respective subdirs. This, too, may still take a while due to a plethora of error conditions that need to be handled but there is light at the end of this tunnel and it's not a freight train coming at me! No build linked because this new functionality isn't user facing yet. Want to package ancient games from 1981-1984 without any subdirs? Use build 92 for now. Zork II was kind of fun!

[niallquinn]

Just so you know, I downloaded dosc version 100, I'm sure its a false positive, but both windows builds, 32 and 64bit, windows defender isn't a happy bunny with them!

Thanks, if I could understand this all I'd have a go at some pc games myself! Reading the manual, seeing if I can get my head around it!!

[Bas]

There's ancient boot sector code from IBM DOS inside the executable, but it's in a data segment so that is bound to look a lot like a malware dropper to Defender. Now I could externalize those bits or try to obfuscate them, but this program does what it does: create bootable disks, so it'll always look suspicious. That's why I publish it as open source.

[Bas]

Riight.. so stuff happened in my life. I'm getting to grips with a new job that's taking a lot of my time so DOSContainer is now on a very slow simmer until I get my bearings. I'm responsible for the networking team at a bank with a few million customers now, so that's been something of a timesink over the past weeks. My laptop also broke so I had to get a new one that took a while to arrive.

That said, DOSContainer is still on my radar and I made a few tiny improvements to the build process. Builds are now cryptographically signed. You can use GnuPG to verify the signature on the distribution ZIP-file. In the downloads folder of my FTP you'll find a zip as well as an accompanying .asc file that contains the signature. These must match, or something is definitely wrong.

I also expanded a little more in the README on the false positive notifications you may get from malware detection software.

Feature-wise I'm still thoroughly blocked on properly supporting recursive file copying. Hopefully I'll get past this hurdle. It's the only thing stopping DOSContainer from being usable for real, big DOS games and all of its other projected features.

[Bas]

Small update since it's been a while. I'm getting to grips with my new job and finding some time to code on DOSContainer again. The good news is that I finally cracked the puzzle on coding a tree of directories and files in Rust. The lesser news is that I have nothing to show for it yet. I'm splitting DOSContainer into a separate number of libraries, each dealing only with its own concerns. The first library I'm writing in this manner is the FAT implementation, which is by far the biggest chunk and the most blocking. I need this to be 100% correct, so I'm writing more unit tests than actual functioning code. I had thought to borrow some existing code that already worked, but that's proving less useful than I had hoped. Running into earlier bugs and a lack of structure. So here I am, writing a complete historic file system implementation from scratch in a language I didn't know at all and am currently still a novice in.

By the time the new FAT implementation is on par with the current half-baked effort I'll replace it and publish new builds. For now, bear with me as I learn and make mistakes. Code for the FAT implementation lives here:

https://code.area536.com/DOSContainers/fat

I'm now about to scrap the AllocationTable struct and start that from scratch. Anyone who knows Rust and is willing to help, drop me a note!

[Bas]

Funny bug here. When working on games from the early 1980's things are mostly fine. I tested a big fat CD-ROM title that weighs a few hundred MB the other day and my MiSTer ran out of memory (yay for testing!). Turns out I should not buffer the entire uncompressed game plus the VHD file in RAM at the same time. That's just sloppy engineering on my part, RAM is not limitless. Refactoring just about the whole thing yet again to keep RAM usage low. Also adding in floppy support. It felt kind of idiotic to create VHD's for games that only weigh a few kilobytes each. So floppies it is. Now that I'm touching all of the disk image generation code anyway, it feels like the right time for it.

Same recipe though, I'm going for accuracy here so everything from MS-DOS 1.0 on 8" monstrosities through 3.5" 2.88MB with MS-DOS 6.22 should work. This project is much larger than anticipated. There's so much plumbing to do with so little to show for it that I'm starting to regret announcing this thing so early, apologies for getting your hopes up so soon. I am still working on this though and things are indeed moving forward, even if it doesn't look like they are.

[AmintaMister]

Funny bug here. When working on games from the early 1980's things are mostly fine. I tested a big fat CD-ROM title that weighs a few hundred MB the other day and my MiSTer ran out of memory (yay for testing!). Turns out I should not buffer the entire uncompressed game plus the VHD file in RAM at the same time. That's just sloppy engineering on my part, RAM is not limitless. Refactoring just about the whole thing yet again to keep RAM usage low. Also adding in floppy support. It felt kind of idiotic to create VHD's for games that only weigh a few kilobytes each. So floppies it is. Now that I'm touching all of the disk image generation code anyway, it feels like the right time for it.

Same recipe though, I'm going for accuracy here so everything from MS-DOS 1.0 on 8" monstrosities through 3.5" 2.88MB with MS-DOS 6.22 should work. This project is much larger than anticipated. There's so much plumbing to do with so little to show for it that I'm starting to regret announcing this thing so early, apologies for getting your hopes up so soon. I am still working on this though and things are indeed moving forward, even if it doesn't look like they are.

once again: thank you!

[niallquinn]

JUst download build 100, read the manual,still don't grasp it, but I guess that's aprt of the fun.

I'm going to get pga 486 golf to work if it kills me!

[Bas]

Don't beat yourself up over this. Any available build right now is an unfinished burning pile of.. well.. something that burns poorly and smells. Don't invest too much time into it.

The good news is that I actually did manage to crack the biggest puzzle that was blocking my progress. Current state of affairs: still implementing FAT12 on floppy images, doing it in a way that will scale not just to FAT16/32 but across each and every single cringe-inducing variant as needed without breaking the lib's architecture yet again.

As it stands, I really do regret announcing this so early.

[AmintaMister]

Don't beat yourself up over this. Any available build right now is an unfinished burning pile of.. well.. something that burns poorly and smells. Don't invest too much time into it.

The good news is that I actually did manage to crack the biggest puzzle that was blocking my progress. Current state of affairs: still implementing FAT12 on floppy images, doing it in a way that will scale not just to FAT16/32 but across each and every single cringe-inducing variant as needed without breaking the lib's architecture yet again.

As it stands, I really do regret announcing this so early.

Again: THANKS!

[niallquinn]

Well faint never won heart maiden er, or somthing like that.

It's good fun messing about. Got the week off work, spending it equallly between Mister and 00 gauge. As an old man, it doesn't get better than this!!!

[Bas]

You won't get anything beyond very simple one-file games working with currently available builds.. and that's only if I didn't accidentally destroy the distfiles that these builds may or may not depend on. I really would advise you to not invest any time into existing builds.. If you're up for a challenge, learn Rust and hack on the project itself!

A somewhat positive turn of events because I had a day to spend coding today. Since I'm working on this in a very rigorous manner, I started at the original IBM PC with IBM PC-DOS 1.00. Today I wrote my very first empty 160KB 5,25" floppy image from DOSContainer. It is bit-for-bit identical to what IBM's FORMAT command does including IBM's corporate mainframe-ism of writing 0xF6 bytes all over the disk's data area and preallocating fake "deleted" entries in the root directory. I'll post a blog about it sometime later this week.

File allocation is still buggy right now so I'm botching up the FAT when creating system files, but I'm this close to nailing it. Once this part works reliably and has al testing code crud removed, I'll start releasing DOSContainer builds again but they'll be appropriate mostly for the PCXT core and other emulators of ancient IBM gear like 86Box.

Hang in there folks!

[Bas]

Here's a bit of info on what goes into creating an empty floppy that is IBM PC compatible:

https://www.area536.com/posts/anatomy-of-a-floppy/

[Bas]

Tiny update: just succeeded at booting IBM PC-DOS 1.00 from a generated floppy in 86Box. I was never a fan of IBM. I'm still not. The generated floppy is not 100% bit-for-bit identical yet, but I'm getting there. Interesting little factoid: IBM did not use DOS on the PC itself to write its release floppies. The allocation table is not what DOS would have written at the time. IBM probably used a build environment on one of their big systems to master the PC-DOS 1.0 release floppies. My target for DOSContainer remains: write disks identical to how DOS would have done it, regardless of how much IBM cheats.

[Bas]

The floppy now also passes CHKDSK with flying colors. Should be obvious because it's now bit-for-bit identical to what IBM does, except for a small amount of deep weirdness. I lifted the original system files off the installation floppy by mounting it in Linux, reading the files with the Linux FAT driver and wrapping them up into a distribution ZIP file that DOSContainer uses as the inception layer for the entire Docker-workalike concept I'm chasing.

Seemed like Linux missed a bunch of bytes at the very end of each of these files. So I tried again and used Windows: same story. FreeBSD? Same story. That's three distinct FAT driver implementations. My hunch now is that these implementations are all correct, but IBM's is flawed. I'm leaning that way because FAT drivers regularly missing bytes would cause all manner of mayhem in the world at large.

The file size on each of these files as registered in the root directory of the original installation floppy adds up to a point before those weird bytes. Yet the FORMAT utility seems to just copy whole sectors instead of the files themselves to any target floppy so it rounds the size up to multiples of the 512-byte sector size. This is a hypothesis I'm going to figure out later this week by using DOS itself to copy files over with the COPY command rather than FORMAT /S and figuring out what ends up on the actual disk.

If anyone has actual source code to the FORMAT command of IBM PC-DOS 1.00 that would be most welcome. I can only find documentation for this version (which is unhelpful) and source code of newer versions of DOS, which don't seem to show behavior like this.

For now I'm off to do a little happy dance though: my software is finally working as it should. Now to strip out all the debugging cruft and make something of a presentable build available to all of you out there. That won't happen this weekend unfortunately, but I'm close!