DOSContainer Alpha Release

[Bas]

Dear fellow PC enthusiasts. Enjoy!

https://gitlab.com/doscontainer/framework/

The DOSContainer project started out over a year ago in response to me becoming aware of community-created "packs" of games. These come in the form of large hard drive images with a menu system layered on top, allowing for an easy end-user experience in playing a large number of retro games in platforms like the IBM-compatible PC and Amiga. While I tip my hat to the creators of these packs in admiration of their skill and dedication, these were not what I was personally looking for. DOSContainer was born.

So what is it?
DOSContainer aims to be a tool that sits in the middle between the retro purist who does everything by hand, and the casual gamer who enjoys playing from game-packs. Note that there is absolutely no value judgement in there on my part. Everything goes, I'm just scratching my own personal itch here. Now if I had to try and sell DOSContainer it'd be something along the lines of a declarative solution to generating MS-DOS compatible hard drive images in a reliable and repeatable way. You can very easily mix and match different versions of operating systems, drivers and utilities with your games to see which combination is optimal for you. Just write a configuration YAML, run the script, and there you are.

Probably very buggy alpha release, but hey, it's something and it's out now. I'll keep working on this over the next few weeks but the shop is now open for suggestions and pull requests. The code is still a bit messy and there are some known issues, particularly in handling full OS installations and the installation/extraction of DOS itself. Feel free to dig in though.

Usage
https://gitlab.com/doscontainer/framewo ... s/usage.md

Build your own game configs
https://gitlab.com/doscontainer/framewo ... igfiles.md

[Bas]

Service announcement: if you downloaded yesterday's version, you'll need to update framework.sh to the current version in the main branch. I refactored a lot of code around handling MS-DOS installs and replaced distfiles for them, so checksums will break on previous versions of the script. Bumped the framework version to 0.0.2 for now.

[PistolsAtDawn]

Cool! Thank you for this!

I believe I fall into the "just wants to play" camp and not the "retro purist" camp. In fact, I can't think of a single thing where I'd consider myself a purist, so this sort of utility is perfect for me. I have limited time to dedicate to playing games, and therefore spending time learning about platforms first to be able to play games after just doesn't appeal to me. The Amiga is a really interesting system that I never got a chance to use during its heyday, and I have no real love for DOS systems even though that's where I started with PCs. If this takes the work out of experiencing games on those platforms, then I'm all for it!

[Bas]

The point of my tool is indeed to take most -if not all- of the toil out of DOS gaming. You can just download a disk image (I'm building a library of them here: https://www.area536.com/posts/doscontainer-releases/). Those are plug-n-play: stick in your MiSTer, boot, and play. Right now they are early days and not all of them are fully tested. The page I just linked to will receive a bit more love as well.. but you get the idea.

You an also use DOSContainer on your own computer and build your own disk images. The YAML files are tweak-friendly in that you can simply set different values for whatever you like and build new images in a few seconds or build large numbers of them in bulk.

There's also a set of configs for all MS-DOS versions the tool supports. These give you a really easy base to start building other stuff from if you want to geek out in DOS. Need a hard drive of exactly 43MB in size with MS-DOS 4.01 on it? Just tweak the msdos401.yaml to set the disk size, run DOSContainer, and you can then simply take it from there.. which is a heck of a lot quicker than installing MS-DOS 4.01 used to be.

I'm also considering putting DOSContainer behind a web form on my own webpage. Drop in a valid YAML, and a .vhd comes out.. but that's something I'm only considering for sometime late next year.

[DiamondDave]

Awesome! Will try this out over the holidays

[Bas]

I hope you'll like it. Please report any issues you run into. I'm developing on something of a niche set of systems so not every bug is immediately apparent to me.

[Bas]

Open issues and feature requests are on this page by the way. You can log in here with your GitHub credentials if you don't have an account with GitLab yet.

https://gitlab.com/doscontainer/framework/-/issues

[Newsdee]

Nice! Yes, for some games I find it comvenient to have a single image - especially adventure ot RPGs that would need a few hours of play and keeping saves.

Backing up your saves from a big image could get tedious. Whereas I can easily locate and copy a single game image via SFTP...

By the way, there was similar work in the past that added ta "MiSTer" mode to the Exodos Converter. Perhaps you can build upon that to facilitate creation of your HDD images?

https://github.com/Voljega/ExoDOSConverter
.

[Bas]

Call for testing: I just merged a fix which should make the tool work with yq version 4.x and up (which you can pull in from snap in current Ubuntu LTS). For some reason my Gentoo box only comes with the comparatively ancient yq 3.2.3 right now. My problem is: I don't have a properly working Ubuntu system up and running, plus I'd be very anxious to know if DOSContainer actually runs from WSL2.

[RealLarry]

Going to test the new version today. yq on ArchLinux is also at 3.2.3.
One thing I 've observed is that the field publisher is looking like to have some quirks. I'm working on Terminal Velocity and the publisher is named 3D Realms Entertainment , which seems too long. A shorter form is working.

[Bas]

That's funny.. the metadata section is completely unused yet. But It'll look into it! It shouldn't have any effect whatsoever so I'd like to understand why if it does. Thanks!

[Bas]

I can't reproduce the issue you're seeing with 3D Realms Entertainment. Have you tried pulling your YAML through yamllint manually?

[RealLarry]

That's really funny. Just finished some games a few minutes ago and re-tried Terminal Velocity with the full length text for "publisher" after your message...and guess: it's working now. The only line I've changed in this yaml is a last line "FILES=50". Weird.

Shall I upload them?

[Bas]

Sure, I'm intrigued! Plus.. hey.. games!!

[Bas]

The Rust version of DOSContainer is progressing at a paintstaking rate. The upside: it really is becoming 100% compatible (with IBM PC-DOS 2.00 at this point). The downside: progress is slow because I'm also still teaching myself Rust on the back of this project. Progress report:

• 100% identical MBR to what IBM generates in DOS 2.00
• Allocation of partitions is 100% identical to what IBM does.
• 100% identical Volume Boot Record compared to IBM DOS 2.00

..and in the process it can now:

• Read and parse part of the Bash-version's YAML manifest format much more reliably than Bash every did.
• Analyze and report on the technical specifics of VHD-format disk images

Next steps:

• Format C: as empty, with FAT12 structures 100% identical to IBM.
• Write PC-DOS 2.00 system files 100% identical to what IBM does.

[Bas]

Format C: as FAT12 now also works.

IBM and MS made a lot of assumptions that they never actually put into prose. It's not easy getting a faithfully emulated XT to eat just any random variety of hard disk, even if it's no bigger than 32MB. Good thing is that the IBM 5160 technical manual comes with the full Assembler listings of both the HDD controller and the system's own BIOS and Microsoft published early DOS sources some time ago. Code beats documentation any day, and heck.. learning 8088 assembly was on my bucket list anyway.

[AmintaMister]

Format C: as FAT12 now also works.

IBM and MS made a lot of assumptions that they never actually put into prose. It's not easy getting a faithfully emulated XT to eat just any random variety of hard disk, even if it's no bigger than 32MB. Good thing is that the IBM 5160 technical manual comes with the full Assembler listings of both the HDD controller and the system's own BIOS and Microsoft published early DOS sources some time ago. Code beats documentation any day, and heck.. learning 8088 assembly was on my bucket list anyway.

Thanks for your precious work, again!

[Bas]

There's a whole paragraph from MS on how to determine sectors per cluster. But if you're PC-DOS 2.00 the number is just 8 no matter the drive. Funny one that, IBM.

Anyone interested in tracking my progress or suggesting improvements (those are welcome!!) can look here:

https://code.area536.com/DOSContainers/doscontainer

[Bas]

DOSContainer Rust just wrote its first file (IBMBIO.COM) to a VHD that was bootstrapped completely from scratch, zero external dependencies. Things still aren't ready for prime time and the code surrounding the writes is messy, but I'm very slowly getting there.

[virtuali]

Nice so, you are creating .VHDs programmatically, right?

Have you considered some kind of variable parameter to tell, game by game, how much extra free space each game might need when running? I found a couple of cases where the .VHD was made so tight, so the game started, but didn't have enough space for the Saved games.

[Bas]

The parameter is, essentially, there even if not directly. The program takes a YAML file where everything including the disk size is defined. If you are short a few (dozen) megabytes, you change the number and it rebuilds a new clean and fully installed VHD.

The command for that now:

Code: Select all

$doscontainer build mygame.yaml

..so it's fairly simple from a game packaging standpoint. I just benchmarked my laptop from 2018 and it goes from zero to VHD in literally 0.006 seconds with game assets pre-downloaded. At this speed I can easily spit out dozens of permutations for different game-configurations like sound, video, controller, languages etc.

At this point in time I'm still working on a 100% accurate period-appropriate FAT implementation and I'm not there yet (learning to code Rust at the same time). DOSContainer Rust can already be used to spit out bare-but-valid VHD's with a partition table on them. If that's already useful to anyone, shoot me a message and I'll provide binaries that can do just that part.

The Bash version that's referenced at the top of this thread does the whole shebang already, but it's a baroque mess of dependencies and is maintenance hell.

[kevind]

Another feature request:

If it is possible to read files from the vhd as well, it would be nice to have a “save regular expression” per config that would allow you to extract the save files from all of the containers and zip them up.

[Bas]

That'll be "collateral" from the analysis part of the tool. Until then you could use GNU Mtools to do this.

[Bas]

cicd_dosk8s.jpg (529.55 KiB) Viewed 12774 times

Gearing up nicely. Builds are now generated automatically for platforms other than just my local laptop. Unfortunately I'm not (yet) able to do Mac builds from Linux due to licensing issues from Apple.

More info here: https://www.area536.com/posts/doscontainer-rust-cicd/.

[Bas]

For now, dev builds of the Rust version will live at:

ftp://ftp.area536.com/doscontainer/builds/

That is, until I build a decent home for DOSContainer on the web. The build pipeline will publish each and every single build from the main branch in here. The number in the file name signifies the build numer, which increments. Any gaps will be failed builds that didn't produce an artifact. Build 43 is the first successfully uploaded version.

Do treat this as pre-alpha quality software. It does something but is not at allsuitable for productive use. I would be interested to know on which versions of Windows these builds actually run, as I'm not a Windows user myself. If you get usage information when starting the executable, the build works. If you have older Windows versions running, please let me know if the executable (either the 64 or 32-bit) runs at all.

In the screenshot above you see a build step 'Malware scan'. That's an antimalware run by Clamav to catch supply chain attacks on any of the 140+ dependency "crates" my Rust project uses. It's not airtight by any means, but at least I'm doing my part to try and prevent issues. I will provide PGP signed builds in a later stage when the project actually becomes usable.

If you want to build your own, the source is freely available from https://code.area536.com/DOSContainers/doscontainer. You'll need a Rust environment on your machine (see https://rustup.rs). Clone the repo to your machine, then from the root of the repository run 'cargo build --release'. Your very own binary will be in the 'target/release' directory when it's done.

[ItalianGrandma]

Awesome! Definitely going to take a look at this.

[Bas]

Anyone here up to speed on how to exactly compose a FAT12 allocation table? I'm sooo very close right now to getting PC DOS 2.00 booting correctly, but my implementation is ever so slightly off..

[Bas]

That's what always happens when I curl into a tearful ball of desperation on the floor in front of my computer: it gets unstuck. I just generated my first FAT12 file system from scratch with a file on it, and not even Norton Utilities finds it suspicious anymore. To be continued!

[Bas]

Finally! PC-DOS 2.00 boots cleanly in PCem using nothing but the latest build of DOSContainer to create a VHD. This means you can pull the executable from my FTP server (build 59, currently in progress) and combine it with a simple testing YAML. I'm still quite a distance away from creating and filling subdirectories on these disks, but this is a very significant "one small step" moment for this project.

If you feel like testing, fire up your FTP client and connect to ftp.area536.com anonymously. The build for all supported platforms will drop in the /doscontainer/builds folder in a few minutes as doscontainer-59.zip. There is, of course, an ARMv7 executable in there that runs straight from MiSTer itself.

Place the following YAML next to it and fire it up. You'll end up with a 10MB VHD that should boot (it does for me on PCem). I have a lot more testing and tweaking to do, but family duties come first now.. so hopefully you'll be able to tell me in how many ways this build broke for you when I get back.

Code: Select all

---
metadata:
  title: Alley Cat
  publisher: IBM
  year: 1984
  comment: Testing DOSContainer, not functional yet.
  filename: alleycat.zip
diskname: alleycat.vhd
disktype: pcxt
disksize: 10
distfiles:
  - name: alleycat.zip
    checksum: 0316fb862c67fdf9318a5c8513d99e5af185ce10306d20c27f5c6da099b5b176
gamefiles:
  - ALLEYCAT
os: IBMDOS200
AUTOEXEC: |
CONFIG: |

PS. The PDF manual you'll find in there is a.. umm.. work in progress. Bear with me please.

[NML32]

Finally! PC-DOS 2.00 boots cleanly in PCem using nothing but the latest build of DOSContainer to create a VHD. This means you can pull the executable from my FTP server (build 59, currently in progress) and combine it with a simple testing YAML. I'm still quite a distance away from creating and filling subdirectories on these disks, but this is a very significant "one small step" moment for this project.

If you feel like testing, fire up your FTP client and connect to ftp.area536.com anonymously. The build for all supported platforms will drop in the /doscontainer/builds folder in a few minutes as doscontainer-59.zip. There is, of course, an ARMv7 executable in there that runs straight from MiSTer itself.

Place the following YAML next to it and fire it up. You'll end up with a 10MB VHD that should boot (it does for me on PCem). I have a lot more testing and tweaking to do, but family duties come first now.. so hopefully you'll be able to tell me in how many ways this build broke for you when I get back.

Code: Select all

---
metadata:
  title: Alley Cat
  publisher: IBM
  year: 1984
  comment: Testing DOSContainer, not functional yet.
  filename: alleycat.zip
diskname: alleycat.vhd
disktype: pcxt
disksize: 10
distfiles:
  - name: alleycat.zip
    checksum: 0316fb862c67fdf9318a5c8513d99e5af185ce10306d20c27f5c6da099b5b176
gamefiles:
  - ALLEYCAT
os: IBMDOS200
AUTOEXEC: |
CONFIG: |

PS. The PDF manual you'll find in there is a.. umm.. work in progress. Bear with me please.

Is there anything else needed besides the doscontainer.exe (using Windows 10) and the yaml file to build the allycat bootable VHD?

Update: I was able to create a bootable VHD in Windows 10 using doscontainer.exe and the allycat.yaml. Booted from ao486 Nice.

20240411_163801-screen.png (69.64 KiB) Viewed 12321 times