MiSTer FPGA Forum

Has anyone gotten dynamic VHDs working/booting in the AO486 core?


End goal feature:
Allow for Dynamic VHD, very small initial load of base OS, tools etc (call it under 100MB) with a VHD size set to grow up to XX GB.

Goal, very small initial load of a VHD with just a layer of DOS/FreeDOS setup that is sized using dynamic VHD.
Call it 20MB consumed with a dynamic VHD of 16GB. The actual size of the VHD would be small but the hard drive values would be reported as: Sectors:63, Heads: 16. Cylinder: 32507, Size: 15999mb.

This works well in pcEM, windows, etc for VHDs.


What works:
Github scripts (already created) that will allow for users to select the open source/freeware/shareware games that they would like to pull down and include in their image. mount VHD from script, Script pulls games down that were selected and any updates to them, VHD grows based on data consumed, new index generated for games selected and inserted into the AO486 launcher.

This would allow for a 100% legal way to host shareware games on github allowing for people to enjoy Shareware DOS games like we used to. I've already curated the pack, tested each game, built the scripts, but I am blocked by the dynamic VHD issue. Right now, I can create flat VHDs of varying sizes that are formatted but empty (500MB, 1GB , 3GB etc) I zip them to bring down the size to the actual consumed size, mister script pulls those down from github, unzips, mounts them, then transfer data to the new larger VHD. Then remove all the junk from this process.

Issue:
It looks like the Bochs BOIS that is used is reporting the VHD referenced above as a 284MB HDD instead of 16GB. Is that one of the "features" passed in on line 78 of ide.v?

Is the bios defaulting to flat and can that be changed to "growing"?

Or to give me some more troubleshooting capabilities, can the bios line for IDE0 master: display the Sectors, Heads, and Cylinders it read from the VHD? IF not, can the MiSTer OSD show that data when we mount the VHD to IDE 0-0?

Would be awesome. I had experimented a bit hoping they could work with AO486, but found it did not.

Less locked down when its dynamic.

I had a look at it, and it seems to be possible:
https://github.com/MiSTer-devel/Main_MiSTer/issues/395

Is the bios defaulting to flat and can that be changed to "growing"?

You still need to set the (maximum) capacity of the VHD when you create it, this is the capacity that the AO486 sees.
As soon as you create it, you get a one-time file size overhead to store a table of if and where the data for each chunk of of hard-disk data can be found in the file (my empty 1 GB , FAT16 formatted vhd has a size of 16 MB).
The overhead will be bigger for a bigger capacity drive, but the overhead does not grow with the amount of data on the disk.

Create a big VHD file. Make sure it's initialized with zeros (important point!). Format it in ao486 and fill with initial freeware dos, games, etc.. ZIP it. You will get very small zip of size almost equal to files you copied into VHD. Then you can distribute this ZIP.
User will need just unzip it to USB drive and task is done.

Sorgelig wrote: ↑Tue May 11, 2021 5:12 am Create a big VHD file. Make sure it's initialized with zeros (important point!). Format it in ao486 and fill with initial freeware dos, games, etc.. ZIP it. You will get very small zip of size almost equal to files you copied into VHD. Then you can distribute this ZIP.
User will need just unzip it to USB drive and task is done.

Thank you, yes that is the path I took, I can get the vhds down to about 1MB and have a mister script that expands them and then pulls everything down from github and "hydrates" the image. The goal was a single script with the ability to run everything from the mister and allow change control and updates to the vhd directly from GH.

This works fine but if you ever hit the limit of the larger vhd then I have to use a different script that pulls down a larger VHD and then does that same dance again. The extraction on the mister with either gzip or zip is around the same (3 min per 1GB).

Empty VHDs for some reason benefits highly from being compressed twice and ends up just being a few kilobytes in size.
Here is an example:
https://dionysus.dk/software/DOS/8GB_Fat32.7z

There is a thread in this AO486 forum that shares several VHDs of varying sizes.
But it sure would be a LOT more useful with dynamic size VHDs. It would save so much space instead of having several 2-8gb VHDs.

The AO486 core is one of the reasons why I am now using a 1tb SSD with my MiSTer. I have some 30gb in VHDs I think it must be, on top of all the CD images and such. Its partly due to needing to clean up some of my VHDs though, but still.

Also as Flynnsbit points out, its really slow to extract several GB on the MiSTer or transferring to the MiSTer. Which is also why I stick with high speed USB storage instead of the SD cards now. I would still do this even if this core got support for variable VHDs, but especially for Flynnsbit with what he is doing with the large DOS games collections, I can see why a variable VHDs would help a lot.

No idea how complicated it might be to implement though.

But I found an interesting thread where some developers comment on how they work, in a PCem forum:
https://pcem-emulator.co.uk/phpBB3/view ... 8&start=30

The one disadvantage of the proposed zip solution is that you cannot put too many large, but empty, unzipped vhd files on your USB drive, and you would loose data if you unzipped them before mounting, but did not zip them again after un-mounting.

I understand that adding complexity is not good.
I mainly did this because I had to disappoint fynnsbit in a previous discussion, and I thought "Lets see if I could fix that?"


Below I add what I learened about VHD in case others would be interested in it, or in case you change your mind.


I implemented support for the [Connectix VHD format](https://en.wikipedia.org/wiki/VHD_(file_format)).
It may be that other emulators use the extension .VHD for other formats, so I use "Connectix VHD" to refer to this format.

The Connectix VHD format is a relatively well established format for virtual hard disk emulation. It was developed by Connectix around 1997 (so it is about the same age as the PC we are trying to emulate). The format is currently owned by Microsoft, and has an open source, but probably not "libre" license. It is the file format that you create when you use FDisk to create a vhd. It can then be mounted as a hard-disk on most current computers and emulators (Windows, Linux using Guestfs, VirtualBox).

The MiniVHD library is a relatively small open source library for dealing with virtual hard disks in the Connectix VHD format (but not for accessing the file system). This library is also used by other PC emulators, like pcEM.
The implementation is not very complex, but yes, the library is probably comparable in size to MiniZip, and the utility is less.

A Connectix VHD file has a footer that contains meta-data about the disk (heads, cylinders, sectors, ...), and a way to recognize that it is a Connectix VHD file. There are fixed size and dynamic size VHD files (and others, we do not care about here).
In the implementation, FileOpenEx reads the location of the footer meta-data to check if the file is a Connectix VHD file, and then opens, closes, seeks (well, not really), writes and reads the file using the MiniVHD library code. If it is not a Connectix VHD file, it will deal with it as before so cores that have VHD files that are not Connectix VHD should continue to work.

The fixed size Connectix VHD is already be handled by MiSTer now, except that the current way of mounting a general VHD ignores the meta-data footer, so the size of a Connectix VHD disk file is not reported correctly by MiSTer. Of course, there would probably be an easier fix for that.

A dynamic size VHD looks to the emulated computer like a fixed size volume. As mentioned before, the formatting and the file system do not have to be visible to the library, so it can have any format supported by the program that creates the VHD and the emulated computer (OK, probably not any, but at least FAT16, FAT32, NTFS).
The meta-data in dynamic VHDs also contains a chunk table. The rest of the file to consist of fixed-size chunks of data that are usually larger than a hard-disk logical block.
When the implementation gets a request for a logical block addresses, it calculates its chunk index.
It then indexes the chunk table for the location of the chunk in the file.
If the chunk is in the file, it goes to the chunk location and reads or writes the data there.
If the chunk is not in the file, and it is a read, it returns zeros,
If the chunk is not in the file, and it is a write, it adds a chunk with the new data after the last chunk in the file, and adds the chunk to the index.
I guess you can see that reading and writing would be almost as fast as a non-dynamic vhd file, especially if your USB drive is an SSD or Flash.

I think it may bring unexpected behaviour and complain from users.
For example user copy such dynamic VHD with size bigger than empty space on USB drive. Then it will start to grow beyond the available space while ao486 has no idea about it. So it will crash somewhere in the middle.
another issue is to work with such VHD. It will require a special software understanding this spaghetti of chunks. Currently i simply mount VHD by imDisk on Windows and work as with normal local disk. On MiSTer i also mount VHD and can access it remotely.

schendel wrote: ↑Tue May 11, 2021 5:44 pm and you would loose data if you unzipped them before mounting, but did not zip them again after un-mounting.

there is no "zip again". It's supposed to be unzipped once from distribution to USB drive and use. You misunderstood the option.

I don't see much advantage of dynamic VHD. There is really no problem. Make big VHD and use it. You will be always sure that whole space got reserved and always available. If you want to supply pre-made image then as i've told above zipped version will work fine.

I don't like the whole idea of dynamic VHD. To many potential issues without much advantages.

Compression on the filesystem layer on the Linux side would be viable and probably doable if you sacrifice usability of the SD card in non-Linux systems. I honestly couldn't care less about the filesystem because I always manage my MiSTer's content over the network, but for others this is probably an issue. I guess nobody is stopping anyone from trying something like this on their own device.

Sorgelig wrote: ↑Tue May 11, 2021 6:26 pm I think it may bring unexpected behaviour and complain from users.
For example user copy such dynamic VHD with size bigger than empty space on USB drive. Then it will start to grow beyond the available space while ao486 has no idea about it. So it will crash somewhere in the middle.
another issue is to work with such VHD. It will require a special software understanding this spaghetti of chunks. Currently i simply mount VHD by imDisk on Windows and work as with normal local disk. On MiSTer i also mount VHD and can access it remotely.

schendel wrote: ↑Tue May 11, 2021 5:44 pm and you would loose data if you unzipped them before mounting, but did not zip them again after un-mounting.

there is no "zip again". It's supposed to be unzipped once from distribution to USB drive and use. You misunderstood the option.

I don't see much advantage of dynamic VHD. There is really no problem. Make big VHD and use it. You will be always sure that whole space got reserved and always available. If you want to supply pre-made image then as i've told above zipped version will work fine.

I don't like the whole idea of dynamic VHD. To many potential issues without much advantages.

The advantage is not having to use several gigabytes of space without good reason. It ends up being several gigabytes for me. I have used a 128gb microSD card and it ended up frying. So being able to use smaller SD cards seems like a big advantage to me. I have now ended up using a full SSD with my MiSTer to get around the space issues.

Sure, people might run out of space because they expanded their dynamic VHD, but that does not really seem like a problem. People who decided to use dynamic VHDs ought to know they need to have space free before adding more files to them.

But for the most part I think the CD support that got added solved a lot of these issues, as it means I can use small installs of CD games that then use the data directly from the CD. Back in the day that was a problem because CDs were really slow to use compared to HDDs, but on the MiSTer they are at least as fast as the systems disk.

Moving files to the VHDs are not the issue, its just that they have to be the full 4-32gb in size, without actually having put that much on them. Also while working on one VHD you might install a game which needs a lot of temporary files, maybe you need to unzip several install files and so on. But then when you are done working you can clean that up and you might be down to having less on the disk than you started out with.

But I see the issue with there being a spaghetti of chunks with the VHD data being spread out on whatever storage its on, because it dynamically has data added to it. I would have hoped it might be something the Linux system could handle?

Might depend on the file system according to this:
https://superuser.com/questions/1096549 ... k-on-linux

But I do not have a solution on how to implement it, so I do not expect to see it happen before someone has a solution to that.

Caldor wrote: ↑Wed May 19, 2021 11:28 am Moving files to the VHDs are not the issue, its just that they have to be the full 4-32gb in size, without actually having put that much on them. Also while working on one VHD you might install a game which needs a lot of temporary files, maybe you need to unzip several install files and so on. But then when you are done working you can clean that up and you might be down to having less on the disk than you started out with.

Why that big? I use all sort of vhd sizes, my smallest is 100MB. They work just fine.

I'm making my own DOS collection (based on TDC) and so far everything between 1981-1992 takes up only 4.7GB. After that the CD games got popular and the directory sizes do get a bit wild, so it's probably better to either use a hard drive or make small vhds with selected games one wants to play.

akeley wrote: ↑Wed May 19, 2021 3:03 pm

Caldor wrote: ↑Wed May 19, 2021 11:28 am Moving files to the VHDs are not the issue, its just that they have to be the full 4-32gb in size, without actually having put that much on them. Also while working on one VHD you might install a game which needs a lot of temporary files, maybe you need to unzip several install files and so on. But then when you are done working you can clean that up and you might be down to having less on the disk than you started out with.

Why that big? I use all sort of vhd sizes, my smallest is 100MB. They work just fine.

I'm making my own DOS collection (based on TDC) and so far everything between 1981-1992 takes up only 4.7GB. After that the CD games got popular and the directory sizes do get a bit wild, so it's probably better to either use a hard drive or make small vhds with selected games one wants to play.

Well, that is just it, the main reason that I used 8gb VHDs was to store a lot of CD images on them before there was CD support. So... with CD support I can just have the CD images on the disk ready for mounting, instead of putting them on a VHD.

But I still have 20 VHDs, so I try to avoid using 100MB VHDs, otherwise I quickly end up with 50 VHDs. I think my smallest VHDs are therefore 500MB. But would be very nice if I could just have a 50MB VHD that could then be expanded to 8GB if and when it made sense to be, rather than having to fidget around with so many different VHDs of various sizes.