HOW TO: setup Total DOS Frontend packages on AO486.

[Bandit721]

Just a quick shoutout to Flynnsbit, BinaryBond007, Voljega, and anyone else involved in this effort. Great idea basing the set on ExoDOS and using TDL, and the strides made in such a short period of time is amazing. Thanks so much for all involved on the recent core improvements as well, just awesome!

[flynnsbit]

Very welcome. I just completed the last build tonight and got all 7000 games into a single VHD into the TDL front end. 7002 Entries, 760,187 files, and 17 revisions later we have the complete eXoDOS Collection in a front end on the MiSTer. The latest code changes that Voljega checked in should be the last of it. I'll do a demo soon.

AHHH!! I need to update the original post with all the new ExoDOSConverter script usage.

[Bandit721]

Wow! That is amazing news...congrats on getting this over the finish line and thanks again!

[nizmox]

I've been trying to follow the youtube guide but cannot quite manage to get it to work, if anyone can help that would be much appreciated.
Using this video https://www.youtube.com/watch?v=PVBkEm0lGsI

I think there are a few steps that aren't exactly clear, perhaps I need to watch the earlier videos? If you could help me out that would be much appreciated.

1. What VHD should I be using to boot?
2. What drives should be mounted? I imagine 0-0 for boot vhd and 0-1 for the games?
3. What path should the vhd, cd and floppy folders be on? Watching the video it appears that all files are in /games/ao486 (boot roms, cd, floppy and vhds). I noticed if I try use /games/ao486 it does not boot at all, for me it seems the boot roms must be in /ao486 (top level, not in games). This might be because I'm using an SD card and not an external harddrive, but it did confuse me.

Initially I was trying some of the boot vhds I found on the archive.org. There is quite a few out there, but they don't seem to work with these custom game collections. I think the "imgset" command to set the iso does not work.

I noticed there was a new release to archive.org with the new video posted by flynn, so I tried it this evening with my custom games collection built using the steps in the video. Firstly there seems to be two boot drives, and I'm not exactly sure which one to use.

If I try to boot it with 0-0 as the boot drive (I tried both), and 0-1 as my games collection I get the error...
"Please mount a games VHD to the "Mount Secondary HDD *.VHD" on your MiSTer and reset."

If I mount my games collection to both 0-1 and 1-1, it boots into tdl ok but then the games don't seem to launch. Probably because the cd mount command then replaces 1-1 with the iso? Not exactly sure.

Sorry if these are dumb questions, but I've been trying to follow the guide closely and still struggling to get this to work. I am downloading the large collection which i'm hopeful will work, but it's many many more games than I need.

EDIT: So I think the problem seems to be that if I use the boot .vhd's I've found on archive.org, these don't support "imgset" to mount the cds. If I use the flynnbits boot .vhd from his newest release something seems to go wrong with mounting the games .vhd in 0-1. Even if I exit TDL, and try to navigate to d:\ this does not work "General failure reading drive F", weird that it says "F" not "D" ? I've seen other users mention weird quirks around ide devices with windows boot vhds?

[flynnsbit]

My VHDs are setup very specifically to not only work together, but to also setup the drives where the exodos collection expects them to be. There are some crazy hacks in the vhds to accomplish this.

1. Boot is C: for the Primary VHD

2. Secondary VHD is actually two partitions, D: is a 10MB stub partition so that dos takes it and doesn't assign the TDL Games partition to it so we can use it later.

3. Second partition is the E: and has the main games, TDL, etc.

4. F: is the first CD

5. I am using the ASSIGN dos command in the autoexec.bat to assign F: to D: and take over the D: disk as the CD so now you have D: and F: as the same disk.

Now after all of that, if there are games that look specifically for a CD in the D: drive it will work. The reason we have to do this is because dosbox set is normally Z: on boot, and then allows you to mount anything to C: and D:. We don't have that luxury with real hardware.

If CD's are in the right folder on your MiSTer under /usb or /fat /games/ao486/cd AND the file is either a bin/cue , iso, or iso/cue, then it should mount and function. If you are trying to use a game that uses a .img file for CD images then that will not work as there is a bug. You can manually rename the .img to .iso and it will work. If you do that, you will need to change the 1_start.bat, for the imgset command to mount it properly.

[nizmox]

Thanks for this, that certainly helps. I've since been able to have some success using your v17 pack and extracting the cds from ExoDOS with the converter. I think my expectation was everything would just straight up work but it seems that isn't the case yet. Changing .img files to .iso has help resolve some I was struggling with.

Is there any way for me to raise bug reports for games not working?

For example...
* Dark Forces - mounted cd seems to be inaccessible, files are not visible in F:\
* Crusader No Remorse - Worked after renaming .img to .iso, but no sound. Setup.bat doesn't launch the correct setup, but was able to manually run setup to fix the sound.
* Crusader No Regret - Did not work, even after renaming .img to .iso. Setup.bat doesn't launch the correct setup, but was able to manually run setup to fix the sound which then fixed the game launching.
* Eye of the beholder 2 - doesn't convert properly with the converter. Also launching the game tries to mount eob1, I assume to import save, but this doesn't work. Removing the command in the start.bat to mount eob1 resolves it.

Also, just wondering if there is any way to know which memory profile to use?

Many thanks for all your hard work!

[cursedverses]

Hey guys,

I had some trouble with the Powershell - partially because of Windows Defender marking files as dodgy and removing them, and also the Compress-Archive response to "Quest ][" which blew its search.

So I rewrote it. And added a check for failed files. And imported a 7zip module. And a filter to bypass any already complete files (so you can rerun it if you have trouble).

Code: Select all

if (-not (Get-Module -ListAvailable | ? { $_.Name -match '7Zip4Powershell' })) {
    Install-Module -Scope CurrentUser -Name 7Zip4Powershell 
}
Import-Module 7Zip4Powershell

$workingDir = "C:\Users\Development\Documents\ExoDOS_Work"
$newDir = "C:\Users\Development\Documents\EXODOS" 
$fileDir = "D:\EXODOS\EXODOS\Games" 


cls
$failedItems = @()

$items = Get-ChildItem -Path $filedir

if (-not (Test-Path $workingDir)) {
    New-Item -Path $workingDir -ItemType Directory
}

$items = Get-ChildItem -Path $filedir

foreach ($item in $items)
{
    if (-not (Test-Path "$($newDir)\$($item.name)")) {
        $tempDir = Join-Path $workingdir $item.basename
        New-Item -Path $tempDir -ItemType Directory
        $Error.Clear()
        Expand-7Zip -ArchiveFileName $item.FullName -TargetPath $tempDir
        if ($Error.Count -eq 0) {
            $folder = Get-ChildItem $tempDir
            Compress-7Zip -Path "$($folder.FullName)\" -ArchiveFileName "$newDir\$($item.name)" -CompressionLevel Ultra
        }
        else {
            $failedItems.Add($item.name)
        }
        Remove-Item $tempdir -Recurse -Force
    }
}
cls

if ($failedItems.Count -gt 0) {
    Write-Output "Failed Items`r`n============"
    Write-Output $failedItems
} else {
    Write-Output "All OK!"
}

[flynnsbit]

Thanks for this, that certainly helps. I've since been able to have some success using your v17 pack and extracting the cds from ExoDOS with the converter. I think my expectation was everything would just straight up work but it seems that isn't the case yet. Changing .img files to .iso has help resolve some I was struggling with.

Is there any way for me to raise bug reports for games not working?

For example...
* Dark Forces - mounted cd seems to be inaccessible, files are not visible in F:\
* Crusader No Remorse - Worked after renaming .img to .iso, but no sound. Setup.bat doesn't launch the correct setup, but was able to manually run setup to fix the sound.
* Crusader No Regret - Did not work, even after renaming .img to .iso. Setup.bat doesn't launch the correct setup, but was able to manually run setup to fix the sound which then fixed the game launching.
* Eye of the beholder 2 - doesn't convert properly with the converter. Also launching the game tries to mount eob1, I assume to import save, but this doesn't work. Removing the command in the start.bat to mount eob1 resolves it.

Also, just wondering if there is any way to know which memory profile to use?

Many thanks for all your hard work!

Yes. Submit a bug report for those games on the ExoDOSConverter github page. I'm sorry if I mislead you down a path to thinking that 7000 games were tested and would just work. There is still a LOT of work to do. The only major goal accomplished so far is having the converter get the scripts from exo in a reasonable format and most of the cd/floppy setups where dosbox would use imgmount corrected. Then all of that in a single interface on the mister.

That does not mean all games are going to work. I am curating a Top 300 games VHD and testing every game in that list and I probably have 15 hours into correcting issues like you have outlined. If we can get more people reporting problems and finding the solution (like your list and what you did to fix "Had to correct setup path as it didn't cd down far enough," or "sound exe was setsound.exe which wasn't called") then we can correct it in the ExoConverter so the game works for the next person.

The img bug is a major one and having to rename every game that is .bin or .img is really draining.

For memory, I default to HIRAM unless there are problems (Lock ups on start, or game complaining of EMS/XMS missing), then I go to EMM386.

[Caldor]

Did ExoDOS v5 just get released recently? I downloaded ExoDOS v4 just a week ago and it is almost 500gb. I do not see much about ExoDOS news so I am not sure when this happened, but I am guessing it must have been pretty recently. v5 is 560gb, but it has some very interesting new features:

"Download on Demand: A 'lite' version of the project that just contains metadata. Games are downloaded the first time you try to run them."
and
"eXoDOS Updater: Download and auto-apply updates to your eXoDOS collection without having to wait until the next release."

I doubt I will end up using the full ExoDOS collection all at once given how big it is. I only got about 256gb storage for my MiSTer... but seeing how it seems to be getting support for Playstation and its support for DOS / Windows 95 and Amiga is going so well, I am thinking I might as well begin to consider some 1 or 2tb solution for it. DOS and Amiga alone could use up a terabyte each I guess.

When it comes to ExoDOS v5, I am guessing the support right now is only for v4. The converter at least only seems to support v4. I have not gotten around to actually try it, but just hte fact that so much work has been put into this, I need to try it out I think.

[flynnsbit]

It happened last night, you can seed the ExoDOS v4 with the v5 so it will only download the differences. Here's how:
To upgrade from v4 to v5, you can save time by doing the following:
Rename the eXoDOS folder in your v4 directory to eXo
Go into the new eXo folder, and rename the GAMES folder to eXoDOS

Now hash and it should use as much as your previous files as possible

Once you are done, you can use an app like TorrentCheck to remove any leftover files that are unneeded.

Note, nothing I am doing has been tested with v5 yet. You are on your own if you go that route right now.

[buttersoft]

...I am curating a Top 300 games VHD and testing every game in that list and I probably have 15 hours into correcting issues like you have outlined.

I've been following your efforts, and they're much appreciate. I can't deny a pre-made VHD collection would be awesome. Any idea how big such a VHD is going to be?

Also, while i'm flapping my gums - is there a chance to include needed/wanted patches for games such as Quest For Glory 4 and Mater of Magic?

[Caldor]

...I am curating a Top 300 games VHD and testing every game in that list and I probably have 15 hours into correcting issues like you have outlined.

I've been following your efforts, and they're much appreciate. I can't deny a pre-made VHD collection would be awesome. Any idea how big such a VHD is going to be?

Also, while i'm flapping my gums - is there a chance to include needed/wanted patches for games such as Quest For Glory 4 and Mater of Magic?

Ohh, running the updated Master of Magic on AO486 That could be pretty awesome. I am not sure, but it might have ended up needed a modern system with one of the updates that added for Master of Magic?

I recently tried AO486 using FreeDOS and FreeDOS CD-ROM drivers to run Kingdom O Magic from CD images using the new CD support, and it worked great. Still looking forward to trying this frontend package though when I find the time.

[Caldor]

It happened last night, you can seed the ExoDOS v4 with the v5 so it will only download the differences. Here's how:
To upgrade from v4 to v5, you can save time by doing the following:
Rename the eXoDOS folder in your v4 directory to eXo
Go into the new eXo folder, and rename the GAMES folder to eXoDOS

Now hash and it should use as much as your previous files as possible

Once you are done, you can use an app like TorrentCheck to remove any leftover files that are unneeded.

Note, nothing I am doing has been tested with v5 yet. You are on your own if you go that route right now.

I am already downloading both. So far I think the converter only supports v4 anyway, so I am mainly getting v5 to see what might have changed. For once I have plenty of free disk space, so I figured I might as well just download both. I should also have some space to then run the converter on v4 and prepare at least some of it for the MiSTer. Since the MiSTer so far wont have enough space. Maybe I should just find some cheap external HDD that has 2tb or more for the MiSTer.... and such a disk would probably be possible to use with a Pi and other similar devices as needed.

[evilsim]

@flynnsbit cant thank you enough for your time you have put into all this. i got the exodosconverter working in lubuntu after grabbing the one python package needed, but it would not work for me in Windows after building it, got GUI but would throw an error building the mister customizations, other emulation systems like retropie worked without fault however.

"MiSTer post-conversion
Error <class 'OSError'> while converting game $100000P"

Anyway, looking forward to getting a nice custom VHD up and running. as I said before THANKS

[flynnsbit]

I'll go test on my linux box today. Have you tried the very latest ExoDOSConverter build that has TDLIndexer built in or was this a different version from the master branch?

[Voljega]

Hello guys I'm the developer of ExoDOSConverter.

Just released the newest version with V5 support.
https://github.com/Voljega/ExoDOSConver ... g/0.7-beta

V4 support is dropped as too much things have changed

I fully plan to support updates and download-on-demand but this will require to integrate a torrent client in the converter, so it's no easy feat and it's not gonna be done tomorrow.

At the moment, I'm more willing to concentrate with flynnsbit on getting rid of the last bugs, and of all the new ones that V5 might have introduced

well If you encounter any bugs, don't hesitate on the github issues page, I will come here only from time to time

[Caldor]

Awesome. Thanks Voljega Did not expect to see V5 support so soon. Could be very useful with that torrent feature.

I will try to start out with using V5 then when I find the time to test all this out.

[Voljega]

Fix release as the one from yesterday had a lot of major bugs !
https://github.com/Voljega/ExoDOSConver ... 0.7.1-beta

[Caldor]

Me and Flynnsbit talked about how its a problem that we do not know which games uses DOS4GW or not, and whether they need to be stubbed by DOS32A f.ex. Especially because some games have a file called DOS4GW which makes it obvious that they use it, but some games just have it embedded in an EXE or DAT file.

I have decided to try to map which files in the ExoDOS collection has DOS4GW by using various methods. First the method of elimination. Games that are less than 1mb, probably wont use DOS4GW as it takes up about 250kb on its own. Then there are all the games that have DOS4GW as its own file. They are easy to figure out as well. Next part of the elimination process I will do is go through all the games larger than 1mb and see if they have any files larger than 250kb at all, as otherwise there is no chance they use it. I think its safe to assume files larger than... 100mb I guess, can also be excluded.

Then comes the tricky part. To test the rest of the files. My plan is to use automation for all of it, using a C# app to scan everything. I got a fast SSD which really helps. It took my computer 11 seconds to make a list of all files in ExoDOS v4. A total of 688475 files. The list generated has the full path to all the file-names.

But what I hope I can do is use a program that is part of DOS32A, which will check files for what type of DOS extender can be found in them, if any.

My hope is I can end up outputting a list of games that use DOS extenders and what type they use, and which file it is in and games that does not have any DOS extenders at all.

[Caldor]

So far so good. I have managed to collect data about the ExoDOS games. Which ones have a DOS4GW file, their total size, which games have files between 250kb and 100mb and the names of those files.

But.. DOS32A is a DOS application. So I cannot run it under Windows 10. I had even found a way to get the output from it. I have found something called vDOS but that does not seem to help much. Like DOSBox it creates a virtual system so I would have to get my program to copy the game to that virtual system before starting it and even then I am not sure I would be able to catch the output from DOS32A this way.

Hmmm... I think I found a simpler method. At least it has worked on several games. I simple look through all files of a certain size for the text: "DOS/4G Copyright (C) Rational Systems".

I ran it on the "smallgames" collection I had made. If this works I will try to convert my data into a CSV file so I can share it here.

[Caldor]

I think it worked. I should probably try to make more than one CSV file, a list of games without DOS4GW files, a list of games with DOS4GW files and such.

https://docs.google.com/spreadsheets/d/ ... sp=sharing

I have tried making it so its easy to filter. The first and last column are probably the most relevant. They contain the name of the game (or at least the zip file) and the last one contains the names of all files that seem to have DOS4GW embedded or maybe it is a DOS4GW file. In the column is an array of the files and their full path.

I did a test, it does not seem it is automatically filterable. But almost, you just need to go to one of the columns, like A, and hover the cursor at the right side of it where a icon with an arrow down should become visible. Click it and click sort sheet A-Z and it should put the whole sheet in a filter mode. Filtering based on Column F will make the DOS4GW games be at the top or not depending on whether you sort A-Z or Z-A.

[flynnsbit]

Nice, good idea looking in the binary for DOS4GW! FYI you can run the images I built in dosbox as well if you just mount the vhd in windows first and then run mount e: F:\ (or whatever drive windows mounted it as. Then you can use is mostly the same way as on the mister.

[Caldor]

I think I might compile this into a tool with some input parameters that can be run in the command prompt. That way it could be used to find other specific files in a collection of games or apps. So there would have to be one folder where all the folders in that folder is expected to be the games or apps. Another option will be specific filenames to search for, and another will be text to look for in these files and then it will output a CVS file.

Because I am thinking there are other DOS extenders, but also specific audio drivers and such that this could help find.

[flynnsbit]

Love it. That would be super useful.

[Caldor]

I got caught up in work yesterday. I will be away for the weekend, but I will try to see if I can get it done Sunday or Monday. If you have any suggestions as to how to output the data I am open to ideas. As a CSV file it should be possible to filter it all and also search for specific games.

But another option could be to make it possible to output BAT files, or even just one big batch file, that assumes it is in a specific folder and then uses runs the stub program on all of the files found to contain the DOS4GW in some way. But... I am pretty sure some of these files / games needs to be excluded though as they probably will either not work with DOS32A or maybe it wont work in AO486 at all anyway.

It would assume a certain folder structure, as I would have to generate it for a specific VHD. That is probably the closest we can come to auto-stubbing it all.

[Voljega]

@Caldor
I can enrich the format of the csv file in my tool with your infos to help keep or pass games during conversion if needed

[Caldor]

@Caldor
I can enrich the format of the csv file in my tool with your infos to help keep or pass games during conversion if needed

You are the one who made the conversion tool for the ExoDOS collection?

[Voljega]

@Caldor yes Iam.
Well I would say I am the one making it as it's not finished yet

[flynnsbit]

@Caldor
I can enrich the format of the csv file in my tool with your infos to help keep or pass games during conversion if needed

You are the one who made the conversion tool for the ExoDOS collection?

He is! https://github.com/Voljega/ExoDOSConverter it's in the name

[Caldor]

Nice

Well, I finished the tool:
http://dionysus.dk/software/DOS/GameInfoTool.zip

So, it has several command line options. But you can just run it in a game collection folder and it should do its thing using default options expecting the folder the tool is run in to be a game collection folder.

Command line options:
-p : Path to folder where the game collection is.
-t : Text to search for in all the files in all subfolders.
-f : Filename to search for in subfolders.

What this tool does is it expects the game collection folder to have a bunch of game folders in it. If there is a folder with a collection of games, it will expect that to be just one game. It outputs a CSV and a JSON file. Not sure if the JSON file is useful. The CSV file has 6 columns.
The first is the name of the game (well, the name of the folder containing the game),
Second column is the path of the game. Third is the total size in bytes. Fourth is the path to the first DOS4GW.EXE file found. (or something else if you changed the -f option)
Fourth is a list of all files between 250kb and 100mb in size.
Fifth column is a list of all files containing DOS4GW... or something else if you changed the -t option.

"I can enrich the format of the csv file in my tool with your infos to help keep or pass games during conversion if needed"
Might make sense to do this. Not sure. Now it should be easy to make collect information about games at least.

Any suggestions for the tool? I can share the source on Github I guess.