Fixing the Secret of Monkey Island Freezing

[user7182]

The Problem:
The Secret to Monkey Island would intermittently freeze. Easy places to reproduce the freezes were in rooms with a lot of sounds:

• The kitchen in the SCUMM bar with the pot bubbling, keg pouring, and voice speaking.
• The bottom of the boat with water dripping and voice speaking.
• Many of the cut scenes.

Investigation:
I was able to reproduce the problem in a debugger and found the game was stuck in an infinite loop waiting for sound to stop playing.
The logic in the game is:

Code: Select all

if (voice_playing) {
    ctv_terminate() // TERMINATE_PROCESS=8 in CT Voice driver
    while (voice_playing) {} // <--- This is the loop it was frozen in.
}
ctv_output(); // OUTPUT_VOICE=6 in CT Voice driver

The `voice_playing` variable is set when calling `ctv_output()`, and then cleared when the Sound Blaster IRQ executes.

What I found was the game would freeze when starting a new sound before the previous one finished. When this happens the game expects to provide a certain length of sound bytes, but the sound_dsp module in the AO486 core was still counting down bytes from the previous sound.

Because AO486 core never received all the bytes from the first sound, it never triggered the IRQ to clear the `voice_running` flag, and the game freezes, stuck waiting forever for the sound to finish.

For example in the SCUMM bar kitchen the game would play bubble sound effects from the pot:

1. Game requests output of a 7,430 byte block
2. DSP signals IRQ after 7,430 bytes.

The bubble sounds run in a loop, but can be interrupted by interacting in the room. When filling a cup from the keg:

1. Game requests output of a 35,786 byte sound block to play.
2. With 8,693 bytes (0x21F9) remaining of the active sound, the game requests a bubble sound of 7,430 (0x1D06) bytes.
3. Both the game and the AO486 are deadlocked after 7,430 bytes are played following the interruption of the bubble sound. AO486 expects another 1,267 bytes that never come.

Solution
I've updated the sound_dsp module in a fork of the AO486 core which updates the DMA length of expected bytes when a new command is received, rather than waiting for an active sound to finish. I believe interrupting the current sound is in line with what the DOS sound driver is expecting.

https://github.com/user7182/ao486_MiST ... e293847a7a

With this change I was able to play The Secret of Monkey Island from start to finish without it freezing.

I plan to make a pull request to the main repository, but I don't have a lot of confidence I didn't introduce any new problems. I published a pre-built core RBF file in the hopes a few more people could test it in other games they know work, or with other games that also froze.

https://github.com/user7182/ao486_MiSTe ... 6_20220102

[wark91]

Thank you for proposing this fix!
I made some tests on games I played in ao486 (Nicky Boom, Civilization, Colonization and the Settlers) and I don't see any regression.

[monsterchief]

Monkey Island 2 Ultimate Talkie Edition used to crash for me too but this fork of AO486 has fixed it! Excellent work!

[Moondandy]

Good effort for digging into this issue and finding a fix! What version of the game were you playing? I had a lot of freezes playing the talkie version and using my MT-32 Pi, would be awesome if this fixed those.

[flynnsbit]

No glaring issues. I went through a few mins of gameplay in each of the games in the Top 300. The only interesting change between yours and AO486 21-10-10 is with Discworld 2. Discworld 2 didn't work on the core anyway as it has a screeching audio problem when on the main menu or at different times in the game. The interesting thing is that in your build you only get audio from the initial PSY Owl intro boot screen and then nothing after that. That is the only repeatable issue that I can see between the official build and yours.

I'll keep smoke testing when I have time but initial tests look good.

[HarborSeal]

Thank you. I can confirm that I can play the SOTM talkie version all the way through without freezing.

This fix also helps MI2 talkie version as well, but it does freeze at the very end when the big whoop amusement park employee yells at the kids. I'm using the Top 300 pack with HIMEM enabled.

[monsterchief]

Thank you. I can confirm that I can play the SOTM talkie version all the way through without freezing.

This fix also helps MI2 talkie version as well, but it does freeze at the very end when the big whoop amusement park employee yells at the kids. I'm using the Top 300 pack with HIMEM enabled.

I second this result. I loaded up a save file and the game doesn't load the park employee speech after u rip Lechuck's leg off. you have to press "." to skip his dialogue to continue. Is this fixable too?

[virtuali]

I had high hopes for this fix to solve the freezing problem in Falcon 3.0, which is included in the Top 300 compilation, SEEMS to load fine, but it freezes 10-20 seconds after starting a flight.

The only thing uncommon about Falcon 3.0, is the "hi-fi" flight model required a math co-processor, which is not supported by the AO486 core. However, it should work in a way that, if there's no FPU available, the flight model should revert to the lower fidelity options ( Complex and below ) but, to be sure, I set it to Complex anyway before starting a flight.

Unfortunately, the game still freezes even with your fix so, it might not be sound-related. Is too much to ask if you could take a look at it ? Perhaps some issue related to the way the FPU is detected ? The FPU requirement for the hi-fidelity flight model is the only thing uncommon about this game, other than it required a fairly high amount of conventional memory, but this doesn't seem the issue, since it won't even start with not enough memory.

[docbobo]

It does seem to fix a problem with Jill of the Jungle. Tried three times without the patch - hangs every time. Three times with the patch - works.

[user7182]

Thank you. I can confirm that I can play the SOTM talkie version all the way through without freezing.

This fix also helps MI2 talkie version as well, but it does freeze at the very end when the big whoop amusement park employee yells at the kids. I'm using the Top 300 pack with HIMEM enabled.

I second this result. I loaded up a save file and the game doesn't load the park employee speech after u rip Lechuck's leg off. you have to press "." to skip his dialogue to continue. Is this fixable too?

I'm playing MI2 for the first time now. let me get up to the point where it freezes and I'll check it out.

[user7182]

No glaring issues. I went through a few mins of gameplay in each of the games in the Top 300. The only interesting change between yours and AO486 21-10-10 is with Discworld 2. Discworld 2 didn't work on the core anyway as it has a screeching audio problem when on the main menu or at different times in the game. The interesting thing is that in your build you only get audio from the initial PSY Owl intro boot screen and then nothing after that. That is the only repeatable issue that I can see between the official build and yours.

I'll keep smoke testing when I have time but initial tests look good.

I made a change to fix this problem. Stopping the current sound when any DSP command is received wasn't the right way to go. I still don't think I've arrived at the correct fix, but this is less of a regression.

https://github.com/user7182/ao486_MiSTe ... 6_20220105

[monsterchief]

I made a change to fix this problem. Stopping the current sound when any DSP command is received wasn't the right way to go. I still don't think I've arrived at the correct fix, but this is less of a regression.

https://github.com/user7182/ao486_MiSTe ... 6_20220105

It's fixed! That part in MI2 works now.

[HarborSeal]

It's fixed! That part in MI2 works now.

Yep. There's no freezing anymore at that point in the game..

[user7182]

I had high hopes for this fix to solve the freezing problem in Falcon 3.0, which is included in the Top 300 compilation, SEEMS to load fine, but it freezes 10-20 seconds after starting a flight.

The only thing uncommon about Falcon 3.0, is the "hi-fi" flight model required a math co-processor, which is not supported by the AO486 core. However, it should work in a way that, if there's no FPU available, the flight model should revert to the lower fidelity options ( Complex and below ) but, to be sure, I set it to Complex anyway before starting a flight.

Unfortunately, the game still freezes even with your fix so, it might not be sound-related. Is too much to ask if you could take a look at it ? Perhaps some issue related to the way the FPU is detected ? The FPU requirement for the hi-fidelity flight model is the only thing uncommon about this game, other than it required a fairly high amount of conventional memory, but this doesn't seem the issue, since it won't even start with not enough memory.

I looked at Falcon 3 from the perspective of the sound card. The game outputs 8-bit PCM data directly to the DAC (DSP command 0x10), so none of the changes I've made are going to effect that. I imagine that makes this game fairly CPU intensive. The game takes up too much memory so there isn't enough room to load a debugger to see what it is was doing when it froze.

I did find if I go to the Config/System Options menu and reduce the detail to Medium I stopped experiencing any crashes. Not a satisfying answer, but that's what I know.

[user7182]

The root cause for Monkey Island was not what I originally thought. The AO486 sound module was correctly playing all the samples it was asked to play from start to finish (DSP Command 0x14). My change to stop the current output when a new time constant command (0x40) worked because I was creating a pause event that should have come from the game, but was missing. Monkey Island (or really the CT-VOICE driver built into it) incorrectly started sending samples from a new sound without first pausing the current output.

The way it's supposed to work is the game checks to see if a sound is playing, and if it is, then it sends a pause command (0xD0) before sending the play command (0x14). If the game doesn't believe theres a sound playing it just issues the play command (0x14).

To check whether a sound is playing the game polls the write status port (address 2xCh), and when it sees the busy flag set it will issue the pause (0xD0). The game seems to poll the port thousands of times waiting for a busy flag bit set.

The code from the game that should have sent the pause looks like this:

Code: Select all

{  
  port = (2xC, write buffer status)  
  retry_count = 65535;
  while( true ) {
    if (voicePlaying == none) {  // 0 = none, 1 = output, 2 = input
      return;
    }

    /* Read the write port status */
    write_port_status = in(port);
    
    /* If bit 7 is set, we need to send a pause, break out. */
    if (write_port_status & 0x80) break;
    retry_count--;
    if (retry_count == 0) {
      return;
    }
  }
  
  /* Wait for the write port not to be busy, then send a pause command. */  
  do {
    write_port_status = in(port);
  } while (write_port_status & 0x80);

  /* Send pause command */
  out(port, 0xd0);
}

The DSP normally runs on an 8051 microcontroller, and the busy flag would be set when the timer interrupt triggers to handle each sample. The AO486 sound module doesn't really require the busy bit as flow control, but because the software expects the busy time to be there it needs to be replicated.

Here's a build if anyone wants to try it:
https://github.com/user7182/ao486_MiSTe ... 220122.rbf

It should work as well as the last version at fixing the freezing, but I believe it's addressing the real underlying issue.

[flynnsbit]

The root cause for Monkey Island was not what I originally thought. The AO486 sound module was correctly playing all the samples it was asked to play from start to finish (DSP Command 0x14). My change to stop the current output when a new time constant command (0x40) worked because I was creating a pause event that should have come from the game, but was missing. Monkey Island (or really the CT-VOICE driver built into it) incorrectly started sending samples from a new sound without first pausing the current output.

The way it's supposed to work is the game checks to see if a sound is playing, and if it is, then it sends a pause command (0xD0) before sending the play command (0x14). If the game doesn't believe theres a sound playing it just issues the play command (0x14).

To check whether a sound is playing the game polls the write status port (address 2xCh), and when it sees the busy flag set it will issue the pause (0xD0). The game seems to poll the port thousands of times waiting for a busy flag bit set.

The code from the game that should have sent the pause looks like this:

Code: Select all

{  
  port = (2xC, write buffer status)  
  retry_count = 65535;
  while( true ) {
    if (voicePlaying == none) {  // 0 = none, 1 = output, 2 = input
      return;
    }

    /* Read the write port status */
    write_port_status = in(port);
    
    /* If bit 7 is set, we need to send a pause, break out. */
    if (write_port_status & 0x80) break;
    retry_count--;
    if (retry_count == 0) {
      return;
    }
  }
  
  /* Wait for the write port not to be busy, then send a pause command. */  
  do {
    write_port_status = in(port);
  } while (write_port_status & 0x80);

  /* Send pause command */
  out(port, 0xd0);
}

The DSP normally runs on an 8051 microcontroller, and the busy flag would be set when the timer interrupt triggers to handle each sample. The AO486 sound module doesn't really require the busy bit as flow control, but because the software expects the busy time to be there it needs to be replicated.

Here's a build if anyone wants to try it:
https://github.com/user7182/ao486_MiSTe ... 220122.rbf

It should work as well as the last version at fixing the freezing, but I believe it's addressing the real underlying issue.

Congrats on the commit to the master branch! What's next for AO486?

The DOS Protected Mode Interface (DPMI) bug in the core is a pretty big one that no one has been able to figure out. It affects many many games that only bomb on the core and not in emulators or real machines. For some reason, DOS32a works around it most of the time. https://github.com/MiSTer-devel/ao486_MiSTer/issues/45

Any game referenced here had the issue with dos4gw and would bomb in the core without the switch. I have a feeling this would fix many many things if found. https://github.com/flynnsbit/Top300_upd ... h?q=dos32a

[luishg]

Congrats on the commit to the master branch! What's next for AO486?

The DOS Protected Mode Interface (DPMI) bug in the core is a pretty big one that no one has been able to figure out. It affects many many games that only bomb on the core and not in emulators or real machines. For some reason, DOS32a works around it most of the time. https://github.com/MiSTer-devel/ao486_MiSTer/issues/45

Any game referenced here had the issue with dos4gw and would bomb in the core without the switch. I have a feeling this would fix many many things if found. https://github.com/flynnsbit/Top300_upd ... h?q=dos32a

+1000 to this. Dos4gw is currently the most important difference with any of the original machines. You can run much more complex software like Windows with no problem, but fail to run several DOS classic games that rarely would fail on real HW.

[user7182]

Congrats on the commit to the master branch! What's next for AO486?

The DOS Protected Mode Interface (DPMI) bug in the core is a pretty big one that no one has been able to figure out. It affects many many games that only bomb on the core and not in emulators or real machines. For some reason, DOS32a works around it most of the time. https://github.com/MiSTer-devel/ao486_MiSTer/issues/45

Any game referenced here had the issue with dos4gw and would bomb in the core without the switch. I have a feeling this would fix many many things if found. https://github.com/flynnsbit/Top300_upd ... h?q=dos32a

This bug was fun to investigate, it took a lot longer to track down then I expected it would. This is the first time I've done any development with an FPGA, so there was a bit of a learning curve.

I started looking into why DiscWorld II sounds like it has noise mixed in with real background music when you get to the menu. I was hoping it would be a problem in the same area of code I'm familiar with now, and might be an easy fix. As far as I can tell the movie's that play before the menu are the same sample rate, same mixer settings, and use the same autoinit commands. There's no obvious difference between audio that sounds good and audio with static, so far not an easy fix .

I know next to nothing about DPMI (but I guess that's a starting point). It's good that it's very easy to reproduce. It's going to be hard to debug if DOS4GW is a black box where I don't know what I expect it to do. If there are programming guides for DOS4GW it might make sense to make an application that can reproduce the problem. We may be able to write a memory test that could narrow down the problem if its related to extended memory.

[HarborSeal]

I know next to nothing about DPMI (but I guess that's a starting point). It's good that it's very easy to reproduce. It's going to be hard to debug if DOS4GW is a black box where I don't know what I expect it to do. If there are programming guides for DOS4GW it might make sense to make an application that can reproduce the problem. We may be able to write a memory test that could narrow down the problem if its related to extended memory.

This may be a good place to start.

https://web.archive.org/web/20180628101 ... index.html

[flynnsbit]

I think it is the way memory is handled by the core, but couldn't quite figure out why specifically the 16MB threshold caused issues. I think Sorg was onto something with this comment:

"memory limit is based on BIOS Int 15h return. Regardless 16MB or 256MB option, a whole 256MB is available always. If you want to have other memory limits, then you can try to alter BIOS sources to report required amount."

So if we could figure out how to truly limit the memory to 16MB then that might solve the DPMI issue as DOS4GW was never planned to use more than that. The weird thing is that dos4gw has a limit of 64MB but for some reason things start breaking down at the 16MB size.

Anyway, it was super cool of you to find and fix that bug with Monkey Island. That is probably the most popular game played on the core.

[Caldor]

Generally this core expects the memory stuff to be used... "correctly"... from what I understand DOS4GW was often not very good, especially the old versions.

If this is using DOS4GW in any way, has DOS32A been tested? It usually gets around the memory issues.

[ToothbrushThreepwood]

I’m just in complete awe. I mean, it’s one thing to boot up your computer and try to figure out how to beat a game, but doing it while casually figuring out how to beat the architecture of your computer at the same time is more than impressive.

[flynnsbit]

Generally this core expects the memory stuff to be used... "correctly"... from what I understand DOS4GW was often not very good, especially the old versions.

If this is using DOS4GW in any way, has DOS32A been tested? It usually gets around the memory issues.

I don't think that is right. I don't think the core IS using the "memory stuff" correctly.

Right, we know DOS32A works around it. The goal is to figure out why these version of DOS4GW that come with the games don't. They work on every other emulator with the exact same vhd/memory/even bios as close as I can get to it with PCem and 86Box. Every one of my 7 real DOS machines also work fine.

If we find this, it could be a underlying problem that not only effects DOS4GW but other applications that are not using protected mode and are manifesting an issue or crashing. Those would not be fixable with dos32a...

[Caldor]

I noticed this got pushed to the Main branch... awesome Thanks for sharing this fix.

Generally this core expects the memory stuff to be used... "correctly"... from what I understand DOS4GW was often not very good, especially the old versions.

If this is using DOS4GW in any way, has DOS32A been tested? It usually gets around the memory issues.

I don't think that is right. I don't think the core IS using the "memory stuff" correctly.

Right, we know DOS32A works around it. The goal is to figure out why these version of DOS4GW that come with the games don't. They work on every other emulator with the exact same vhd/memory/even bios as close as I can get to it with PCem and 86Box. Every one of my 7 real DOS machines also work fine.

If we find this, it could be a underlying problem that not only effects DOS4GW but other applications that are not using protected mode and are manifesting an issue or crashing. Those would not be fixable with dos32a...

I remember when a lot of things were getting added to AO486 and it began being possible to run Doom and such at playable speeds with this core, I was researching a lot of things setting up DOS and trying out different new tools like DOS32A, all the FreeDOS stuff, new ways to optimize memory with some very optimized drivers and I ended up finding a lot of interesting posts and documents. I think some of it was about DOS32A and why it existed and what its goal was.

Apparently most games were released with some old versions of DOS4GW that essentially had bugs in them. So I think in many cases, it would actually be enough to replace DOS4GW in one game with the newest version of DOS4GW if found. I already made a tool that could find all files in a folder and all its subfolders that had DOS4GW in it, to try to help figure out what games would make use of DOS32A, because I thought it would actually solve all problems, or at least that every game that used DOS4GW would be better off with DOS32A. But I have later found that some games do not work with DOS32A and needs DOS4GW.

What I also read was that the reason most games used such old versions of DOS4GW was because of how expensive it was. One problem we know very well that it has, is that it was early on build for machines with 16MB RAM or less, because machines with more than that did not exist at the time anyway. At least not home computers. But that was just one of the strange memory problems that DOS4GW did not handle correctly.

Vogons is already a good source for DOS information. I found this thread which has some stuff about DOS4GW:
https://www.vogons.org/viewtopic.php?t=69852

I also found the place that went into how most games have old versions of DOS4GW and says that the latest version is 2.01a but also that it costs 50 dollars:
https://maximumhoyt.blogspot.com/2008/1 ... ative.html