Slow Write Speeds When Using External HDD

[akeley]

I 've started using an external HDD as storage for some cores not long ago. It's a normal disk, not an SSD - a few years old 2TB Seagate. I put CD images for PSX, Mega CD and PCE on it and they all work ok.

A few days ago I've also migrated ao486 there, basically had copied the whole ao486 directory from the SD as-is, plus added some new vhds. Everything seemed to work okay - my old games started and ran fine - but after a while I observed that keyboard response can be a bit sluggish at times and, more importantly, the write speeds are now very, very slow. Simply trying to copy Doom v1.1 from one vhd to another took over 15 mins, and installation over 30. That same operation on my SD card was completed under 5 min.

Do you think it might be the fault of the HDD not being an SSD, or is it something to do with configs and/or using external storage in general?

[Solskogen]

The USB is quite slow on the De10-Nano. That's the reason.

[akeley]

Are you sure? I don't know if I'm reading this right, but looking at this test by Porkchop, USB is not that far off the SD card's speeds. Even assuming the slowest write speed he mentions, ~1MB per second, my Doom example should've been accomplished in much less time than the nearly one hour it took (in fact I got bored when it was unpacking the wad, taking forever, and I switched it off, so maybe it'd be even longer).

[WILL1384]

I can think of a few things that limit the speed of an external hard drive on the MiSTer FPGA:

USB 2.0, the MiSTer FPGA is USB 2.0

Power, "some" external hard drives need a lot of power, the MiSTer FPGA hub only seems to put out enough power for the average dongle, mouse or keyboard.

A lot of "portable" USB hard drives are slow by nature because they were intended to be used has "backup devices" only, and some of them cheat to get faster speeds by having high-speed cache memory that quickly fills up and then the portable USB hard drive slows back down to turtle like speeds.

That being said even the worst portable USB hard drive should still get speeds close to, or likely far better than IDE seeds on a 486 computer, I know this because I ran a computer in the early 2000s that had a USB 2.0 to IDE adapter for it's main drive and it ran just fine, at the time I needed a drive that I could quickly take with me for security reasons, and yea it worked just fine, and it was a bit slower at running things, but it was only a few seconds slower at most.

Have you tried a different USB hard drive?

[akeley]

Obviously, I don't expect some blazing speeds, but I've just tried it again on a similar portable HDD and copying a 20MB directory from one vhd to another in Minimig took over 15 minutes. Also, when I open the OSD during this process, I can barely move between the options, and the scrolling text slows down massively - it's as if the whole system is choked.

If that's an expected behaviour then so be it, but I'd appreciate if somebody who had actually tried doing this too could confirm it. Like I said, I haven't done anything else apart from making the relevant directories on my PC and then plugging the HDDs into MiSTer, so maybe I'm missing some necessary step.

[jca]

I made a test using ao486 running Win95 and copied a 15MB directory from one vhd to another vhd. It took between 2 and 3 min. During the copy the OSD behaved as usual, no slowdown at all.
I use a 1TB WD Elements.

[akeley]

Thanks for trying...that speed sounds much more reasonable. Did you do anything particular for this setup to work, or maybe could think of something I should try?

[catallo]

I had the same problem with an external USB drive recently. For me, the problem was NTFS. After I reformatted the drive using FAT32 it ran significantly faster.

[jca]

Drive formatted ext4.
Everything (almost) on USB drive with symlinks from SD card to USB HD.
But basically nothing particular and I cannot thing of anything regarding your problem.

Edit: in the past I used exFAT and NTFS without any problem I can remember. At least I did not get horrific write speed.

[Beeble]

I had super slow FTP transfer speeds with ntfs on the external SSD, changed to Fat32 or exfat, improved write speed via FTP a lot.

[akeley]

Thanks, the file system thing seems a likely culprit. I will try to reformat in FAT32 later, though it means nuking/repopulating about half a terabyte of games already on it...

I've never done any symlinks thou, but the PSX and other game games worked ok.

[jca]

The only directories on /media/fat are:
config, linux and Scripts.
The only files I have on /media/fat are:
downloader.ini, menu.rbf, MISTer and MISTer.ini.
Everything else is on /media/usb0 with symlinks from /media/fat to /media/usb0.

[akeley]

Ok, but how do I make these symlinks / where to put them / is it necessary?

[jca]

If you only have the games directory on USB you do not need to do anything. It seems that some users put some games subdirectories directly on USB, for example /media/usb0/PSX but I don't know anything about that.
Initially I was using "Switch to USB" until Sorge said it was kind of obsolete and one should use symlinks.
After playing around my final setup is as mentioned above.
To create the symlinks you can use WinSCP:
On the right panel go to /media/fat, use New -> Link, name the link (ex: _Arcade), say where it points to (ex: ../usb0/_Arcade), do NOT select Hard link and click OK.
Now if you double click on _Arcade while the right panel is still in /media/fat you will be redirected to /media/usb0/_Arcade.
In a sense it is like if you had everything on the SD card, for example my /media/fat/downloader.ini contains base_path = '/media/fat/'.
There are 2 exceptions: Mame Getter and HbMame Getter do not like it and I had to create 2 ini files in /media/Fat/Scripts (I kept the Scripts on the SD card):
update_mame-getter.ini
ROMMAME="/media/usb0/games/mame"
MRADIR="/media/usb0/_Arcade"
update_hbmame-getter.ini
ROMHBMAME="/media/usb0/games/hbmame"
MRADIR="/media/usb0/_Arcade"
This setup revealed a small glitch with MISTer: contrary to the case where one use "Switch to USB" and the menu is not displayed before the USB drive is mounted the menu is displayed right away after boot and as the drive is not yet mounted it displays "No Files". You have to wait until the drive is mounted hit F12 twice to go back and forth for the menu to display Arcade, Computer, ...
Having symlinks solve also a problem with Minimig which does not like very much USB:
Even if you have only the games directory on USB and still have the games directory on the SD card you can delete the /media/fat/games directory from the SD card and create a symlink from /media/fat/games to /media/usb0/games to fix the problem with Minimig.
The problem with Minimig is that when the games directory is on USB you have to create the Amiga directory in /media/usb0/Amiga and not /media/usb0/games/Amiga and you can have only one KisckStar rom which must be /media/usb0/KICK.ROM. By deleting /media/fat/games and creating instead a symlink you trick the core as is everything was on the SD card. Now you can have /media/usb0/games/Amiga and use any KickStart rom with any name you wish. It necessitate that all your games subdirectories are on USB, if you have some in /media/usb0/games and some on /media/fat/games you have to keep both directories and cannot create the symlink.
Using symlinks you also do not need to put everything on USB like I did: you can choose what you want to move to USB and create symlinks on the SD card only for what you moved.
I hope my post is not too convoluted and don't hesitate to ask if it not clear.

[akeley]

Thanks for the write up. Although I'm not sure if using symlinks (or the switch to USB thingy) is still necessary, seeing as the cores I have tried "just worked" for me (with the exception of slow write) without making any. Basically, all I did was to make the games/core directories on that HDD with and moved roms/vhds/isos there. All the main mister stuff remains on the SD, plus most of my other cores with smaller sized libraries.

Yesterday I formatted my HDD in exFat and after that was able to copy & install Doom in a0486 under 10 minutes - so this definitely affects write speeds. There was still a slight lag when moving in the OSD, but not as bad as before. Have to test it a bit more today, see how it all behaves.

[jca]

If you just moved the games directory to the USB drive you do not have anything to do. The thing I am curious about is in regard of the Minimig core. Where is your Amiga directory located and do you have any problem with the KickStart roms?

[akeley]

The thing I am curious about is in regard of the Minimig core. Where is your Amiga directory located and do you have any problem with the KickStart roms?

It's just /media/usb0//games/Amiga. I haven't tested it extensively yet, but so far several kickstarts I've tried seem to work ok (3.1, 1.3 & MegaAGS Kickstart).

[jca]

Thanks for the info. It must have been fixed at some point.