MiSTer Storage Requirements

[Lamaman1971]

I plan to use my Mister for retro computing - Amiga mainly. Just ordered the parts but left pondering the storage requirements. Would you all recommend a single big SD card or a smaller SD supplemented with a USB hard drive?

[ZigZag]

A big, fast MicroSD. I found HDD speed disappointing.

[jlancaster86]

If you have a NAS, you can setup a "MiSTer" SMB share and use that (that's what I do).

[Bas]

Get a big SD. If you're mainly going for Amiga, you'll have plenty of space on anything bigger than 64GB. Big storage mainly comes into play when you start loading CD-ROM images, which isn't really front and center on the Amiga platform. I started my MiSTer with an 8GB card and was actually fine for the most part except CD's. Now I'm on 64GB and have room to spare.

[bazza_12]

defo if you're only using amiga use get a big SDcard.. my amiga folder is 8gig ish.. that includes MegaAGS, Shapeshifter, kickstart roms and 21meg of demos. I agree with Bas above.. 64 gig+

[Bas]

Regarding eggs and baskets.. SD isn't exactly reliable storage, so make backup copies of anything valuable.

[kathleen]

A big, fast MicroSD. I found HDD speed disappointing.

Is using an external SSD HDD could help the speed concern ? Or is it due to the USB transfer ?

[ZigZag]

I think it's USB hub speed. Could just be my set up, but I find MicroSD faster than an external SSD. It's noticeable mostly with cores like PC & Amiga. This is just my personal experience though, and I haven't updated in a few months.

[Solskogen]

I've found that ethernet is faster than USB and SD card, so I use a small/cheap sdcard for the cores and data on SMB(NAS).

[RealLarry]

I've found that ethernet is faster than USB and SD card, so I use a small/cheap sdcard for the cores and data on SMB(NAS).

An ethernet connection is faster. That's why I'm using an old and decommissioned mini PC for MiSTer's data, which is booting up a stripped down Debian for max. speed and greatest flexibility. I've tested 256GB SD cards and ready-to-run NAS' but wasn't ever satisfied. Better building such things by myself, also in terms of data privacy and trustworthiness.

[akeley]

I'm fine with my SD card's speed. I mean, how much faster can the other methods be, when you consider the kind of data we're shifting?

[Bas]

The LAN interface is 100Mbit right? That caps the network at around 12MB/s throughput, which is faster than just about anything you can expect from a retro core except maybe big box Amiga (which MiSTer doesn't have) or AO486. Any faster won't be useful until you reach the extremes like Saturn or Playstation, but probably not even those.

[elvis]

The LAN interface is 100Mbit right?

The DE10nano's Ethernet port is gigabit, full duplex.

The USB port is USB2, and the issue that USB2 is half duplex. For storage, that's a larger impact than its 480Mbit/s bandwidth.

I already had a NAS configured for use centrally in my house across multiple gaming systems, modded consoles, arcade machines, TVs, etc. So that was a simple choice for MiSTer. The speed is certainly nicer than either SD cards or USB2 connected storage.

Additional features like transparent compression, deduplication, snapshots and redundancy courtesy of BtrFS are all nice too.

[aberu]

I agree with most people here, a big MicroSD is still pretty cheap. A 128GB if you are primarily just doing Amiga and some other stuff. If you get into AO486, then 256GB IMO. If you need more you just plop the extras on a usb stick and navigate to the extra files there.

[Bas]

Elvis, did you benchmark the speeds?

[bazza_12]

Elvis, did you benchmark the speeds?

he's left the building...




sorry.. couldn't resist

[elvis]

Elvis, did you benchmark the speeds?

Using iperf3 for raw TCP performance in both directions:

MiSTer to NAS:

Code: Select all

/root# iperf3 -i1 -c 192.168.3.254
Connecting to host 192.168.3.254, port 5201
[  5] local 192.168.3.47 port 57584 connected to 192.168.3.254 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec  75.0 MBytes   623 Mbits/sec    0    167 KBytes       
[  5]   1.01-2.01   sec  73.8 MBytes   619 Mbits/sec    0    177 KBytes       
[  5]   2.01-3.01   sec  75.0 MBytes   631 Mbits/sec    0    177 KBytes       
[  5]   3.01-4.01   sec  75.0 MBytes   630 Mbits/sec    0    177 KBytes       
[  5]   4.01-5.01   sec  75.0 MBytes   629 Mbits/sec    0    177 KBytes       
[  5]   5.01-6.01   sec  75.0 MBytes   630 Mbits/sec    0    177 KBytes       
[  5]   6.01-7.01   sec  73.8 MBytes   618 Mbits/sec    0    177 KBytes       
[  5]   7.01-8.00   sec  75.0 MBytes   631 Mbits/sec    0    177 KBytes       
[  5]   8.00-9.00   sec  75.0 MBytes   632 Mbits/sec    0    177 KBytes       
[  5]   9.00-10.02  sec  76.2 MBytes   630 Mbits/sec    0    177 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec   749 MBytes   627 Mbits/sec    0             sender
[  5]   0.00-10.05  sec   749 MBytes   625 Mbits/sec                  receiver
iperf Done.

NAS to MiSTer

Code: Select all

/root# iperf3 -i1 -R -c 192.168.3.254
Connecting to host 192.168.3.254, port 5201
Reverse mode, remote host 192.168.3.254 is sending
[  5] local 192.168.3.47 port 57588 connected to 192.168.3.254 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  85.6 MBytes   718 Mbits/sec                  
[  5]   1.00-2.00   sec  85.1 MBytes   714 Mbits/sec                  
[  5]   2.00-3.00   sec  87.1 MBytes   730 Mbits/sec                  
[  5]   3.00-4.00   sec  86.1 MBytes   722 Mbits/sec                  
[  5]   4.00-5.01   sec  87.0 MBytes   726 Mbits/sec                  
[  5]   5.01-6.00   sec  87.9 MBytes   738 Mbits/sec                  
[  5]   6.00-7.01   sec  84.9 MBytes   712 Mbits/sec                  
[  5]   7.01-8.01   sec  87.4 MBytes   732 Mbits/sec                  
[  5]   8.01-9.00   sec  86.7 MBytes   732 Mbits/sec                  
[  5]   9.00-10.00  sec  86.4 MBytes   724 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   867 MBytes   725 Mbits/sec  222             sender
[  5]   0.00-10.00  sec   864 MBytes   725 Mbits/sec                  receiver
iperf Done.

And a real-world test: Here's my CIFS mount with my ROMS. 235MB of files in this particular directory. I use the Linux command "cat" to stream them over the CIFS mount, pass them through a tool called "pv" (short for "pipe view") which just reports the transfer speeds, and then throws them away (passed to /dev/null). This is the same as reading files into memory on MiSTer.

Before any disk-read tests, I send "3" to a special file called /proc/sys/vm/drop_caches. This tells Linux's local file system cache to invalidate all entries and empty the cache, forcing a new read from the backing storage. Note that this doesn't clear the cache on the remote system (i.e: my NAS still has it's own cache, independent of clients connected).

Code: Select all

/media/fat/SNES/1 US - A-E# echo '3' > /proc/sys/vm/drop_caches ; cat * | pv >/dev/null
 235MiB 0:00:05 [45.8MiB/s] 
 

45.8MB/s, or around 400Mbit/s real world. My NAS's storage is much faster than that (can quite happily saturate a couple of 1GbE PCs in the house while feeding media to my TVs - storage was idle at time of benchmark), so the bottleneck there is definitely the DE10-Nano itself.

Let's compare reading off the MicroSD card I have. It's not particularly impressive or high speed, a SanDisk Ultra 64GB Micro SDXC UHS Speed Class 1, rated to 10MB/s. Using MiSTer's local Linux image:

Code: Select all

media/fat/linux# echo '3' > /proc/sys/vm/drop_caches ; cat linux.img | pv >/dev/null
 312MiB 0:00:14 [22.3MiB/s]

Exceeds the minimum 10MB/s guarantee by a factor of 2, but still slower than my NAS. I don't have faster MicroSD cards to test, as again I can't be stuffed buying them when my NAS has more convenient features than just raw speed. So I'm not sure what the maximum speed through the MiSTer's MicroSD slot is.

[drpaneas]

Easier thing is to use an SD Card.

But I second what @Elvis said -- the network connection is faster. I did lot's of benchmarks with iperf3 and game streaming (using Moonlight -- this is outside of the context of Mister though) and the speed was mindblowing, even with 5G wifi. If you can setup a network storage, go for it

[german_user]

Only or mainly for the Amiga Core I would stick with an SD card. I am currently using inexpensive 64GB cards myself, these cost less than € 4 each and are relatively easy to copy. CD32 games isos are the real memory hogs, the same games as the A1200 version rarely use more than 200MB.

[elvis]

PorkChopExpress from MiSTeraddons has published a great breakdown on MiSTer storage. Confirming my previous findings that wired Ethernet is by far the fastest option for storage.

https://misteraddons.com/blogs/news/mis ... d-shootout

I've got a project in the works to assist people with configuring network storage for MiSTer. When that's ready for public consumption, I'll make a separate post.