Game/ROM List for all Cores

[JWDog]

I am in development of the ultimate game/rom list-database for all cores currently on the MiSTer. This list includes meta-data for each title. I am not looking to create a frontend for the MiSTer, but a way to have a paper and/or electronic copy of all games and notes regarding their settings and setup. I know the meta data is spread around to multiple websites, with some websites that contradict each other in data, so wanted to have a discussion on which websites are best to find lists of games for the cores and which have the best metadata available. Also, if there are multiple releases to a title which there are... what are the best sources to find info on these releases?

A good example of multiple release is 10-Yard fight for NES. So far I have found 5 different cart releases. Maybe there are more out there. Please help if anyone knows.

10-Yard Fight, IF-02, Japan
10-Yard Fight, NES-TY-USA, USA
10-Yard Fight, NES-TY-CAN, Canada
10-Yard Fight, NES-TY-GBR, UK (Great Britain)
10-Yard Fight, NES-TY, front of cart doesn't have a serial number, any one know the region?

Also, does anyone know how roms are related to these carts, trying to understand that the rom chip inside carts for example could be the same for multiple cart releases.

Also, if anyone has a better way of having this type of database let me know if I am just creating the wheel. Then it might be easier to pay for something rather then create one.

Sorry for multiple questions just trying to wrap my head around this project.

Anyway, thoughts and ideas are appreciated.

[aberu]

DAT-O-Matic DATfiles and Redump (for CDs) DATfiles are the best sources for this information typically.

[akeley]

Good luck, but see you in ~100 years

Seriously though, it's mission impossible if you really want to cover every game on every core. There are ~30K games for the C64 alone.

Trying to make descriptions for all of them seems a bit pointless too: most of console games will just work, without the need for special setups. With microcomputers it's bit different, but most games will also fall into few startup categories, even the more convoluted ones such as MSX or Atari 8-bit.

If you just want ~complete lists to see what's what, for microcomputers I recommend Gamebase collections, since they are one-game-one-rom without fluff and with lots of data. For consoles, No-Intro (these have dedicated .dats you can manipulate with clrmamepro) or Htgdb packs.

[aberu]

You could probably create descriptions for them automatically from the datfiles, honestly. As long as you automate a script to process the datfiles.

[akeley]

That depends what OP wants to include in the "descriptions". I don't think the standard dats (TOSEC et al) contain anything more than what's in the rom's name already (year, publisher, tags, etc)+checksums. I think some do include a short hint on how to run a file (eg VIC 20 carts), but not always.

It's of course one way to do it, but the biggest problem is that these lists will snow you in with a bazillion available game versions. There are 50+ different Airborne Ranger C64 ones, for example. I think some dat frontends offer 1G1R filter options but am not sure how well it works. On the other hand Gamebase, and other quality packs, have been already manually filtered.

[JWDog]

Thank you for reminding me of dat files, they have some good info. I am mostly mixing information from different sources (websites) to encounter the meta data needed for good identification of the games. Here are some examples of websites I am using: MobyGames, LaunchBox, Video Games Museum, and The Games DB. However, like I said I am encounter many inconsistencies in data for example release dates, publishers, designers, etc.

Akeley, what is the purpose of 50+ different Airborne Ranger C64, mostly hacks? or actual versions updated due to bugs or enhancements? releases in different regions? translations?

My list of metadata includes:
Game Title (usually in English or transliteration of original language in english)
Game Alternative Title
Game Region Title (Game title in original regional language) for example famicom games are in Japanese
Game Serial Number
Game Release Date
Game Number of Max Players
Game Is Cooperative?
Game Controls
Game Extra Peripherals
Game Released Region
Game Box Cover Art
Game Media Picture (ie picture of the cart, disk, cd, rom)
Game Notes (setup and settings notes)
Game release type (licensed, unlicensed, unreleased, hack, homebrew)
Game Physical media type
Game Publisher
Game Developer
Game Designer
Game Programmer
Game Genre
Game Gameplay (arcade, shooter, card/tiles)
Game Perspective (Top-down, 1s-person, 3rd-person)
Game Visual (2D scrolling, side scrolling)
Game Scenario/Setting (horror, war, sports)
Game ESRB Rating
Game Version

I haven't included checksums for roms yet in this metadata, as maybe it is too much detail for what I am trying to accomplish.

[MiSTer_Kirk]

My list of metadata includes:

That list is way, way too long, most of it is not needed.
Plus, you are never going to get ALL that info, for ALL games, it's just too big of a task.

And going on from what Akeley said, there are tens of thousands of games for just a few systems, like the ZX Spectrum, Amstrad CPC, C64, BBC Micro, etc... Most of those games won't come with the info you are wanting, such as serial number (Why is that important ?), ESRB Rating, or game version.
Plus, there are still, probably, thousands of missing games, across all systems, that may never get a dump to digital media, let alone be found.

Akeley, what is the purpose of 50+ different Airborne Ranger C64, mostly hacks?

When there are often many different copies of one game, you will find bad dumps, alternative versions, different releases (full release, budget release), etc... This brings up one problem, which version do you say is the definitive version ? Normally, this would be a properly dumped release which includes all copy protections, such as .g64, or .t64, or Amiga ipf files.

[JWDog]

That list is way, way too long, most of it is not needed.
Plus, you are never going to get ALL that info, for ALL games, it's just too big of a task.

And going on from what Akeley said, there are tens of thousands of games for just a few systems, like the ZX Spectrum, Amstrad CPC, C64, BBC Micro, etc... Most of those games won't come with the info you are wanting, such as serial number (Why is that important ?), ESRB Rating, or game version.
Plus, there are still, probably, thousands of missing games, across all systems, that may never get a dump to digital media, let alone be found.

When there are often many different copies of one game, you will find bad dumps, alternative versions, different releases (full release, budget release), etc... This brings up one problem, which version do you say is the definitive version ? Normally, this would be a properly dumped release which includes all copy protections, such as .g64, or .t64, or Amiga ipf files.

Thank you for your response. I agree that the list of categories of the metadata may be long, but all of this metadata belongs to the games. Obviously there is so much more data I could include, but I felt unnecessary like music producer, composer, character design, and list goes on. And I know that there will be many blanks of info such as serial number, esrb rating, game version etc because it is just not available or was never there to begin with. I also recognize that some of this info may not be necessary for everybody... beside that is just your opinion, that information is important to me or possible someone else, you don't know. I have tried to keep the info simple. I am not looking to publish unless a demand is created for such a list. But, if the info is available why not put it in a database and be able to use it for those that want? I am not creating a database of roms, or dumps, that is why I stated that I may not need the check sum data. I am creating a database of games, usually these are full release, but with few exceptions like available or known, demos, promos, hacks, translations etc. Again, I am looking for metadata about games, and it seems that I am having to go to many different sites to bring it to one place my database. I am looking for suggestions on where game lists are and metadata availability, if you have suggestions about sites I could use then, I would appreciate it. If there are games that don't have a digital dump, that is ok. You may never know when one is. If never that is ok too.

[MiSTer_Kirk]

Your best place to start is a Raspberry Pi, or PC front-end, gamelist.xml files.
These contain, (for example, this is Atari 2600 3-D Tic-Tac-Toe)

<game id="14001" source="ScreenScraper">
<path>./3-D Tic-Tac-Toe.zip</path>
<name>3-D TIC-TAC-TOE</name>
<desc>THE GAME IS SIMILAR TO THE TRADITIONAL GAME OF TIC-TAC-TO.</desc>
<image>./downloaded_images/3-D Tic-Tac-Toe-image.png</image>
<rating>0.5</rating>
<releasedate>19800101T000000</releasedate>
<developer>SEARS, ROEBUCK AND CO.</developer>
<publisher>ATARI</publisher>
<genre>SIMULATION</genre>
<players>1-2</players>
<region/>

You're not going to find much more info than that, possibly more if you use many different websites. To start off, I would goto Moby Games.
Here is an example of Super Mario Bros 2 on NES.

https://www.mobygames.com/game/super-mario-bros-2

Lots of info there, as a start.

[JWDog]

Your best place to start is a Raspberry Pi, or PC front-end, gamelist.xml files.
These contain, (for example, this is Atari 2600 3-D Tic-Tac-Toe)

<game id="14001" source="ScreenScraper">
<path>./3-D Tic-Tac-Toe.zip</path>
<name>3-D TIC-TAC-TOE</name>
<desc>THE GAME IS SIMILAR TO THE TRADITIONAL GAME OF TIC-TAC-TO.</desc>
<image>./downloaded_images/3-D Tic-Tac-Toe-image.png</image>
<rating>0.5</rating>
<releasedate>19800101T000000</releasedate>
<developer>SEARS, ROEBUCK AND CO.</developer>
<publisher>ATARI</publisher>
<genre>SIMULATION</genre>
<players>1-2</players>
<region/>

You're not going to find much more info than that, possibly more if you use many different websites. To start off, I would goto Moby Games.
Here is an example of Super Mario Bros 2 on NES.

https://www.mobygames.com/game/super-mario-bros-2

Lots of info there, as a start.

Thank you for the suggestions. The gamelist.xml files never really thought of that. Thanks!

[aberu]

I think it's fine to have a lot of categories for the metadata like you have, but treat it like a community sourced thing, like MP3 tags in databases already. Creating the framework and getting a database setup or maybe just a github page where people can contribute is a good first step.

All of the games don't need all of the metadata perfectly filled in right off the bat.

[darksakul]

I am in development of the ultimate game/rom list-database for all cores currently on the MiSTer. This list includes meta-data for each title. I am not looking to create a frontend for the MiSTer, but a way to have a paper and/or electronic copy of all games and notes regarding their settings and setup. I know the meta data is spread around to multiple websites, with some websites that contradict each other in data, so wanted to have a discussion on which websites are best to find lists of games for the cores and which have the best metadata available. Also, if there are multiple releases to a title which there are... what are the best sources to find info on these releases?

A good example of multiple release is 10-Yard fight for NES. So far I have found 5 different cart releases. Maybe there are more out there. Please help if anyone knows.

10-Yard Fight, IF-02, Japan
10-Yard Fight, NES-TY-USA, USA
10-Yard Fight, NES-TY-CAN, Canada
10-Yard Fight, NES-TY-GBR, UK (Great Britain)
10-Yard Fight, NES-TY, front of cart doesn't have a serial number, any one know the region?

Also, does anyone know how roms are related to these carts, trying to understand that the rom chip inside carts for example could be the same for multiple cart releases.

Also, if anyone has a better way of having this type of database let me know if I am just creating the wheel. Then it might be easier to pay for something rather then create one.

Sorry for multiple questions just trying to wrap my head around this project.

Anyway, thoughts and ideas are appreciated.

Sure there different cart releases, but there not always that many rom versions

Sometimes the US, Pal and Japanese games is the exact same rom. Hence why some roms have multiple regions in it's rom name.
10 Yard fight for example might have 5 to 7 different versions of the cart but there rom is the same for each.
Early NES games are also have some major physical differences, many of the very early NES carts are actually Japanese Famicom boards with a Famicom to NES adapter in it's shell (like for example Gyromite)

Another example is there no difference between the US and Japanese versions of Super Metroid when you look at the rom it self, only the physical shell the board is inside of.

For the most part, the Smoke Monster packs have all the roms organized by region, and very few roms have later revisions to the code it self.
That covers most of the 8 bit and 16 bit consoles, or at least the consoles that also has a Everdrive.

Weirdness goes into when you get onto stuff like the C64 Library

MSX is mostly listed out in the vhd pack that floating about.

[JWDog]

treat it like a community sourced thing, like MP3 tags in databases already. Creating the framework and getting a database setup or maybe just a github page where people can contribute is a good first step.

All of the games don't need all of the metadata perfectly filled in right off the bat.

Aberu, can you please explain what you mean by community sourced thing, MP3 tags? I have never developed in this method, only for my own use. It would be great if this could be more collaborative. How would I set that up?

[JWDog]

Sure there different cart releases, but there not always that many rom versions

Sometimes the US, Pal and Japanese games is the exact same rom. Hence why some roms have multiple regions in it's rom name.
10 Yard fight for example might have 5 to 7 different versions of the cart but there rom is the same for each.
Early NES games are also have some major physical differences, many of the very early NES carts are actually Japanese Famicom boards with a Famicom to NES adapter in it's shell (like for example Gyromite)

Another example is there no difference between the US and Japanese versions of Super Metroid when you look at the rom it self, only the physical shell the board is inside of.

For the most part, the Smoke Monster packs have all the roms organized by region, and very few roms have later revisions to the code it self.
That covers most of the 8 bit and 16 bit consoles, or at least the consoles that also has a Everdrive.

Weirdness goes into when you get onto stuff like the C64 Library

MSX is mostly listed out in the vhd pack that floating about.

Darksakul, thank you for the rom info. What you said makes sense. I am just not sure, though how I would integrate the roms into the database, since there are so many different dumps. Each MiSTer user may not use the same rom set.
TOSEC vs No-Intro for example.

This game database is more based on game releases for example. The example of 10-Yard Fight which has multiple different carts is my example. And like you said it is possible to have the same rom chip in many different region carts. I have included in the database a field for parentgame titles to group related game carts together such as 10-Yard fight releases. But it also shows the different releases. I also want to have a field for series. To show related games in sequence.

[akeley]

You should really look at Gamebase. It has most of the metadata you want, already painstakingly researched over many years, and includes extras such as covers and manuals. Everything can be searched for and organised using numerous tags within a frontend. GB can then export these lists, and even roms themselves according to certain criteria, and so you can make your own packs (eg all the turn based NES rpg's from years 1989-1991)

Alternatively you can also check out Launchbox, which scrapes data from their database (built from Gamebase and other sources) and which is a bit more modern and better looking, according to some people at least. I prefer GB because it's more thorough, flexible, and free.

You can of course try and make your own lists, but trust me, it's a lot of work. Even simple 1G1R filtering takes forever (is what I'm now doing with Total Dos Collection, by year), never mind adding metadata to records.

[aberu]

treat it like a community sourced thing, like MP3 tags in databases already. Creating the framework and getting a database setup or maybe just a github page where people can contribute is a good first step.

All of the games don't need all of the metadata perfectly filled in right off the bat.

Aberu, can you please explain what you mean by community sourced thing, MP3 tags? I have never developed in this method, only for my own use. It would be great if this could be more collaborative. How would I set that up?

It's an analogy to this --> https://en.wikipedia.org/wiki/ID3

Basically ID3 is a metadata embedded into MP3's that contain information about the publisher, album title, title of the song, track number, etc... tons of different categories of information.

I see the database you are making as an equivalent to this, so it's fine to have a bunch of categories because you don't need all of them filled in straight away as the information could be crowd-sourced by contributors. Doing it all yourself would be impossible.

[JWDog]

It's an analogy to this --> https://en.wikipedia.org/wiki/ID3

Basically ID3 is a metadata embedded into MP3's that contain information about the publisher, album title, title of the song, track number, etc... tons of different categories of information.

I see the database you are making as an equivalent to this, so it's fine to have a bunch of categories because you don't need all of them filled in straight away as the information could be crowd-sourced by contributors. Doing it all yourself would be impossible.

Basically, the database is setup this way. Just need to figure out how to do the crowd sourcing. Any ideas? You mentioned github, I am not knowledgeable about how that works. Please explain.

[akeley]

I'm scratching my head trying to figure out why would you want to start this gigantic undertaking, when it mostly has all been done already?

If you really need to have your own datatbase, better way to build it imo would be to figure out a way to import data from various existing sources mentioned in this thread. It wouldn't be trivial, but at least doable

Look at MobyGames: it's the best-known crowd-sourced gaming database, which has been around since forever, and yet compared to individual efforts focusing on single platforms it's rather lacking.

[aberu]

I'm scratching my head trying to figure out why would you want to start this gigantic undertaking, when it mostly has all been done already?

If you really need to have your own datatbase, better way to build it imo would be to figure out a way to import data from various existing sources mentioned in this thread. It wouldn't be trivial, but at least doable

Look at MobyGames: it's the best-known crowd-sourced gaming database, which has been around since forever, and yet compared to individual efforts focusing on single platforms it's rather lacking.

Right, this is a good way to do it, I think they were more talking about a MiSTer focused import of this kind of information that then can be used for maybe things like symlink scripts for rom organization, and stuff like that, down the line.

[JWDog]

I'm scratching my head trying to figure out why would you want to start this gigantic undertaking, when it mostly has all been done already?

If you really need to have your own datatbase, better way to build it imo would be to figure out a way to import data from various existing sources mentioned in this thread. It wouldn't be trivial, but at least doable

Look at MobyGames: it's the best-known crowd-sourced gaming database, which has been around since forever, and yet compared to individual efforts focusing on single platforms it's rather lacking.

To be really honest, what led me down this road of creating this project, was that I have been using different sources websites to find the list of games for different systems. Printed them out, and then used these lists to confirm that I had all of the roms for the cores. At the same time discovering that some games and systems needed different setup and settings. Then I thought I could try and combine these lists to create a game binder that had the games by system, region, alphabetical order etc. It would also be used to create different ways of organizing games, then using this to organize the roms. I had read in the forums that there wasn't going to be a true frontend created for the MiSTer, which doesn't bother me. I like the way it is setup now. But, I could use this source on the side as a resource for settings etc. So, I have started with the famicom games, nes games, family disk system, super famicom, and super nintendo. This obviously is still a work in process. So, then I wanted to reach out to you guys, here on the forum to ask where I could find the best sources of info (metadata) as I was encountering metadata that was contradictory and not complete. I was also encountering info where I discovered multiple releases for games and was curious about how the roms were inter-related with the games. Obviously, some carts have the same board inside, therefore have roms with multiple regions. The misc metadata that seems like extra, I could use at a glance to know the genre, gameplay, scenarios etc. So, this is where I am at. You gave me some great ideas, that I have started to use, and Aberu suggested that instead of only me putting this together that we could make it a community project and thought that would be cool and I would be open to it, would just need to learn the logistics. I also like the idea of eventually, being able to use it to have symlinks and other potential projects. So, here I am. I am not sure what you guys use to confirm roms or games, or remember settings. All Ideas are welcome.