etabeta's playground

MESS 0.138 will finally add a long-awaited new feature to the long list of options for the user: the software lists.

The idea is to provide a confirmed list of good dumps for computers and console, in the same fashion as MAME does for arcade games. For MAME command line users, this also means to have an ever more
familiar way to launch games in MESS: if in MAME you would simply type

mame gamename

now in MESS you will be able to simply type

mess system -cart software

where “software” is a 8-char long name, which is stored in the .xml software lists offered in bundle with the emulator.

While the emulator will always be capable of loading your own dumps and test programs by using
the old loading options, this new feature will allow to

• simplify bug reports, because you will be able to add the software name to any bug report and we will know for sure that the problem is in the emulator and not in a possible corrupted image (claimed to be good by the website you find it in);
• create compatibility lists, because you will be informed about which games are not supported (due to missing parts of the emulation) and which ones are expected to work;
• document real software that was released for a system, because lists will try to cover all the known dumped software both for common systems and for the more obscure ones, whose available games are seldom mentioned in internet (e.g. Hartung GameMaster, Epoch Super Cassette Vision, Koei Pasogo, etc.);
• easily identify spare .bin files you have in your HD, because you will be able to use the -romident option also with the software lists.

Overall, the improvements are in my opinion well worth the addition, despite the support being still preliminary and more work being in progress. What follows is a preliminary FAQ for software lists, covering basic usage of the lists as they are implemented in MESS 0.138. Once the code stabilizes and all missing features are added, these FAQs (and some more) will be integrated in the MESS User Manual on MESS Wiki.

Q1. What are these software lists you talk about?
Q2. Where can I find the lists?
Q3. Where shall I put the lists?
Q4. Where shall I put the software?
Q5. How can I launch a game with the new commands?
Q6. Can I use MESSUI or newui with software lists?
Q7. What if I want to launch a game which is not recognized in the software lists?
Q8. What is the exact format of the lists?
Q9. What else can I use softlists for?
Q10. How will software lists be updated?
Q11. How can I contribute to software lists?
Q12. Can I use clrmame to build a full romset for MESS or do I need to manually rename thousands of files?
Q13. Are software lists recommended over the good old launching method?
Q14. What is still missing in software list support?
Q15. Are MESS lists going to make GoodTools, No-Intro and TOSEC collections useless?

Notice that the sections below written in italic font are quite technical, so you can safely skip them if you are only interested in using lists in 0.138 and not in their future developments.

Q1. What are these softlists you talk about?

A1. As of 0.138 release, MESS starts to document in xml format the games/programs which have been released for various consoles and home computers, in addition to the systems themselves (which were already documented in the source). At the moment, only carts have been documented, and not all the systems are covered; however, the plan is to extend the support to most systems and to tapes and disks as well, given the high risk for info on obscure and not so obscure systems & software to be lost forever.

The chosen (minimal) format contains a shortname for each game (like MAME shortnames), a longer name (which should match the title screen, or the box screen if no title screen exists), the release year, the publisher name and the CRC32/SHA1 checksums.
Additional info like alternative titles, development teams, precise release dates, serial numbers, available languages, etc. can be stored externally in datfiles to be displayed by frontends (even if some more optional fields might be eventually added).

Q2. Where can I find the lists?

A2. If you have downloaded MESS 0.138 or you have checked out the svn repository, you already have the lists in the hash\ directory. However, you can check the various lists in the git repository.

Q3. Where shall I put the lists?

A3. By default, software lists in .xml format are stored in the hash/ directory inside the MESS directory. You can however decide to store them elsewhere, by adding a different directory as “hashpath” in mess.ini.

Q4. Where shall I put the software?

A4. Software storage is slightly more complicate than list storage. The basic principles are:

• different systems will have their software stored in different directories (so that you can separate e.g. SNES games from MegaDrive ones);
• different media for a given system will be in different directories as well (so that e.g. C64 tapes will be separated from C64 carts and floppies).

In order to reflect these principles, when you load software using software lists MESS will look for files inside subdirectories of your “roms\” directory.
Namely, MESS expects to find subfolders named like the software lists: i.e. MESS will

• look for Epoch Super Cassette Vision carts inside “roms\scv\”;
• look for megadriv, genesis and megadrij carts inside “roms\megadriv\” (because all these systems use megadiv.xml which contains a software list with name=”megadriv”);
• look for IBM PCJr. carts in roms\ibmpcjr_cart/.

As these few examples show, we tried to use for lists either the system name (or the parent name, when multiple systems share the same list); or the system name followed by the media type, for systems which support several media type like most home computers (see ibmpcjr example, which uses a ibmpcjr_cart list, in view of a future addition of floppy disks and tapes lists).

Of course, MESS also allow you to store your carts in different folders than “roms”, e.g. you can use “C:\console_games\snes” and “C:\console_games\gbcolor”, as long as you add the root directory “C:\console_games” to your rompaths in mess.ini. The only things you have to care about is that the subfolders (“snes\” and “gbcolor\”, in the examples above) are correctly named as the software lists they refer to.

A full list of supported xml files, with the correspondent names, will be added to MESS Wiki soon-ish to help users (even if the correct subfolders can also be generated by clrmame, see Q10 below).

Summing up, at the moment, you are allowed to store software in a directory tree like the following one


roms/gameboy/
roms/megadriv/
roms/scv/
rompath_1/ibmpcjr_cart/
rompath_1/crvision
rompath_2/wswan


as long as your mess.ini contains a “rompath roms/;rompath_1/;rompath_2/” line.

Eventually, we might add some more storing options, like leaving roms in the main rompath, but this is currently not so easy to implement, so please stick to the folder structure above, or use the old loading options with absolute paths.

Q5. How can I launch a game with the new commands?

A5. If you have stored the software roms as explained in Q4, and you want to launch “Super Mario Land” for the Game Boy (which corresponds to the shortname ‘sml’), then you simply have to type

mess gameboy -cart sml

and MESS will load the game. Notice that MESS also allow to load the same game in gbcolor, by using the following command

mess gbcolor -cart gameboy:sml

where the “gameboy:” part tells MESS that ‘sml’ belongs to the gameboy list. Similarly, you can load “Legend of Zelda Oracle of Ages” for the Game Boy Color (whose shortname is ‘zeldaage’) both in gbcolor with the command

mess gbcolor -cart zeldaage

and in gameboy with the command

mess gameboy -cart gbcolor:zeldaage

Of course, this can only be done for systems which were compatible, i.e. which had the same physical cart interface (this is reflected in the use of the same “interface” parameter in the xml list): current compatible systems are

gameboy - gbcolor
pce - sgx

i.e. you can load Game Boy Color games in the Game Boy and viceversa, and you can load PC-Engine games in the Supergrafx. On the other hand, you cannot load a Turbografx 16 game into the PC-Engine emulation, because they were not compatible (US carts were ‘encrypted’).

Eventually, also Neo Geo Pocket and its Color version and Wonderswan and its Color version might be converted to use separate but compatible lists. However, at the moment, they use a common list (which means you have to store all the Wonderswan games in “roms\wswan” and all Neo Geo Pocket games in “roms\ngp”, rather than in separate folders depending on the color support).

Q6. Can I use MESSUI or newui with software lists?

A6. Not yet, unfortunately. At the moment, we are still working on the core support so that software list loading is only a command line feature. Nothing stops you from renaming your roms and then point MESSUI and newui to load the renamed files: most of the files will load correctly, even if no year/publisher info will be displayed. However, a few items (e.g. scv, bbcbc and ibmpcjr carts consisting of multiple files) won’t work if not loaded from command line, even if we are talking of around 10/10000 files.

Q7. What if I want to launch a game which is not recognized in the software lists?

A7. MESS still supports the old loading procedure consisting in typing

mess system -cart C:\path\to\my\carts\file.bin

or in launching games from MESSUI and newui, so you can simply keep using the old procedure for unrecognized dumps or for your own programs.

Q8. What is the exact format of the lists?

A8. The chosen (minimal) format is the following

<software name="shortname" cloneof="parentname" supported="partial">
	<description>Game Title (Region, vx.x)</description>
	<year>19??</year>
	<publisher>Software House</publisher>
	<part name="cart" interface="xxx_cart" feature="yyyyy">
		<dataarea name="rom" size="131072">
			<rom name="romname.bin" size="131072" crc="aaaaaaaa" 
				sha1="aaaaaaaabbbbbbbbccccccccddddddddeeeeeeee" offset="0" />
		</dataarea>
	</part>
</software>

 

• The software node contains the “shortname” which identifies the software (like in MAME, less than 8 chars for parents and less than 16 chars for clones), the “parentname” if this software is a clone, the “support status” (yes if missing, no or partial if the software is known to be not completely working).
• The description, year and publisher nodes contain info for UIs and frontends.
• he part node(s) represent the actual content of the software: “name” entry should match the media type for this software; the “interface” allows for compatibility across different lists (see the gameboy/gbcolor example in Q.5); the “feature” field allow for specifying mappers and other game-specific features which MESS have to handle when loading this software.
• Finally, dataarea and rom contain the actual software file specifications (with “size” and “checksums”). Notice that we might well have several “dataarea” in a single “part”, and several “roms” in a single “dataarea”.

Also, as mentioned above, some more optional fields might be eventually added.

Q9. What else can I use softlists for?

A9. One of the main reasons to have a database of known dumps, is to be able to identify spare files or dumps you do by yourself. This has been an appreciated feature in MAME for years, when dealing with arcade dumps, and we wanted to offer something similar for consoles and home computers. Software lists allow for this: indeed, if you have some unknown files in “C:\unk\files\”, then you can simply launch

mess -romident C:\unk\files\

and MESS will compare size and checksums with the ones recognized by its source (for BIOS items) and by its software lists (for cart/floppy/tape items).

Q10. How will software lists be updated?

A10. Well, being external .xml files, any update to current lists will simply require to overwrite the old .xml file with the new one, inside the “hash\” directory.

Q11. How can I contribute to software lists?

A11.Any contribution (be it a fix or an improvement) can be simply posted in the MESS board.

Q12. Can I use clrmame to build a full romset for MESS or do I need to manually rename thousands of files?

A12. Of course, manually renaming thousand of files is not a very funny work and it’s definitely better to spend the time testing new features than renaming files. Hence, I strongly suggest to use clrmame, even if software lists are not 100% integrated with clrmame. In other words, be warned that some work at your end is required, and not everything is automated.

WARNING: the instructions below only apply to next cmpro version, not to the current 3.133a which does not support our xml format.

Single list handling
First of all, let assume you are only interested in a single software list, say the Arcadia one. Since at the moment clrmame cannot directly extract the list info from MESS, you have to manually feed the software list into the rommanager. Two options are available (and they are basically equivalent): either you point clrmame directly to the arcadia.xml file in the “hash\” directory (by selecting “Add DatFile…” in the Profiler window, and browsing your HD until you find the “hash\” directory inside your MESS directory) or you generate a separate xml file by using

mess arcadia -listsoftware > arc.xml

and then point clrmame to the newly generated arc.xml file (again, by selecting “Add DatFile…” in the Profiler window, and browsing your HD until you find the file you wanted).
In fact, there is no reason not to directly use the xml lists in the “hash\” directory, but I wanted to stress that MESS also allows to export lists with the -listsoftware command.

Once you have added the list to clrmame, a new profile will be created and, by loading it, you can finally setup the directory where you want to store your arcadia software, and rebuild any directory containing arcadia roms to create a romset compatible with MESS.

Multiple lists handling
Now, what if you want to get all the supported lists at once? In this case, a first dumb solution would be to repeat for each list the procedure above. However, there is also a smarter and easier procedure, thanks to the clrmame batchrun capabilities!
Indeed, to rebuild all the software lists at once, you can simply follow these steps:

1. you point clrmame to all the lists you are interested in (by selecting “Add DatFile…” in the Profiler window and choosing the xml lists you want)
2. after clrmame has created a new profile for each of the lists, you select all of them
3. by pressing Enter, you are prompted with a “Profile Batchrun” window where you can choose among plenty of batchrun options
4. in the Misc submenu, you select the fundamental “Create rompath for new dat” option and choose as root folder either your roms/ directory or any other rompath you like
5. in the Rebuilder submenu, you select “Rebuilder run before scan” and “Use source folder”, and you enter the location of the directory which contains the roms you want to rebuild
6. you press OK and you wait for clrmame to do its job (remember to confirm OK when the statistics are prompted)

At the end, you will have the folders you need inside “roms\” and all the available roms will be in the correct location. Of course, at Step 5. you can also select additional Rebuild and Scan options as you like, or you can decide to only Rebuild without scanning.

Notice that, after the first run, you can always select multiple profiles to scan more software lists at once, with or without any Rebuild procedure.

Conclusion
Eventually, we might add better integration between clrmame and MESS software lists (e.g. clrmame could be able to choose proper batchrun options after simply parsing the MESS -listxml output), but for the time being the procedure described above should allow everyone to test the new feature.

Alternatively, you can also generate a single xml containing all the software lists by using

mess -listsoftware > all_lists.xml

and then feed the resulting xml file into clrmame. However, the duplicate filenames across different lists are not supported at the moment, so clrmame will try to append some _0 or _1 to your romset names, making them not directly usable in MESS. Therefore, I currently suggest to use separate xml files as described above, rather than a single huge xml file.

Q13. Are software lists recommended over the good old launching method?

A13. To say the truth, at the moment software lists are not the suggested method for software loading, especially if you usually launch games through a frontend / UI. Indeed, we haven’t still added support for lists to MESSUI and newui, and no frontend supports them.
However, we strongly recommend to use the -romident feature whenever you want to report a bug for a system which supports software lists. Adding the shortname of the software which exhibit problems to your bug report would help to exclude that you are using a bad dump and to confirm that the bug is really due to MESS.

Q14. What is still missing in software list support?

A14. Well, in fact there are various aspects which still need to be improved.

Core improvements
As already mentioned, we need to integrate lists with UI and newui, to both display software info of the selected image and to handle software consisting of several files.

Moreover, we need to show some warnings when the user loads a software item with known problems (either the game requires unemulated features, or the dump is known to be bad).

Finally, only a few systems have been updated to really exploit all the features which lists allow, like installing per game handlers and decoding routines, and this will need per-system fixes (i.e. it will require some time).

List improvements
While lists already contain most of the known software for each systems, additional info like year and publisher are often incomplete or missing. And even when present, they still have to be verified against title screens (this especially holds true for larger lists like megadriv.xml, snes.xml, gba.xml, etc. since smaller lists have been mostly fixed already).

Also, the “supported” attribute has still to be correctly set for all games. This will require some time, given the size of some of the lists, but it is in progress.

Q15. Are MESS lists going to make GoodTools, No-Intro and TOSEC collections useless?

A15. Well, despite what some MESS supporter might claim, in my opinion we won’t replace GoodTools, No-Intro or TOSEC anytime soon. And the simple reason is that we all have different goals, so that different software collections will always appeal many users which would not like our software lists. Let me analyze more specifically the various projects (rigorously in alphabetical order):

• GoodTools: Cowering’s tools have helped a *LOT* to make order among messy dumps floating in the internet, especially back when console emulation got started. Without GoodTools, we would probably only have tons of file generically named as HEBEREKE.BIN, which could hardly be identified as correct dumps or recognized as faulty and broken files. While often criticized for spreading out bad dumps (when it’s actually fault of websites which keeps offering corrupted dumps for download, together with the correct ones, even now that Cow’s tools don’t show missing bad dumps anymore), GoodTools have many great capabilities [1] and, in particular, they can recognize hacks, translations and demos, making them a **MUST** for people interested in homebrew & romhacking development.
• No-Intro: No-Intro’s work is probably closer to our software lists than any other project, and I have to really say thanks to all the “datters” involved in No-Intro for the huge efforts they have done in redumping and in properly documenting console dumps. Without No-Intro, it would have been way harder to create a list of confirmed dumps, and in particular it would have been impossible for GBA. However, differently from MESS software lists, they only focus on Official Releases, making their collection a MUST for people which are not interested in pirate carts and prefer to have a clean collection of all the Jp/Eu/US software ever released for a given system.
• TOSEC: TOSEC collections offer basically the same pros and cons of Good sets, but they cover a huge amount of obscure systems, whose software would have probably got lost without them. Despite the inclusion of bad dumps, TOSEC collections represent a MUST for people interested in home computers (e.g. for the comprehensive documentation about demos, which often represented a way more creative world than the official programs/games releases)

Summing up, if people want a collection of software which worked on the real hardware (both officially released, and pirated) and they want to launch it easily from command line MESS, then they will for sure drop other collections in favor of our software lists.
But if they want collections including demos, translations and hacks (which are often as good as the original software), or if they only want a “perfect collection” of the official software, then they will never abandon other collections to use our software lists.

And it is also for these users, which are not interested in our software lists, that MESS keeps supporting the old loading procedure, through absolute paths, so that files from the different collections can still be easily launched in MESS 🙂

[1] Actually, GoodTools have many more qualities than the few I focused on: they can fix NES headers, they can convert SNES and MegaDrive images from one format to another, etc. However, I wanted to give reasons why people might want to collect Good sets and probably hacks, demos and translations are more appropriate than these additional features 😉