David Haywood's Homepage

Aaron has checked in a change to the CHD format featuring his further work on the format, and bumping the version to v5.

This includes a refined / cleaned up version of the FLAC support I was working on (which is now official) as well as a number of other improvements (LZMA support based on the 7-zip code is also in there)

I haven’t tested / used this code yet, so I can’t say how compressions compare (if data is being optimally ordered for FLAC etc.) but you should see substantial gains, even over the previous versions I posted due to the use of LZMA which in tests on regular romsets usually yields around a 20% improvement in compression. The LZMA improvements mean there will also be benefits to non-CD CHDs. LD ones are also likely to shrink very slightly as FLAC is being used for the audio part of those now (don’t expect big savings tho, they’ve always been video-heavy and lossless video compression is always going to give big files)

The created CD CHDs are SHA1-compatible with the previous ones, so none of the software lists need updating thankfully.

Just a heads up, because unless there are any bugs this will be the expected version from the next release, as opposed to the previous FLAC trial runs.

In case you haven’t been following any of this here’s a summary:
Old V4 (and below) CHD format used ‘zip’ compression internally.
New V5 format supports ‘zip’ , ‘lzma’ (7-zip) and ‘flac’

FLAC seems to give about 40% better compression than ‘zip’ for audio data (CD AUDIO tracks)
LZMA seems to give about 20% better compression than ‘zip’ on regular data.

Of course some data just plain doesn’t compress well, so you’re not going to see those saving everywhere but you’re likely to see on average a 20-40% reduction in CHD size for all non Laserdisc CHDs.

This naturally makes CHD better for storing large amounts of data, which was it’s original purpose.

As it transpires the release of MAME 0.145 was meant to come with a ‘promo’ combined MAME / MESS build, at least that was the plan at one stage. Why that didn’t happen I don’t know. Maybe concerns about the overall stability of the 0.145 release giving a bad impression, maybe a lack of time, maybe some more boxes needing to be ticked with various devs, maybe some last minute conflicts which prevented it going ahead. Either way, it didn’t happen.

Anyhow, there was some *official* code written for this purpose, similar to the last Ultimate patch I posted, but stripped down a bit further to the bare minimum of changes needed to build it. The code, written by Micko (MESS Co-ordinator) has been given to me, and I’ve uploaded it here. That code builds a variant called ‘universal.exe’ / ‘universal64.exe’. Given that it was the plan to include it at one point I’m cautiously optimistic that this may yet see the light of day in the official source trees (and maybe even as binaries) but we’ll see.

Personally I find ‘universal.exe’ a bit of a mouthful, so in building a new EXE myself I’ve shortened that to simply ‘uni.exe and ‘uni64.exe’ (UNIversal emulator).. actually UNE (UNiversal Emulator) would would also be a reasonable executable name. But, minor detail, this combined build has yet to really be given a proper name, so for now I’m still just gong to call it ‘Ultimate MAME’

Ultimate MAME 0.145+ (MESS SVN r14446) (GIT rev)

This as suggested is built from the MESS SVN r14446 code. It includes everything from the MAME/MESS of that revision, including the 7-zip support for ROMs.

As an additional option I’ve included extra binaries which are built as ‘MESS’. Those binaries will identify themselves as ‘MESS’ to external applications instead of Ultimate/Universal/MAME. This is done for compatibility with the QMC2 frontend. The MESS Version of the QMC2 frontend can be used with the ‘MESS’ binaries, because as far as it’s concerned it’s just a build of MESS with additional arcade titles. The actual MAME version of QMC2 is more stripped down and lacking in features needed to run various systems included in the Ultimate version, so you have to trick it into thinking it’s running MESS. If you’re not running / wanting to run QMC2 you have no need for these binaries.

EASY WAY, FOR THE LAZY

64-bit link (RAR) (includes resources, source, 64-bit uni.exe binary, and a 64-bit all-inclusive binary built as mess.exe instead for if you want to use QMC2)
32-bit link (RAR) (same as above, but 32-bit)

MANUAL WAY, FOR THE TECHNICAL

the following patches were used
Universal / Ultimate MAME build target
Patch to resolve build conflict with ‘bullet’ (name was used for both Wave Mate Bullet in MESS and Sega’s Bullet in MAME)
APE support patch for Samples (not vital but you can play with it if you want, see prior update for details)

SVN r14446 Common MAME/MESS Resource Files YOU WILL NEED THIS FOR ANY OF THE BINARIES / SOURCES BELOW TO WORK CORRECTLY (it includes the softlists, and artwork used by a number of MESS Systems)
Source code for builds below (patches pre applied)
64-bit Binaries (with tools) (drop these in the folder where you extract the above resources)
32-bit Binaries (with tools) (or these if you’re on 32-bit)
64-bit Binaries (with tools) (built as MESS)
32-bit Binaries (with tools) (built as MESS)
(note, to build like the above 2 just copy mess.c (from /mess) over uni.c (in /uni) renaming it to uni.c, obviously)

—-

Due to the change in build process (using the Official patch instead of my own) there is a slight chance that some things might not behave quite as before, for example in my own builds I was making sure that the same CPU cores were built by each project, although with the modern MAME architecture it looks like such concerns were unnecessary (it used to be if you changed the CPU cores included you had to do a full recompile) Everything here is built first by building MESS (make) then building the Ultimate / Universal build using that as a base (make TARGET=uni) Finally tools were built (make tools)

—-

I usually provide a demo of some of the cool stuff / comparisons when I post one of these builds, but I haven’t really had much time to do anything like that with adding the various 7-zip, FLAC, APE and JPEG support as shown in prior posts. I’ll try and find something worth showing in the coming days tho, there is a lot of neat stuff happening in MESS lately such as the ability to run old versions of MAME on the 486 driver, but it doesn’t make for much in the way of screenshots, and it is a little slow to be of practical use regardless of the ‘cool’ factor it has :-) ..

Charles MacDonald has recently been posting information about the DECO Cassette system in order to get some more tapes dumped because there hasn’t really been any activity on them since way back when the initial batch was added.

This is good news because these things (being magnetic media and custom dongles) are extremely fragile, and the remaining ones REALLY need sourcing and dumping before it’s too late. Of course, all that work isn’t much good if there’s nothing to show for it, but luckily there is. Here is ‘The Grand Sumo’ one of the previously undumped DECO Cassettes. In the end it was a simple slot-in to the driver (Type 4 dongle, BIOS Type A) so I can’t really take credit for anything, but Charles’ fantastic work, and the contributors who have helped buy the game and equipment needed deserve a shout out here.

AFAIK the Dumping Union *don’t* currently have access to the other rare games, and if they show up they’re bound to be pricey, so please consider donating to them if you want to help source these rare and fragile pieces of history.

One of the most important things to note about Charles’ work is it means the dumping is now a NON-DESTRUCTIVE process. The dongles are dumped without even opening them, so it’s risk free compared to older methods. This is good news for people who own the cassettes if they’re considering loaning them out.

Credit this time round goes to Charles MacDonald, Dr. Spankenstein, Kevin Eshbach, T. Huff, SteveS, E. Page-Hanify, Hikari, ArcadeDude, F. Bukor, N. Francfort, jmurjr, arcade-history.com, ThumB, Hurray Banana, Paratech, Xiaou2, Cornishdavey, A. Costin, M. Ponweiser, Tormod, Rambo, Smitdogg, The Dumping Union

As promised below I’ve put a bit of time into hooking up 7-zip support in MAME. This is a first pass, so there is still room for improvements (both to code quality, and potentially bug fixes)

The way I see it zip / 7z offer the following pros / cons

.zip
– Aging 32-bit format
– Can only hold 2gb of data (we’re hitting that limit in some cases)
– Only supports UTF-8 filenames
– Single, dated algorithm
+ low memory overhead

.7z
+ Modern 64-bit format
+ No practical limitation to filesizes
+ UTF16 support (debatable advantage)
+ newer more advanced algorithms
– higher memory overhead if ‘solid’ blocks are used

Zlib (.zip support) might be tried and tested, but it’s showing it’s age, it’s a format of the previous decade when FAT32 was the standard, and nobody had large files. .7z is newer and doesn’t have many of the inherent issues of an older format, although does still need some testing. Staying ahead of the game is always a good idea tho, better to be modern and be prepared.

Download the diff here. This has been sent to R.Belmont for portability checking and official inclusion. Right now this can be used for roms, artwork, samples, cheats etc. -romident and romcmp.exe aren’t hooked up to it yet, but that will come with time. As I said, this is first pass, and a demonstration it works.

I’ve tested this with a wide range of settings in the Windows 7-zip binary and haven’t managed to make a file which fails to decompress yet. Both 32-bit and 64-bit Windows MAME builds have been tested and as long as you’re sensible about block sizes and compression used speed / memory overhead is comparable to standard .zip (ie don’t use PPmd with huge block sizes on huge ROMs because that would just be stupid and can take a good 3 minutes to decompress kof2003.zip even with the official tools ;-)

I’m increasingly being sent .7z files when asked to look at things to work on in MAME as opposed to zips in days gone by. I’d attribute a lot of this to the decline of WinZip (they practically nuked their own business model) as well as the horrendous build-in Zip support Windows has, forcing people to look for alternatives. This means having .7z support makes MAME work easier as I don’t have to decompress / recompress everything just to check it out. Weigh that up with the limitations of the format and moving forward seems like a good idea. I’m also often sent .rar files (because people appear to have flocked to either 7-zip or WinRAR in equal numbers) but I need to do further investigation on the suitability of decompression code for those, there are a few strings attached because it’s not a ‘free’ format.

Of course, the original .zip support remains, .7z is just an option. There is no reason to rip out the long standing .zip support, merely supplement it with something more modern.

The other benefit of course is that the LMZA and LMZA2 algorithms will be in the MAME code for leverage elsewhere, and would be ideal for use in CHDs and such. Not done anything on that front tho because it’s not a priority, and the FLAC work is still pending cleanups and being officially turned on.

Ok, this update has nothing to do with Donkey Kong at all, Donkey Kong doesn’t even use samples anymore, but nevertheless it is about Monkeys.

Actually, maybe it isn’t about Monkeys at all either, but it is about the lossless audio compress format ‘Monkey’s Audio’ or, those .APE files you sometimes find.

Monkey’s Audio (MA) is one of the other big players in the lossless audio compression arena along with FLAC, and WavPack.

The main advantages of MA is that it offers better compression rates than FLAC, which for large files can be quite significant. The disadvantage is it’s slower, quite significantly slower. It’s a symmetric algorithm which basically means decompression takes the same amount of time as compression.

0.145 saw the FLAC code being enabled in MAME for use with samples (not yet CHDs), and as a test I’ve done a bit of porting and brought MA over with the same sample decoding functionality. It took a bit of work keeping the compiler happy (wasn’t quite 64-bit safe), and I’ve opted to compile the non-windows codepaths on Windows to ensure that it’s as portable as possible, but it works.

As expected, it’s quite a bit slower than FLAC, and while it’s not quite as fast as it could be due to me disabling some x86 ASM / MMX code the sample implementation still gives a pretty strong indication that it wouldn’t really be fast enough for decoding of CHD data where we need data to decompress with minimal overhead. I know a few people had been wondering / asking about this, which is one reason I looked into it. I might do a hookup later to test, but IMHO it’s not worth it.

You can download the source diff to add MA support to samples here (I don’t claim it to be pretty!)

Also for reference the following Gorf samplesets (note, some files are not recorded, so MAME will still show ignorable NOT FOUND errors on those rather than silently failing as it would prior to 0.145, I’ve also resampled one of the clips to 8-bit to show 8-bit decompression is working)

As APE (860,649 byte zip)
As FLAC (905,243 byte zip)
As WAV (1,110,709 byte zip)

I wouldn’t hold your breath on this being officially included, although I’ll submit it anyway. More of a fun little experiment than anything else.