Ok, this update has nothing to do with Donkey Kong at all, Donkey Kong doesn’t even use samples anymore, but nevertheless it is about Monkeys.
Actually, maybe it isn’t about Monkeys at all either, but it is about the lossless audio compress format ‘Monkey’s Audio’ or, those .APE files you sometimes find.
Monkey’s Audio (MA) is one of the other big players in the lossless audio compression arena along with FLAC, and WavPack.
The main advantages of MA is that it offers better compression rates than FLAC, which for large files can be quite significant. The disadvantage is it’s slower, quite significantly slower. It’s a symmetric algorithm which basically means decompression takes the same amount of time as compression.
0.145 saw the FLAC code being enabled in MAME for use with samples (not yet CHDs), and as a test I’ve done a bit of porting and brought MA over with the same sample decoding functionality. It took a bit of work keeping the compiler happy (wasn’t quite 64-bit safe), and I’ve opted to compile the non-windows codepaths on Windows to ensure that it’s as portable as possible, but it works.
As expected, it’s quite a bit slower than FLAC, and while it’s not quite as fast as it could be due to me disabling some x86 ASM / MMX code the sample implementation still gives a pretty strong indication that it wouldn’t really be fast enough for decoding of CHD data where we need data to decompress with minimal overhead. I know a few people had been wondering / asking about this, which is one reason I looked into it. I might do a hookup later to test, but IMHO it’s not worth it.
You can download the source diff to add MA support to samples here (I don’t claim it to be pretty!)
Also for reference the following Gorf samplesets (note, some files are not recorded, so MAME will still show ignorable NOT FOUND errors on those rather than silently failing as it would prior to 0.145, I’ve also resampled one of the clips to 8-bit to show 8-bit decompression is working)
As APE (860,649 byte zip)
As FLAC (905,243 byte zip)
As WAV (1,110,709 byte zip)
I wouldn’t hold your breath on this being officially included, although I’ll submit it anyway. More of a fun little experiment than anything else.
(I’m going to look at .7z support for ROM loading next… no doubt hell will have to freeze over before that gets added in any official build, but I’ll throw it in an UltimateMAME one because it looks like hell is going to have to freeze over before any of that gets accepted either.. as always, I stop pushing for it and everything stops moving / goes dead)
Well the basic .7z decompression functions compile cleanly on Windows at least, which is a good start.
Just need to see how hard it will be to integrate it into the existing zipped file support framework.
I say, officially speaking, 7zip support would indeed be something quite awesome. Push the envelope, Haze.
Yeah, everything people send me to look at these says is either a .7z or .rar
RAR has a few strings attached, so I’d rather not (afaik you can use the decoder, but with stipulation that nobody use it to develop a competing encoder / decoder, which is a bit of a nasty license term)
7z is more open and the decompression part isn’t too bad, probably cleaner than most of these Audio ones. The license is non-restrictive as well.
It would be nice to actually be able to deal with the .7z files people send without being having to decompress and recompress them all the time which would be my primary reason for doing this (not the space saving, because I’d imagine decompression when using the more tightly packed modes would just be stupidly slow anyway)
Why wouldn’t they support it? Is there any reason, other than historical/ideological, that makes huffmann a better candidate rather than lzma as decompression algorithm for storing mame archives?
One could argue that nowadays both of them are mostly superflous, considering that Hd space is relatively inexpensive, and games that require them most ( Naomi CHDs, for example) already store most of their graphic assets already compressed using jpeg / RLE routines.
As I’ve said above, it’s not so much about space when it comes to zip vs. 7z (even if you were to have an entire set of MAME / MESS roms it pales in comparison to the MESS CHDs and modern HDD sizes for example)
Convenience is the key issue.
Back in the day zip was pretty much standard on PCs / Windows. Everybody had WinZip installed because it was that program that unzipped things, where it would nag you, but you never *really* had to pay for it because it just kept working. Commercial users had licenses for it because it was the software people used at home and it was familiar.
These days nobody has WinZip. They started making the newer versions actually disable themselves / nag you excessively after the trial period, and everybody looked for alternatives. I’ve not even seen a company with a commercial license / install of it in years either because there’s no demand for it, even if Windows build in ZIP support is diabolical. Effectively by nagging people too much, and trying to cut out piracy they nuked their entire business model and widespread use of the format on Windows.
As alternatives people picked up the likes of 7-Zip which gained popularity. Now where I would have once seen WinZip installed I see 7-Zip installed (or WinRar) People even actively *avoid* WinZip because of the new incompatible zip standard they introduced (which was needed, a standard Zip can’t hold >2GB of data)
zips are, at this point, essentially a relic of the previous decade in Windows, whereas I believe Linux has always favoured .tar.gz anyway (and now some distros have LMZA based compressors like 7Z)
With that in mind, I think it makes sense for MAME to keep with the times, not due to better compression ratios, but simply to make life easier for people working with it, and not look like a dinosaur.
At the point where you have the compression algorithms available in the code, providing they offer adequate performance, it makes sense to use them if they’re better too.
In the past 7z has been shunned by the rest of the developers, because it wasn’t a solid standard, and was just about saving space for people. In the past I would have agreed, but I think things have moved on since then, however a lot of dated viewpoints still exist within the team so I’d expect some resistance in trying to get support included, much as there has been with the UltimateMAME concept.
Its always refreshing to read your posts, a mamedever who speaks with a clear understanding of the world outside of mame. Aaron Giles is taking mame nowhere and this is why you should replace him as leader again, you have a vision
7z would be a revelation, please do it, please please please
Hmm… well Aaron hasn’t been in charge for a while anyway, Kale has, not that you’d notice the difference, be it for better or worse, Kale has yet to really make his defining mark or change on the project.
I’m working on 7z anyway, as outlined above, the existing loading using zip is however quite involved, and I need to figure out exactly how it works at present first. Luckily it seems like it already has some useful features I can build on such as caching the zip state which will be handy because 7zip really expects you to have information from the previous call cached if you’re dealing with solid archives.
“Aaron hasn’t been in charge for a while anyway, Kale has, not that you’d notice the difference” That’s exactly what I’ve been thinking. The changelogs look exactly the same as they have for years.
Interesting about the anniversary release of mame too, considering your previous post on the subject. I read the changelog looking for something significant but it seemed actually to be one of the more lackluster release in a long time.
Nicola is right, the project needs a leader with vision if it is to progress rather than move sideways.
Well in all honesty I’m not sure how ‘forward’ I could take the project alone anyway. Sure I could (officially) introduce things like the UltimateMAME, give people options, attempt to bring about a situation where everybody is working together but that still needs people who aren’t afraid to do some actual work.
I could hammer out a plan of how things should probably be, areas for concern, but again making improvements isn’t a lone man job.
Modernizing the codebase is fine, and to a degree needed, but modernizing functionality is just as important, which is why I’ve been looking at things like this 7z support, and FLAC / APE support, but even most of that requires other people to be involved, I can give a proof of concept, example of how something works, but then the rest of the devs still have to bash it into a shape they all agree on.
We’ve seen the traditional December/January ‘blip’ of activity now, if that momentum carries through is yet to be seen. Most of the good stuff has been happening in MESS as per usual tho, half the improvements seen in the last MAME release are just scraps of progress from there (x86 emulation improvements and the like where there has been phenomenal progress in MESS, but just some crappy ChaseHQ rip-off almost working, and running at 9% speed to show for it in MAME, it simply doesn’t do the changes justice at all from the view of somebody just following MAME)
I guess it’s the smaller things like the ones I’ve been suggesting / pushing for which do add up to an overall direction, but by my standards at least, it’s not really enough.
The good thing about the c/c++ modernizations that are going on is that once they’re finished it should become a lot easier to emulate something like Race Drivin’ Panorama without resorting to ugly code, having a level beyond the current ‘machine drivers’ where you can include / configure actual whole machines in the hierarchy then add the glue logic between them would be a lot cleaner than anything you can do now which I guess remains Aarons single real plan or ‘vision’ as the above poster uses, so he does have one, but it is a lot of work with very little interesting to show that can capture the public’s imagination in the meantime.
Obviously we can’t bend to the will of todays emu kids in every way, things like ‘texture packs’ on N64 / PSX emulators are just plain horrific hacks, inspired I guess by the modern remakes. They’re not really emulation at all but none of my suggestions have really been along those lines.
I don’t think things like an official networking model would go amiss, more users of official builds = more testers, as opposed to people resorting to dodgy hacks for netwokring with the risk of having their PC compromised using hacked up drivers which are of no worth at all when it comes to bug reports, again, it would need somebody to implement it tho. ClientServerMAME, the last noteworthy, and decent attempt now seems to be lagging behind at 0.143 (which is a shame, 0.145 brings about some important save state fixes which are essential for it) That tends to be what happens when changes aren’t embraced by the mainline, with people sticking to the dodgy, buggy netplay system they know instead, had support been integrated into mainline it would have been updated with the main builds, and people wouldn’t even be needing to seek out anything else.
That doesn’t mean I see MAME as some toy for playing games either, I’ve outlined other plans, how MAME could be used to provide a virtual drive emulation platform for connecting to real hardware systems to replace failed original drives and such. MAME should strive to be the best repository of component emulations around. All what I’ve said means is that I think MAME should at least make some efforts to keep users on the current mainline builds without compromising the goals of accurate and correct emulation. Without users you could put out something which flashes your screen purple and reboots to a BSOD and you’d never know it. When you’re making BIG changes like the tag ones Aaron made lately you need as many users as possible to catch any possible fallout as quickly as possible, I’d be very surprised if there weren’t still lingering issues in MAME 0.145 / MESS 0.145 due to a lack of testing.
and no.. 0.145 wasn’t very special at all.
I’d heard little rumours of things, some rare old game being fixed, cool riders, hng64, konami m2, sega m2 and other stuff, more legal roms, even official ‘ultimate’ builds but nobody had shown any solid evidence of anything happening and usually screenshots or emails leak out, so I wasn’t exactly optimistic.
Boogna Boonga / Spank ’em was hyped up, but in the end it’s just an overpriced, glorified crane machine / gambling game, where it decides if you win a prize moreso than the actual power level you use.
Add to that there were still bugs with st-v (and others) some of which have since been fixed, and it was a bit of a duff 15th anniversary build in all honesty.
An Ultimate MAME build with instructions on how to get MAME 0.1 running in the PC driver from MESS (obviously without supplying the roms) would have been cool, and reflected what was done for the 10th Anniversary well (the updated 0.1 to run on modern systems) It would have also been a good indication of where progress was at the moment. If there are other emus that run in MESS (and I’d be very surprised if Retrocade and early Raine builds didn’t) then it would have been equally cool to offer similar instructions for those, fulfilling the ‘MAME is Borg’ prophecy…. but instead.. nothing.
That’s what I would have done anyway, but doing it here just doesn’t really have the same effect / cool factor as doing it officially would have had.
(to clarify, as a normal release it would have been fine, albeit a bit buggy, as a 15th anniversary build I feel the opportunity to do something really special has been passed up on)
You make some excellent points, the most important being the need for more participation. People have been clamoring for netplay and 7zip support for years. Netplay alone would bring in a whole group of very enthusiastic users.
UltimateMAME is an out of the blue stroke of genius, flac support is another plus.
As many weird alternate builds exist out there, an UltimateMAME build with flac/7zip/netplay/missing dodonpachi etc. would fit right in. Somebody would compile it as long as the code is available.
I realize that’s not what you want. You’re trying to focus talent into the official project and avoid splintering it. That’s fine, either way I’m positive your new ideas will be incorporated at some point because their time has simply come. If Edison had not made the light bulb any number of other people would have.
BTW, how to contact you privately?
I can’t offer builds with the missing Cave games because Cave requested they be removed / not be worked on. Also I know almost nothing about netplay code, it’s quite a complex subject if done properly, I just fear the window of opportunity for integrating the CSMAME style solution (which unlike previous ones was AFAIK within the license) has passed and now that will just turn into another good side project where the code simple rots away. I think some tighter work with the core devs (standards required, stuff like that) could have resulting in that being more official, but trying to maintain it outside of an ever-changing core project is just too much work for a single external developer.
For contacting me this really is the best place to contact me for 99% of things. I’ve had to close my most recently used public mail account due to spam / hack attempts on it. It’s rare a subject is so sensitive it needs a private mail ;-)
I just mentioned the cave games as something that gets people excited enough to get them to compile their own builds. They already go through the trouble of learning how to use kaillera.
The only reason I want to be private for a brief time is to protect someone else’s privacy, not mine or yours. :) Try
h u ggy ba by_AT_gm a il_DOT_com
How cute♥, wow: ‘♥’ working, which encoding is that?
what would be the advantage of 7zip when some windows-created solid archives cannot be uncompressed by the official library on Unix and Mac?
for sure I can foresee its success in moving away from the project Arbee, judge and other productive devs which do not use Windows for developing…
also, your statement “as always, I stop pushing for it and everything stops moving / goes dead” is plainly ridiculous
if you’re referring to FLAC, the code is going to get added back soon by Aaron but he spent his spare time in bugfixes before the 0.145 release rather than in finishing the chdman changes and I support his choice of priorities
if you’re referring to progress in the project, PC improvements (with nice side effects on systems like FM Towns) are there to prove you wrong and to show that we are definitely far from the project “going dead”
sometimes I think you deliberately forget about stuff you know, to keep supporting your ‘I’m so better than the official team’ arguments
“what would be the advantage of 7zip when some windows-created solid archives cannot be uncompressed by the official library on Unix and Mac?”
I’m yet to see any evidence of this. The source code for the Windows 7-zip is right there, the library it compiles is the same. If there is a bug, it should be fixed, IMHO there isn’t a bug and it’s just some bullshit or corrupt files (maybe created with a too big dictionary size triggering out of memory conditions?). *everybody* else is using .7z now and I’m not seeing complains en mass from users of ports of emus which support .7z
*If* there are problems, the community will work around them anyway, people aren’t stupid, if MAME won’t decompress archives created with certain settings people won’t use those settings. Right now MAME won’t decompress zipfiles made to the new zip standards (zip64 or whatever it’s called) because it simply doesn’t understand them. People know this, and aren’t creating files to those standards either. If certain people are to be believed you can even create standard zips right now that our zip / zlib based decoders can’t decode. Furthermore, you’d have an exact side-by-side test case to debug something if you found a file that decompressed on windows but failed on linux, allowing MAME to contribute back to the rest of the community if needed (or does that not matter to you either?)
Anyway, real reasons, let’s start off with the zip file limit. Happy Fish will have too much data to store in a standard Zip file, we’re going to see more and more games / dumps in the future with huge amount of flash ROM, way beyond what you can even store in a plain old .zip. A fully merged set of one of the gamtor sets hits the limits too, it only works because I ‘merged’ the filenames on some identical files to avoid it. Those are the initial warning signs that the format is outdated it would be foolish not to heed them.
Plus the general reasons outlined in previous posts which you appear to have chosen to ignore.
as for things going dead, it’s pretty obvious I’m talking about the UM stuff.
Also remember, this would be an option, nobody would be pointing a gun at your head and saying ‘use 7z’. It gives people a choice, and makes life easier when people do send .7zs. MAME is an emulation project, that doesn’t mean the actual software needs to be stuck in the last decade too.
your negative attitude to progress and real changes really drives me up the wall.
well, a couple of months ago I had been sent a 7zip file by FatArnold with some png snaps for projectMESS which was recognized as “invalid format” by any 7z.exe on Mac and Unix and only worked on Windows. it was a ~100kb file, but it’s gone with megaupload and I hadn’t saved a copy of the file given that it was useless on my computer :(
I cannot exclude it was a corrupt file, created with some broken version of 7z.exe for Windows, but still FatArnold was perfectly capable of extracting the files on Win and re-compress them to zip with no problems
overall, this made nothing to improve my attitude towards 7z, and as much as I’d like to find a solution for when we hit the zip limit with some romset, I still don’t believe this is the best solution
for the other reasons, I really don’t follow your implication that simply because nobody install anymore Winzip, this should make the whole zlib an outdated library. Windows itself implement zlib allowing you to open zipfiles as directories, and Unix and MacOSX have perfectly good zlib support as well, proving that the library is very solid and portable.
also, I seem to recall that if you want to access a single file inside a solid 7z archive, you had to uncompress in memory the whole thing, meaning that you’d need a lot of RAM and a lot of loading time to e.g. handle some n64 solid-ified romsets, trading off the HD space gain with tedious wait time at start… not exactly my dream ;)
back on general improvements topic, OTOH, I was pretty interested into your APE experiments, even if my personal experience with the library when dealing with pakkiso’d redump sets was not so exciting (loooong compression/decompression times)
too bad the result matched my experience…
For Solid files you have to decompress whatever the solid blockfile size used was, that then gets cached for subsequent calls.
That would be a user choice. Sure, if the user selects 1gb solid data it’s going to be slower / more costly in time, but the user can also choose 0kb solid block. That’s their own choice.
The windows ‘zipfolder’ support is a complete disaster, ever tried decompressing a zip using it across a network, or even the homegroup? it can take HOURS for a 2mb zip, it alone has done much to destroy zip credibility with some users.
7z is a more modern (but still, at this stage mature) format without many of the strings and limits attached to the alternatives.
It might not be *perfect* but nothing starts off *perfect* but it’s a lot more viable than continuing to depend on a creaky old zip format. (AFAIK Mame doesn’t even use the latest zlib, so there are some nice exploits in there too)
zlib will soon IMHO be universally considered an outdated library, it’s useful life for most users ended around the time DVD rips started being commonplace, and FAT32 as a filesystem was deprecated to portable devices. (Win7 won’t even install to FAT32) It’s a legacy format and legacy library at this point.
I think 7z is a ‘nice to have’. Right now the zip format is per file and not solid. So you can not pick off a file and move it around like they are doing in cmp. You just have to recompress it. Other than that the ‘have to decompress the who thing’ is a rather silly argument. You have to decompress it all anyway at load time anyway unless you are using a merged set. Unless I am missing something in the way MAME works?
Probably the best way to get 7z in there would be to put it in CHDs first. You can work out the kinks there. Such as it breaking on one platform vs another. Just putting the 7zip deflate style compressor in chdman would be an immediate improvement of 1-5%. Then leaving the old decompress code there would seem rather odd…
From my experiments 7z beats deflate usually by about 5-15%. Even better if it detects a duplicate file.
@Haze: I know zip/folder support in Windows sucks, it was just an example of how common and cross-platform zlib implementations are
@me: what I was referring to is that (to my knowledge) single files from zips can be extracted without touching the rest of the archive, while if you use solid archives you have to fully decompress a block into memory to access a single item inside the archive.
for small files and archives it’s not a big deal, but in that case you don’t gain that much space with 7z either.
for large files it *might* be a problem, especially if a very large block size has been used to create the solid archive (I’m not an expert about 7z, so I’m not sure if the larger the block, the better the compression or if there is some)
let me take a toy example to fix the ideas: say you create a solid 7z archive containing the NES games zelda1 and zelda2 and the N64 ones, plus their clones. then to get out of the archive the 32k of the NES game you will probably have to uncompress into memory several dozen of MBs (because I’d guess you need larger block to gain compression in the larger N64 roms…)
in your example, if you were to specify a huge solid block size, then yes, you’d have to decompress more files.
that however relies on the unlikely conditions of
a) somebody using the full merge option on MESS sets, ignoring the sub-directory system.
and
b) somebody specifying a stupidly large solid block size on those sets.
for the most common use cases you *want* the majority of the data in a zip, so decompressing the solid block data isn’t an issue because it means decompressing the rest of the files will be instant.
on a modern system with a decent amount of memory the worst case isn’t even that bad anyway, even if it would be a bit stupid to store with such high solid sizes.
solid blocks are an optimization in the format, they’re optional, nobody has to use them. Having that *option* doesn’t make it a worse format, just as having the *option* of MESS stuff in MAME doesn’t make UltimateMAME a worse MAME build.
There are other reasons why you wouldn’t want to use solid blocks anyway (and why people don’t) and I imagine the majority using the format for MAME would be turning them off because it’s easier to add new files to old 7zs that way. That essentially leaves you with a more modern 64 bit container capable of storing more modern compression formats (including the plain old inflate stuff zip uses) If you really wanted you could just see them as zip files with a 64-bit capable header. Having the option there for people who want to use it is no bad thing tho.
You make out ‘solid’ to be some massive problem, it isn’t, at least not unless you specifically make it one.
For the record, the official LZMA SDK definitely cannot extract all valid 7zip files created by the Windows tools. (The Linux/Mac command line 7zip extractor shares this problem). I implemented it years ago in Audio Overload and about 30% of 7zip PSF2s found on the Internet would not extract using that code. The official 7zip binary on Windows always worked fine with the same files.
One thing they don’t make clear: the SDK code is *not* the same code that is used in the Windows 7zip program; it’s developed separately and that’s how these problems occur.
Well you can download both the sources to the Windows 7Zip app, and the SDK.
For the ‘C’ part both seem to contain the same code and same sample app.
If there’s a difference maybe it’s in the C vs. CPP implementation?
I don’t really see it as a major issue tho, right now you could easily produce zips which don’t extract (password protecting them will do that for you in an instant)
I trust the community to decide what works and what doesn’t, they’re used to dealing with the countless other emulators which use .7z by now. MAME will be using the same code on Windows as on other platforms. It’s also possible whatever issues there were in the past are fixed in current versions, I’ve tried a variety of settings on the windows app, and every test case has decompressed fine with the sample code built within the MAME codebase.
Having the option there won’t hurt anybody, and if there are problems with the 7Zip binary producing incompatible files then they’re the ones who will see more of a backlash.
fwiw I currently have kof2000 running from a .7z, bit hacky for now, but basics are in place :-)
Ok, I’ve submitted a first pass working 7-zip support to R. Belmont for portability checking.
Hopefully it will be added to the main builds, but if there’s no sign of it in a few hours I’ll post it here.
Still could do with a few tweaks / cleanups but those will come with time. I tested various options and all files I created decompressed fine…
@etabeta my point was, unless I am missing something here maybe I am, Doesn’t MAME uncompress the whole set into memory anyway? Unless you guys are using memory mapped files and decompressing it on the fly (which if I remember from when I dug thru the code last is only is done for chds).
If I remember correctly also the only reason to have separate would be for something like clrmamepro. Which if I remember correctly has 7z support.
Cross platform compatibility could be a reason not to do it. It could be a touchy issue. Unless someone makes an easy way to convert it back over to zip.
Also your toy example do people really do that? I thought most people kept things as separate compressed archives.
In my experience 7z is usually ~5% or better than the deflate support in 7z (which is already pretty good).
Now where 7z takes it in the shorts is memory usage. It is usually has a much higher requirement to decompress things. Not a real problem on fairly modern hardware. But not everyone has that.