7-Zip-a-Dee-Doo-Dah

As promised below I’ve put a bit of time into hooking up 7-zip support in MAME. This is a first pass, so there is still room for improvements (both to code quality, and potentially bug fixes)

The way I see it zip / 7z offer the following pros / cons

.zip
– Aging 32-bit format
– Can only hold 2gb of data (we’re hitting that limit in some cases)
– Only supports UTF-8 filenames
– Single, dated algorithm
+ low memory overhead

.7z
+ Modern 64-bit format
+ No practical limitation to filesizes
+ UTF16 support (debatable advantage)
+ newer more advanced algorithms
– higher memory overhead if ‘solid’ blocks are used

Zlib (.zip support) might be tried and tested, but it’s showing it’s age, it’s a format of the previous decade when FAT32 was the standard, and nobody had large files. .7z is newer and doesn’t have many of the inherent issues of an older format, although does still need some testing. Staying ahead of the game is always a good idea tho, better to be modern and be prepared.

Download the diff here. This has been sent to R.Belmont for portability checking and official inclusion. Right now this can be used for roms, artwork, samples, cheats etc. -romident and romcmp.exe aren’t hooked up to it yet, but that will come with time. As I said, this is first pass, and a demonstration it works.

I’ve tested this with a wide range of settings in the Windows 7-zip binary and haven’t managed to make a file which fails to decompress yet. Both 32-bit and 64-bit Windows MAME builds have been tested and as long as you’re sensible about block sizes and compression used speed / memory overhead is comparable to standard .zip (ie don’t use PPmd with huge block sizes on huge ROMs because that would just be stupid and can take a good 3 minutes to decompress kof2003.zip even with the official tools ;-)

I’m increasingly being sent .7z files when asked to look at things to work on in MAME as opposed to zips in days gone by. I’d attribute a lot of this to the decline of WinZip (they practically nuked their own business model) as well as the horrendous build-in Zip support Windows has, forcing people to look for alternatives. This means having .7z support makes MAME work easier as I don’t have to decompress / recompress everything just to check it out. Weigh that up with the limitations of the format and moving forward seems like a good idea. I’m also often sent .rar files (because people appear to have flocked to either 7-zip or WinRAR in equal numbers) but I need to do further investigation on the suitability of decompression code for those, there are a few strings attached because it’s not a ‘free’ format.

Of course, the original .zip support remains, .7z is just an option. There is no reason to rip out the long standing .zip support, merely supplement it with something more modern.

The other benefit of course is that the LMZA and LMZA2 algorithms will be in the MAME code for leverage elsewhere, and would be ideal for use in CHDs and such. Not done anything on that front tho because it’s not a priority, and the FLAC work is still pending cleanups and being officially turned on.

 

36 Responses

You can follow any responses to this entry through the RSS 2.0 feed.

Both comments and pings are currently closed.

  1. Darkstar says:

    hmmm… why are you listing “UTF16 support” as pro for 7zip? UTF16 is clearly inferior to UTF-8:
    – UTF16 is not endian-independent so you might end up with difficult-to-track-down bugs in big endian machines
    – UTF-16 requires completely different string handling methods, due to embedded -bytes. For UTF-8 you can use the well-known str* functions as there are no bytes inside the string, ever
    – utf-16 requires basically twice the amount of data than UTF-8 (which is not a big problem for filenames, I admit, but anyway)
    – utf-8 can be grepped etc. with all available tools today (especially if most of the text is basic ascii), whereas utf-16 is mostly unsupported

    so, please don’t switch to utf-16 but instead stick to utf-8

    -Darkstar

  2. Darkstar says:

    Also, I think it’s called “solid” blocks, not “sold” blocks …

  3. Haze says:

    well, typo fixed.

    I still consider it an advantage tho, if we want proper names we’re ultimately going to end up having to encode non-ascii characters and IMHO it’s a simpler way. Support within the format is standard anyway and endian order is dictated by that (and I can’t change that, otherwise it defeats the objective of allowing the 7z files people send me to be used!) This kind of thing is trivial compared to the endian madness MAME has to deal with every day.

    MAME currently only supports UTF8 anyway so the extra data is ignored, extra code would be needed to be fully compliant, and my focus was instead on just getting the decompression support working for the most common use cases.

    Either way, it’s not going to break anything in MAME because MAME loads primarily by the CRC, doesn’t even care what the filename is unless it can’t find any CRC match, the support is simply ‘there if we choose to use it as opposed to ‘not an option at all’

  4. etabeta says:

    I still personally believe it’s not worth addition to the main MAME tree (not to mention that Arbee did not even forwarded it so far, probably being busy with tests on other OSes) but I guess the majority opinion will win

    Let me anyway explain why I’m against before you think I want to slow down the project or I’m against it:
    – you mention utf-16 as a pro, but exactly like Darkstar, I believe it’s not, due to endianness issues
    – for the 2GB limit, you are right, of course, but the rest of the pro/cons are subjective: dated algorithm means also established and reliable standard, while newer one it’s not

    also you (purposely?) ignored in your post the fact that the official 7z exe producing archives incompatible with the official 7z SDK is a huger cons than the possibility to create password protected ZIP archives… producing uncorrupted and unprotected zipfiles that MAME cannot handle is a lot harder (almost impossible since Firewave fixed a bunch of flaws on the MAME side two years ago) than create 7zips that *no* Unix/Mac extractor can handle

    This means that if you use the official portable library, you end up with attracting bug reports about 7zips which cannot be opened by the emulator but can be regularly extracted by 7z.exe… i.e. a lot of user will automatically blame MAME for this. Can you see why my attitude towards 7z is negative?

    I know you keep pushing your point of view as if it’s the Only And Holy Truth, and you don’t care about different opinions, but my objections still stands…

  5. Haze says:

    I have not managed to produce a single 7z archive using current versions which cannot be decoded

    I’ve tested an exhaustive set of options, weird block sizes, dictionary sizes, compression types, word sizes…

    Bugs happen, bugs get fixed. IMHO it isn’t an issue, at all and probably doesn’t even exist anymore.

    *IF* such problems occur and happen only on mac / unix then MAME can give back to the community by fixing them, this is a positive. Note, I’m *using* the primarily Unix/Mac codepaths on Windows.

    I’ve looked at the code, there is nothing magic about it which means such bugs are inherently likely either.

    You can’t block submissions based on some old bugs or would you like me to say MAME is shit and should never be used because there are bugs in the MAME4All versions?

    People on the whole are using .7z rather extensively these days, for all you might not like about it, design decisions you might have made differently about the format etc. etc. you can’t change that. You can be compliant, or you can die. The alternative is you invent a new format and become a laughing stock by supporting something with no support elsewhere, see CHD as a format which for all it’s virtues and ‘good design’ (ha) is rejected by the masses.

    If I’m sent a .7z I want to be able to use it with MAME. This makes my job, as a developer, easier.

    This is especially true for MESS where the majority of console emus support .7z and people expect their .7z files to work. MAME/MESS get seen as very ‘backwards’ there.

    You seem afraid to even *try* and it’s this kind of negativity towards new ideas (both this, UM, and others) which makes me think recommending you for mamedev was one of my biggest mistakes. You cannot progress without taking some risks, without trying new things, without putting things out in the field for testing. Mamedev was always about pushing boundaries.. not finding excuses.

    I find it stupid, you let Aaron run rule over things, breaking whatever he wants, all the time, regressing things which worked fine before, without offering anything new from an end users point of view, but the very chance there *might* be a bug in some code I put forward means you reject it without even giving it a real field test, despite it actually offering something new which people want. The former does more damage to peoples view of the project than actually giving them something new to play with, and if the code does turn out to be too buggy it can be pulled. *Both* are just part of normal development, new features are never perfect first time, changes break things.

    We’re at the start of a ‘u’ cycle, plenty of time for experiments and testing, it’s not like I’m trying to make critical changes right before a major release so how about we give things a shot and deal with any issues if and when they arise? I can’t fix something I have no current test cases for, fighting imaginary monsters here…

  6. Darkstar says:

    …and only now I see that my comment makes no sense… Seems my “backslash-zero” bytes got filtered out. Before I look like a jerk for writing incomprehensible garbage, here’s the fixed “con” point of UTF16… ;-)

    – UTF-16 requires completely different string handling methods, due to embedded 0-bytes. For UTF-8 you can use the well-known str* functions as there are no 0-bytes inside the string, ever

  7. Haze says:

    Yeah proper support will require some different string handling, but it really is a minor issue.

    Worst case scenario, MAME gains a few functions for converting between the filenames in the .7z and what it expects. As I said, endian is dictated by the format, not the endian of the system it was produced on. No big deal, at all. The rest of the world is coping just fine with it. You find basically the same ‘issue’ with any format containing 16-bit data, to highlight it was a blocking issue here is just silly.

  8. Darkstar says:

    Whoa, well I for one am definitely not against the inclusion of 7z support in MAME/MESS. Au contraire. But IMHO the only real benefits right now are the >2gb file support (which will be required at some point) and the addition of new (and probably better/faster) algorithms for CHD files (although I still think that LZO would be even better than LZMA)

    That’s two big benefits (one right now, one possible future benefits) for 7z over zip.

    Your other points: UTF16 support (I don’t think there will be ROMs named 大事なロム.bin for the foreseeable future), better compression (zip files account for 40GB of almost 300GB of MAME data, if 7z saves 20% over zip, which I think is quite optimistic, then that’ll still be only 3% saved overall) are not very strong “pro” points for the 7z format

    Another thing: does 7z still use the good old CRC32 checksums?

    -Darkstar

  9. Haze says:

    Yes, 7z still uses the good old CRC32 in the header, so MAME’s ‘quick lookup’ works (otherwise MAME wouldn’t be able to load by CRC without decompressing the lot). It’s a shame they don’t also include a more secure hash, but I guess speed / storage space was a consideration when the format was designed.

    The storage saving is secondary to me, but obviously some people will like it. The entire MAME set is marginal in size compared to the CHDs, especially MESS CHDs, but given how some people were moaning about the couple of GB the fruit machines added, I guess some people do care.

    You’re right, we probably won’t be naming roms like that in the near future, at least not on the MAME side where we still don’t even allow capitals for whatever reason. Maybe some of the software in MESS would benefit from allowing it tho. I highlighted it as a difference in the format, one which I felt was beneficial (and obviously others disagree), but at this point in time it’s not going to make any difference whatsoever.

    Basically the other alternatives are marginally used with hardly any support (zip64), or non-free (like rar)

    MAME will outgrow .zip sooner or later, so IMHO it makes sense to get the code in now and let it mature with the project then by the time we come to depend on it, it will be ready and trusted.

    By not embracing change and progress MAME gets left behind. MAME could easily be doing what SuperModel does now if 3D hardware had been embraced, and IMHO the emulator as a whole would have been more accurate and advanced than it is now if MESS and the requirements of MESS had been embraced at an earlier stage too. The latter could possibly be blamed on me, I should have worked harder to keep MAME and MESS close when I was in charge, I guess I never realised how similar they really were at the time.

  10. me says:

    @haze there may be one more pro you are missing? If I remember correctly the 7zip library has support for decompressing/compressing a lot more formats such as iso, rar, arj (anyone still use that?), etc etc… Obviously right now your are focused on just getting 7z in there. So it may be worth thinking about how to make that more generic (you may already have the makings of it)? Also I think one of the zip formats 7zip lib handles is that zip64 and decompressing rar formats.

    From the 7zip main page

    Packing / unpacking: 7z, XZ, BZIP2, GZIP, TAR, ZIP and WIM
    Unpacking only: ARJ, CAB, CHM, CPIO, CramFS, DEB, DMG, FAT, HFS, ISO, LZH, LZMA, MBR, MSI, NSIS, NTFS, RAR, RPM, SquashFS, UDF, VHD, WIM, XAR and Z.

    There is one more con you may want to add. The 7zip library is currently what the author considers alpha. The last beta was nearly a year ago. So it is an evolving format. The whole thing is a good add. But some caution will be needed here. I think that is what etabeta is getting at. It is going to kick up some dust as it were…

    @etabeta if you have particular settings that dont work lets get those up front here. If you dont have the settings that is ok. Those just need to be figured out. Then it can be detected in the code and flagged at runtime with maybe a suggestion of different settings to use (some sort of warning as you pointed out it usually works fine in windows)? Instead of guessing something is wrong? Would that be a decent compromise?

  11. Huggybaby says:

    etabeta said: “I know you keep pushing your point of view as if it’s the Only And Holy Truth”…what, Haze should push YOUR point of view? Why don’t you write your own damn blog? And who appointed you “defender of all things mame as they are and are ever meant to be”? Now I see why so many unofficial builds of mame exist: some rotten core of mamedev exists to maintain the castle walls as an homage to past standards. You know, every once in a while it’s good to poke one’s head outside

    “and you don’t care about different opinions”…I guess we’ve been reading an entirely different blog then. AFAICT Haze posts a well reasoned conjecture and asks for comments.

    You are now officially a troll, a word I don’t use lightly.

    “but my objections still stands”…at this point, who can even freaking care? I never see your name on a changelog anyway, but Haze is all over them. Why don’t you go fix some dipswitch defaults, or rename an eeprom location, or mark some clone as “not working”.

  12. me2 says:

    First flac, now 7z. Thanks Haze!

  13. nin says:

    As for rar, maybe unrar-free (http://packages.debian.org/sid/unrar-free on Debian, its real homepage at gna.org is down as of this posting) is an option.

    And about UTF-16, UTF-16 is definitely worse than UTF-8 – not that it would matter for MAME – see http://programmers.stackexchange.com/questions/102205/should-utf-16-be-considered-harmful

  14. Haze says:

    @me

    the 7-zip application supports those formats, and indeed they distribute decompression functions for them, however they’re not part of the 7z format, so I’m not importing that code. It’s a handy resource if you do need the functions to use those formats tho.

    Don’t confuse the 7-zip application and the 7-zip file format, the actual format is pretty stable and hasn’t changed much (just the odd bug fix) The actual 7-zip application is still evolving to support more functions, offer a nicer user interface etc.

  15. Avanze says:

    I see the 7zip code is progressing nicely, I congratulate you, Haze.

  16. Firehawke says:

    Well, eta sure came in fists swinging. That’s not exactly the best way to produce a fruitful debate, and he ended up turning his attacks right back on himself here.

    Civility, people, and civilized discourse. It really does make a difference.

    Now to the topic at hand– is there any noticeable difference in loading speed between 7z and zip? Admittedly, the additional RAM consumption is a slight concern, but I imagine it shouldn’t be a long-term issue. Once the 7z is fully decompressed, the RAM consumption should be equal to zip since the additional buffer space is no longer needed. I can’t think of any ROM sets where the buffer space in RAM would be so big of an issue where the machine wouldn’t already have issues with performance, so this is a pretty marginal issue.

    The endianness of the UTF-16 shouldn’t matter here as a negative. I imagine the 7zip code already handles most of this natively, and anything within MAME itself can get a little extra wrapping to deal with it. It’s a slight pain for porting, but can be documented well enough. MAME is *never* going to have 100% clean code, so trying to keep it 100% clean is effort wasted. Keeping semi-dirty code *well-documented*, on the other hand, seems to be the way going forward.

    The file size limits of ZIP as perceived with MAME are a point I’m uncertain of. Are there really problems at the fringes now? What kinds of sizes are we talking about? What kind of sets? I feel like I need more data to have a real opinion on this one.

    I admit to curiousity about RAR as well. Ignoring the elephant in the room in regards to licensing nightmares, there are questions about RAM consumption, speed, and so forth. I imagine that even if you can’t get the licensing reconciled, you’ll probably do a private prototype build just to compare notes with. Should you do so, please do post the results.

    All in all, promising line of work you’re pursuing. I hope the MAME team will honestly sit down and take a real, honest look at this without dismissing it purely for who suggested it. Eta’s reaction implies heavily that at least one MAMEdev member is unwilling to even retain an open mind…

  17. Haze says:

    yeah there are unrar packages out there for both rar2 and rar3 formats, but anything derived from the official binaries / sources have the stipulation that they’re not used to develop competing rar software. Technically nothing that’s inherently incompatible with MAME, but still a bit of a nasty.

    7-zip with sensible settings is pretty much equal speed with zip, memory requirements are a bit higher.

    MAME has it’s own caching of zipfiles, so that it can open one zipfile and then go back to it later. At present I don’t think it actually frees that cache after loading, only on exit. When applied to 7-zip this is a bit more costly because more data is cached (dictionary size + solid block size) but as long as you’re sensible it’s not prohibitive. The memory management is something that would need bashing out in real use cases (I’m not sure the current strategy is optimal)

    For the plain 7-zip stuff (ignoring additional MAME caching overhead) it allocates memory for the dictionary size, and size of file being decompressed / solid block size. Essentially (basic explanation here..) all ‘solid block’ means is that before compressing the files it divides things internally into chunks of ‘solid block size’ then compresses those instead of the individual files. As I said, it’s not magic, which is why I’m surprised people have issues with it. I guess if a file spans more than 1 solid block then you would need the memory to decompress the additional blocks, but I imagine that depends on however it works internally. The decompressed solid blocks are cached between calls, so the first call will take a while, but subsequent calls for a file contained within the same solid block will return immediately because the data is already decompressed. If you have solid support turned off the memory requirements of the decompresser are just compressed filesize + dictionary size. Obviously if people pick STUPID values for these compression will be slow / require stupid amounts of memory, but you can’t really do much about human stupidity.

    For Happy Fish I believe the game (when correctly dumped) uses 2x2gb Flash ROMs. As a stopgap measure you *could* have a CHD-Flash format because the game does effectively treat them like a disk, but such stopgaps are only going to take you so far, and are hacks moreso than solutions.

    It’s unfortunate that in the worst-case scenario the amount of RAM needed to decompress a zip is also higher but prices of RAM are still coming down, amount of RAM is still increasing, and unlike processor speeds this still seems to be happening at a steady rate. Given the games using such large amount of flash tend to be modern things which will require 64-bit platforms capable of large memory allocations I don’t see this as a major problem going forward. 32-bit MAME is dead for systems with a substantial amount of RAM anyway, even Naomi / Chihiro GD-ROM had a gig of RAM in some cases and that’s ancient at this point.

    The other case where the limit is currently hit is with one of the ‘gamtor’ sets, whereby if you fully merge the sets you will hit the 2gb limit (not exceed it, yet, because I renamed some files to avoid it)

    While in the former case you could potentially just keep the roms unzipped (which would be best for running them because you have *no* additional memory overhead) it doesn’t really provide a real solution going forward, 7zip will. For the second case, of course you can keep them split merged instead but the fact we’re hitting the issue at all should be taken as a warning.

    Flash capacities and prices have increased at exponential rates in recent years, we’re going to see more and more platforms using high capacity flash roms, and we’re bound to see some using them in capacities where you can’t just stop-gap them with a chd solution.

    At the end of the day tho this is just support designed to make life easier for people. Nobody is saying ‘you must store all your MAME Roms as .7z’ but having support for .7z makes processing things which get sent for me to look at less hassle, and testing things you might already have as .7z for other console emulators easier.

    I’d like a .rar decompressor too, for the same reason and if this .7z goes through I probably will propose one based off the code 7-zip code, but you can see the problems and prejudice incurred in simply trying to get another *open* standard supported, nevermind one many people consider non-free.

    MAME should be prepared for the future, not playing catchup at a point where the issue does become critical (which is essentially the position it’s found itself in w/regards modern hardware emulation as well as _real_ levels of accuracy needed for consoles, both for which the framework is/was woefully unprepared for) I’ve done half the work for people here when it comes to a modern compression format, and at worst it gives an extra option for people to use if they want to. It would be a shame to see this code go to waste, rather than being included and refined. Code / projects which don’t get included tend to just die off, the only real exception seems to be Kaillera, which ironically is also quite possibly the worst thing people could keep alive.

  18. JamesW says:

    So, .7z is now effectively equivalent to .zip for archival purposes, lossless compression being considered for samples and CHDs, whatever happened to wasting space with impunity ;)

  19. Haze says:

    Alright, RB has checked in the initial code.

    At this stage I’m sure people will uncover some bugs (as I said, I need to look at the caching strategy, make sure I’m not freeing up things before I should and we should probably make sure to free everything after the initial loading is done. The handling of the UTF16 is just crude and minimal at this point too)

    If certain combinations of 7-zipped files fail then I’d be interested in hearing about it. If they’re non-memory related failures it will be interesting to track down what’s going wrong.

    Improvements are welcome.

    If the issues end up being too large during the u cycle I imagine the code will end up being disabled or removed, it’s experimental afterall so don’t count on it being there forever just yet. I’d like to hope it still gets compiled in so that LMZA can be tested with CHD outside of .7z even in the worst case that .7z support gets disabled tho :-)

  20. me says:

    cool stuff. Is it just the lzma sdk or the other compression types too?

  21. Haze says:

    the ‘7z’ part of the code (lzma sdk basically) based on the sample ‘C’ 7z decompresser they give you. The LMZA SDK code is really just a stripped down version of the 7-zip application without the extra decompression formats (rar etc.) or gui support.

    that means it supports lmza, lmza2, bzip2, ppmd compression types at least.

    for the most part you’re just going to be using lmza tho, which is the default when you use the 7zip application.

    if there do turn out to be issues with it I’ll take a look at porting their C++ decompresser instead, but given that their sample was based on the C code I’m assuming that’s their ‘reference’ code. Right now any issues are more likely to be in my code than the actual library tho, need to iron those out over the u cycle :-)

  22. etabeta says:

    just for the record, I’m not afraid to try new things, or I would not have spent weeks on supporting proper NES split cart dumps in MESS (which is the only emu to support them) nor collaborating on softlist creation, it’s just that I don’t see the advantages of 7z as big as you do, except for the 2GB limit in zlib
    I really don’t understand your oversimplifying black-or-white attitude, where either the world is with you or is against you: here we were discussing about the advantages of 7z, and I think I’m completely entitled to have my view about its pros and cons

    back to 7z: apparently the other devs did not share my concern, given I was the only one explaining what part I was skeptical about on the list, but I have absolutely no problems with the code being now in.
    I still believe it was more fruitful the addition of FLAC than this, but hopefully time will prove me wrong (apparently Aaron will take a look to allow optional LMZA use in CHD)

    finally, about invalid 7zips, just download PSF rips of PS2 music and you shall find what you’re looking for

  23. Haze says:

    you came across as very aggressively against the format when in reality it just provides an additional option.

    you came across as wanting to dismiss / write off any possibility of it due to bugs / issues you’d had with the format at an earlier point in time without any evidence that they still exist, or would affect MAME.

    basically you seemed to have written the whole thing off before even giving it a chance.

    it’s a feature which has been requested enough over the past few years, to make it worth including, at least that way it looks like MAMEdev *do* listen to user input. With support in their hands people can weigh up the pros and cons on their own, much like the APE stuff.

    I don’t see anywhere offering PSF2s as .7z ….

  24. krick says:

    While we’re talking about about archives…

    Would it be possible for MAME to support folders inside the ROM archive?

    For example, when merging a parent and all clones, it would be nice if the parent files could go in the root of the archive, and each clone’s differing files could go in a folder named after the clone set. That way, there could never be a conflict between the parent filename and the clone filename if they are the same.

    Or, to go one step further, you could have an archive at the root of your MAME folder called “roms.zip” and inside would be folders for each parent, and inside those would be folders for each clone.

    Obviously, putting an entire MAME ROM set inside a “roms.7z” would be kind of silly.

  25. Haze says:

    MAME loads by CRC if it can, so subfolders should already in theory work.. (I know some older versions had an issue where they wouldn’t load correctly like that tho, but I think it’s been fixed for a while?)

  26. me says:

    btw mame set 44gig to 33gig converting to 7z.

  27. Marcello says:

    Did you have to modify the 7z library to include it in mame, out of curiosity ?
    On Unix/Linux we like to do dynamic linking with system-wide libraries, if you modified the library it will make packaging mame in Unix/Linux difficult.

  28. Haze says:

    The only thing of note is that it compiles the non-win32 paths on Windows (which in the context of MAME is fine) the Linux/Unix compile should still be compiling the expected codepaths.

    Can’t say I really agree with dynamic linking with the system libraries tho, you’re introducing potential differences in behavior due to the host system which I don’t find desirable, even if I can see the potential security benefits (I don’t really consider MAME security critical tho, it’s an emulator, not a browser, and I’m sure a zip library would be the least of your concerns as far as security goes.)

    I don’t anticipate you’ll have any problems if that is the path you want to take tho.

  29. Haze says:

    is 44 -> 33 gig just using plain .7z compression, or with solid blocks as well?

    I noticed some test cases 125mb zips ending up around 100mb without the need for any solid compression, so 44 -> 33 sounds in the realm of realistic. (20% improvement)

  30. powtrix says:

    Here on linux, 7z with the method “-mx9 -mmt” it results:

    32G=7z
    42G=zip

  31. me says:

    I was using the same settings as powtrix with the windows ver. I think the default is solid blocks with those settings. I will give it a try the other way when I get home. Need to add the switch -s=off. There are crazier settings you can really crank it out. They may not gain you much (less than 1% in my exp) and take a lot longer to compress everything. The mx9 setting is a good balance of speed and max compression.

    I saw some going from 60 to 8 with the solid settings. I saw some of the smaller files actually increase in size.

    Saw this in the docs for solid settings as an advantage “Decreases extraction time of a group of files (or just one file), so long as the group doesn’t contain the entire archive.” So it may actually be faster to decompress with this setting on. Would have to test somehow…

  32. me says:

    Ok
    before (zlib) 44,559,821,329
    after (7z no solid) 36,662,942,534
    after (7z with solid) 33,942,406,156

    so ~2.5 gig larger with solid off.

  33. Marcello says:

    Well with 32 comments and counting, this blog is becoming a forum of its own :)

    @haze:
    > I don’t really consider MAME security critical tho, it’s an emulator, not a browser,
    I agree here, but unfortunately I don’t know any Unix/Linux distribution who accepts in its app repository statically linked binaries.

    ( for instance http://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles Convenience copies of code )

    So the ability to build using the system provided libs is quite important if we want to get Mame in the Debian/Ubuntu/whatevernix app repositories :)

  34. Haze says:

    Yeah, I’d heard about such policies.. surely forcing the use of shared libs is just replicating the old Windows ‘dll hell’ issue.

    TBH it’s one of the thing that puts me off several other things too, Cygwin is a nightmare for it, you have practically no idea if the files you’re using are the same ones the developers were using because it’s all multiple web-pulled downloads. As a result compiling a lot of things because difficult to the point of giving up.

    MAME is a nice solid unit, you know that you’re always running the same basic code as everybody else, and that the code an old version runs will be the same code it ran at the time.

    What if one of the distros started including CPU cores for legacy app compatibility (ARM distros running x86 code and such) I’d hate to think of the compatibility nightmares something like that would cause if CPU cores had to be dynamically linked as a result.

    The basic policy does make sense in security critical areas, but it’s a major user / developer inconvenience.

  35. Zanek says:

    I tried to get MAME set to compress with solid archives but every setting I tried doesn’t seem to work. What is the command line for solid archives? I have checked and when I list the 7z file, “Solid: -“, which should mean that its not a solid archive. I tried on the command line and CLRMamePro and have had not luck getting it to work.

  36. me says:

    @Zanek

    For solid use
    7z a -t7z filename.7z -mx9 -mmt *

    For non solid use
    7z a -t7z filename.7z -mx9 -mmt -ms=off *