[Improvement] Use createdvd instead of createcd when compressing games for certain platforms. And other CHD improvements.

[monyarm]

PSP: hrydgard/ppsspp#18798

PS2 and other DVD systems would also be a good idea. As there are from what I've read not just compression, but also performance improvements from using createdvd for DVDs. But PSP would be the priority.

The best solution in my opinion would be to be able to set settings for which platforms to use createcd/createdvd/createhdd for, and also specify commandline arguments for chdman.

And speaking of chdman, another improvement that would be great would be an option to recompress games (useful when there's an update to chdman, to be able to quickly recompress the games for a platform)

[alucryd]

Thanks for reporting this, had no idea createcd vs createdvd would have performance implications. I can probably automate and use createcd for cue/bin, and createdvd for iso as I believe Redump only uses cue/bin for CDs and iso for everything else.

As for extra arguments, sure I can start implementing something, maybe a global setting for starters, and have the ability to override it per system later.

Recompressing was on my todo, I can move that up the list. Basically I was thinking of adding a -f/--force flag to convert-roms to do just that for any format.

[alucryd]

First part done: 273c86d

CUE/BIN will now be compressed using createcd, and ISO using createdvd. Extracting will first check the file metadata to determine whether extractcd or extractdvd should be used. Will pave the way for the third part.

[alucryd]

feac827

This takes care of the third part, added a -r flag (-f was already used for format, duh). Tested against my PS2 library that was initially compressed using createcd, ISOs are DVDs now.

[alucryd]

@monyarm What kind of additional arguments were you thinking about? hunksize, compression, maybe numprocessors too?

[TylerVigario]

@alucryd Ultimately, this is the wrong place to address this, but it would be good information. Do you know if we need to compress from the original dumps, or can we convert?

[monyarm]

@monyarm What kind of additional arguments were you thinking about? hunksize, compression, maybe numprocessors too?

Ya those should do. Maybe also a parent option, since some emulators do support chds with parents.
And numprocessors should have an auto option, where it detects the right amount.

[monyarm]

@alucryd Ultimately, this is the wrong place to address this, but it would be good information. Do you know if we need to compress from the original dumps, or can we convert?

I theory you can give chdman the chd and have it recompress it. But in practice that doesn't seem to always work.

[TylerVigario]

@monyarm @alucryd I figured that converting directly from CHD to CHD (createcd -> createdvd) wouldn't work, but is it possible to recover the original format (CHD -> ISO, etc) and then compress it to CHD automatically?

EDIT: My position is only wanting archival formats (reversible), which CHD is (sort-of). So, in that position, I should be able to recover my collection from this discovery. If this change lands (which it has/will), then as users, we should have a seamless path forward, right?

[alucryd]

@monyarm What kind of additional arguments were you thinking about? hunksize, compression, maybe numprocessors too?

Ya those should do. Maybe also a parent option, since some emulators do support chds with parents.
And numprocessors should have an auto option, where it detects the right amount.

Will add global settings for hunksize and compression, like I did for RVz. Yeah numprocessors is only needed when you want to limit them, but there's not really a reason to do it.

[alucryd]

@monyarm @alucryd I figured that converting directly from CHD to CHD (createcd -> createdvd) wouldn't work, but is it possible to recover the original format (CHD -> ISO, etc) and then compress it to CHD automatically?

You can do it directly, that's actually what I do behind the scene when you specify recompress: CHD -> ISO -> CHD

[TylerVigario]

@alucryd Thank you very much for clarifying! I figured you would have it covered, but I got apprehensive. Sorry for my unnecessary banter.

EDIT: Time to recompress my collection.

[alucryd]

@monyarm Forgot to address the parent point, this has been sitting on my to-do list for the longest time, ideally this should be automated, I can already group games together to generate playlists, so it's not a huge stretch to apply that to CHD parents.
I've been hesitant because last I checked libchdr didn't support the feature, and since most emulators rely on that lib, it wouldn't have done much good to support it in oxyromon. I should revisit it though, it's been quite a while, libchdr probably supports it now.

Edit: Looks like libchdr supports reading parent/child CHDs now. I'll add it higher up on the todo, this will have several implications I need to be mindful of though, like never to convert a parent back to another format before the children or they'd be essentially lost. Will need to introduce this relationship at the database level.

[alucryd]

Added settings for chdman: 9c28b5f

Separated CD and DVD settings, compression algorithms are different, and chdman defaults to different hunk sizes for both. I wouldn't recommend messing with the hunk sizes, but compression algorithms will allow you to use zstd which isn't enabled by default. I just use zstd+flac for CHDs now.

[alucryd]

I need to make an additional modification to the config subcommand, because right now if you set a hunk size, you're stuck with having it set manually, I'll probably a -u flag for unset that will, as it says, unset a config option.

[alucryd]

Small teaser, CHD parents support is almost there, nice size gains on discs 2 and 3 ;)

❯ oxyromon convert-roms -f CHD -c -d -r -s 'playstation' -n 'final fantasy vii'
Processing "Sony - PlayStation"
⠁                                                                                                                                                                                                                                         Please select games: Final Fantasy VII (USA) (Disc 1), Final Fantasy VII (USA) (Disc 2), Final Fantasy VII (USA) (Disc 3)
Creating "Final Fantasy VII (USA) (Disc 1).chd"
Checking "/home/public/Emulation/Sony - PlayStation/1G1R/Final Fantasy VII (USA) (Disc 1).chd"
Extracting "Final Fantasy VII (USA) (Disc 1).chd"
Checking "/tmp/.tmp2YP1zc/Final Fantasy VII (USA) (Disc 1).bin"
Before: 712.81 MiB (100.0%); After: 462.97 MiB (64.9%); Original: 712.81 MiB
Deleting "/home/public/Emulation/Sony - PlayStation/1G1R/Final Fantasy VII (USA) (Disc 1).bin"
Creating "Final Fantasy VII (USA) (Disc 2).chd"
Checking "/home/public/Emulation/Sony - PlayStation/1G1R/Final Fantasy VII (USA) (Disc 2).chd"
Extracting "Final Fantasy VII (USA) (Disc 2).chd"
Checking "/tmp/.tmp6rMFRc/Final Fantasy VII (USA) (Disc 2).bin"
Before: 698.72 MiB (100.0%); After: 248.48 MiB (35.6%); Original: 698.72 MiB
Deleting "/home/public/Emulation/Sony - PlayStation/1G1R/Final Fantasy VII (USA) (Disc 2).bin"
Creating "Final Fantasy VII (USA) (Disc 3).chd"
Checking "/home/public/Emulation/Sony - PlayStation/1G1R/Final Fantasy VII (USA) (Disc 3).chd"
Extracting "Final Fantasy VII (USA) (Disc 3).chd"
Checking "/tmp/.tmpYMV0VT/Final Fantasy VII (USA) (Disc 3).bin"
Before: 629.01 MiB (100.0%); After: 175.72 MiB (27.9%); Original: 629.01 MiB
Deleting "/home/public/Emulation/Sony - PlayStation/1G1R/Final Fantasy VII (USA) (Disc 3).bin"

Still need some things ironed out, like preventing any modification to the parent while it has children, but it should be up soon.

Edit: Pretty disappointing results with the later Final Fantasy games though, not much shared data between discs I guess.

[monyarm]

Nice, maybe also have it print what it's parent is when it's converting it?
Also, how does it decide what to parent things to?

[alucryd]

Yeah will add a log for the parent. Right now it builds upon the generate-playlists subcommand, and running it is a requirement. It will generate m3u files for every complete multi-disc game, and convert-roms will use that information. The first game of every playlist will be selected as the parent.

[alucryd]

There it is: e0d6019

One noteworthy caveat, parents will always be skipped when convert-roms is asked to recompress CHDs since modifying the parent means losing its children. Still need to add one other safeguard to prevent converting a parent to something else if it still has children. Should not happen when you try to convert all discs at the same time as parents are always converted last, but it could happen if you try to specifically convert the parent by itself using -n.

Mind that the feature is not a huge revolution, only very few games did lose weight in my experience (sometimes even down to around 1% of the original size!), granted I don;t have a huge library, but still. You might want to tinker with the hunk size for CDs, it's quite high by default (19584), I expect going down to something like 4896 (twice the unit size) would help (haven't tried yet). DVDs default to 4096 (which is twice the unit size) and seems to fare better overall.

[monyarm]

Question, would it be possible to specify custom parent child connections somewhere? In order to be able to have variants of games (like how pokemon games have different versions, I know some disc based games have something similar), sequels, hacks and translations be able to be reduced in size.

[alucryd]

Sure, I can come up with something, probably an additional flag to convert-roms that will let you search for/select a parent.

[alucryd]

Well that's unexpected, reducing the hunk size for CDs actually hurts compression (at least for FF7).

Creating "Final Fantasy VII (USA) (Disc 1).chd"
Before: 712.81 MiB (100.0%); After: 473.55 MiB (66.4%); Original: 712.81 MiB
Deleting "/home/public/Emulation/Sony - PlayStation/1G1R/Final Fantasy VII (USA) (Disc 1).bin"
Creating "Final Fantasy VII (USA) (Disc 2).chd"
Using parent "Final Fantasy VII (USA) (Disc 1).chd"
Before: 698.72 MiB (100.0%); After: 252.81 MiB (36.2%); Original: 698.72 MiB
Deleting "/home/public/Emulation/Sony - PlayStation/1G1R/Final Fantasy VII (USA) (Disc 2).bin"
Creating "Final Fantasy VII (USA) (Disc 3).chd"
Using parent "Final Fantasy VII (USA) (Disc 1).chd"
Before: 629.01 MiB (100.0%); After: 180.02 MiB (28.6%); Original: 629.01 MiB
Deleting "/home/public/Emulation/Sony - PlayStation/1G1R/Final Fantasy VII (USA) (Disc 3).bin"

Same story with other games, I recommend leaving hunk sizes to their default values, at least for CDs.

[alucryd]

Added in efb31e8

The new -p flag will prompt you to select a parent for each file, so it's best to use it together with -n. It will let you select an existing CHD that does not have any parent itself, pressing escape will convert normally without a parent.

I thought it would work well for different languages of the same game, but my initial testing is also disappointing, here's symphony of the night, there's barely any difference when using the USA release as parent of the Japanese release (that wouldn't work in practice for me anyway as my Japanese games are not stored in the same directory as my USA games, which is a requirement for parent/children to work).

Please select a system: Castlevania - Symphony of the Night (USA).chd
Extracting "Akumajou Dracula X - Gekka no Yasoukyoku (Japan) (Rev 2).chd"
Copying to "/tmp/.tmpxR5Jfy/Akumajou Dracula X - Gekka no Yasoukyoku (Japan) (Rev 2).cue"
Creating "/tmp/.tmpxR5Jfy/Akumajou Dracula X - Gekka no Yasoukyoku (Japan) (Rev 2) (Track 1).bin"
Creating "/tmp/.tmpxR5Jfy/Akumajou Dracula X - Gekka no Yasoukyoku (Japan) (Rev 2) (Track 2).bin"
Deleting "/tmp/.tmpxR5Jfy/Akumajou Dracula X - Gekka no Yasoukyoku (Japan) (Rev 2).bin"
Creating "Akumajou Dracula X - Gekka no Yasoukyoku (Japan) (Rev 2).chd"
Using parent "Castlevania - Symphony of the Night (USA).chd"
Before: 354.14 MiB (63.0%); After: 354.11 MiB (63.0%); Original: 561.85 MiB
Deleting "/home/public/Emulation/Sony - PlayStation/Akumajou Dracula X - Gekka no Yasoukyoku (Japan) (Rev 2).chd"
Moving to "/home/public/Emulation/Sony - PlayStation/Akumajou Dracula X - Gekka no Yasoukyoku (Japan) (Rev 2).chd"
Copying to "/home/public/Emulation/Sony - PlayStation/Akumajou Dracula X - Gekka no Yasoukyoku (Japan) (Rev 2).chd"
Deleting "/tmp/.tmpxR5Jfy/Akumajou Dracula X - Gekka no Yasoukyoku (Japan) (Rev 2).chd"

[alucryd]

And with that I think I tackled everything, please let me know how it goes, the beta docker image is up for quick testing if you'd like: https://hub.docker.com/r/alucryd/oxyromon

[TylerVigario]

@alucryd I believe I've found an issue

oxyromon convert-roms --format 7Z --recompress --diff --check --system "Nintendo - Game Boy"

It also specifies "Nintendo - Game Boy Advance," leaving no way to select only Game Boy.

[alucryd]

@alucryd I believe I've found an issue

oxyromon convert-roms --format 7Z --recompress --diff --check --system "Nintendo - Game Boy"

It also specifies "Nintendo - Game Boy Advance," leaving no way to select only Game Boy.

That is expected, it's not an exact search, but I can make an exact one if you want to select a single system, maybe a -ss flag if clap let's me do double letter flags.

[alucryd]

@TylerVigario Better yet, not too keen on introducing yet another flag, but I can give more power to the user using the existing flag. Behind that is nothing more than a SQL like (case insensitive), right now I automatically surround whatever is passed with %, but I can leave that up to you.

[TylerVigario]

@TylerVigario Better yet, not too keen on introducing yet another flag, but I can give more power to the user using the existing flag. Behind that is nothing more than a SQL like (case insensitive), right now I automatically surround whatever is passed with %, but I can leave that up to you.

I prefer that. I initially thought it would only do an exact match, but I am good if we have the option.

[alucryd]

Done: f2167b9

Applied that across the board to all commands using -s, also applied it to the game filters and I renamed the flag to -g for game instead of -n for name, that's more explicit.
Also renamed the conflicting -g 1g1r flag to -o for one-g-one-r, although I'm still not really satisfied with the letter xD

[TylerVigario]

Also renamed the conflicting -g 1g1r flag to -o for one-g-one-r, although I'm still not really satisfied with the letter xD

Those are always the most challenging decisions 😝

[monyarm]

Small teaser, CHD parents support is almost there, nice size gains on discs 2 and 3 ;)

I'm having an issue with CHD parent support

❯ oxyromon convert-roms -f CHD -d -s "Sony - PlayStation 2" -g "D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 2) (DX Pack)" -p
Processing "Sony - PlayStation 2"
⠁                                                                                                                                                                                                                                           Please select games: D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 2) (DX Pack)
Please select a ROM file: D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 1) (DX Pack).chd
Creating "D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 2) (DX Pack).chd"
Using parent "D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 1) (DX Pack).chd"
Error: SimpleError { err: "Hunk size 19584 bytes is not a whole multiple of 2048\nFatal error occurred: 1\n" }

It seems to be using the CD hunk size, despite the games being DVD ISOs

[alucryd]

@monyarm I don't see any obvious issue in the associated code path. Acquiring the games so I can reproduce and debug.

[alucryd]

@monyarm Couldn't reproduce, with or without an explicit CD hunk size

~/Emulation/Sony - PlayStation 2 
❯ oxyromon config -s CHD_CD_HUNK_SIZE 19584

~/Emulation/Sony - PlayStation 2 
❯ oxyromon convert-roms -d -f CHD -s '%playstation 2%' -g '%da capo%' -p -r
Processing "Sony - PlayStation 2"
⠁                                                                                                                                                                                     
Please select games: D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 2) (DX Pack)
Please select a ROM file: D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 1) (DX Pack).chd
Extracting "D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 2) (DX Pack).chd"
Using parent "D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 1) (DX Pack).chd"
Creating "D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 2) (DX Pack).chd"
Using parent "D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 1) (DX Pack).chd"
Before: 865.16 MiB (22.6%); After: 865.16 MiB (22.6%); Original: 3.73 GiB
Deleting "/home/public/Emulation/Sony - PlayStation 2/D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 2) (DX Pack).chd"
Moving to "/home/public/Emulation/Sony - PlayStation 2/D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 2) (DX Pack).chd"
Copying to "/home/public/Emulation/Sony - PlayStation 2/D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 2) (DX Pack).chd"
Deleting "/tmp/.tmppXIZMf/D.C.II P.S. - Da Capo II - Plus Situation (Japan) (Disc 2) (DX Pack).chd"

I tried converting from ISO, and reconverting from CHD, both worked. Could you verify oxyromon config -l and make sure you didn't accidentally set CHD_DVD_HUNK_SIZE to 19584?

[alucryd]

I'll close this one as I couldn't reproduce. Please feel free to open a dedicated issue if you still experience it.