how to transcode h264 into h265 without losing quality?

[allenchou217]

i am using this docker on my unraid to transcode my h264 files to h265 to reduce their size,
here is the preset i made,
PresetName": "h265",
"SubtitleAddCC": false,
"SubtitleAddForeignAudioSearch": true,
"SubtitleAddForeignAudioSubtitle": false,
"SubtitleBurnBDSub": true,
"SubtitleBurnBehavior": "foreign",
"SubtitleBurnDVDSub": true,
"SubtitleLanguageList": [],
"SubtitleTrackSelectionBehavior": "none",
"Type": 1,
"UsesPictureFilters": true,
"VideoAvgBitrate": 6000,
"VideoColorMatrixCodeOverride": 0,
"VideoEncoder": "nvenc_h265",
"VideoFramerate": "auto",
"VideoFramerateMode": "vfr",
"VideoGrayScale": false,
"VideoHWDecode": 0,
"VideoLevel": "auto",
"VideoMultiPass": true,
"VideoOptionExtra": "",
"VideoPreset": "fast",
"VideoProfile": "auto",
"VideoQSVDecode": false,
"VideoQualitySlider": 33.0,
"VideoQualityType": 2,
"VideoScaler": "swscale",
"VideoTune": "",
"VideoTurboMultiPass": true,
"x264Option": "",
"x264UseAdvancedOptions": false
}

is there any better options that i can use to transcode them only reducing the size without reducing the quality?
my files are in different resolution, different bitrate,
i hope there is one preset that could be used for all video

[zocker-160]

This is not an easy task, given that any kind of decode - encode process does cause some kind of quality loss unless very high bitrates are used.

How much compression you can achieve also depends on the quality of the source file. If for example the source video is low quality, the compression will not be effective, because you are at some point starting to compress the compression artifacts and noise causing a larger file size.

For the best compression / quality ratio, I would suggest you to try to use a constant quality (CQ) setting (try values around 18) instead of a constant bitrate one together with the encoder preset set to at least slow. (you seem to use fast which is quite bad)

This setting should work for any kind of video, but ofc it can mean that on some videos the resulting file is larger than the source, in which case converting to h265 might be pointless.

One last thing: generally using x265 gives a better compression / quality ratio than using NVEnc and using newer NVEnc hardware (e.g. RTX 4000 series) will also give better results than using older ones like 1000 or 2000 series GPUs.

[yayaasd]

this is my preset, happy to share with you. But honestly I am not the HD/Blueray/et al. guy. Happy to have a nice quality on min. Filesize - so please adjust as you like...

And for sure, may someone also had a better idea about my version, always up for suggestions ;-)

{
    "PresetList": [
        {
            "AlignAVStart": false,
            "AudioCopyMask": [
                "copy:aac"
            ],
            "AudioEncoderFallback": "fdk_aac",
            "AudioLanguageList": [],
            "AudioList": [
                {
                    "AudioBitrate": 160,
                    "AudioCompressionLevel": -1.0,
                    "AudioDitherMethod": "auto",
                    "AudioEncoder": "fdk_aac",
                    "AudioMixdown": "stereo",
                    "AudioNormalizeMixLevel": false,
                    "AudioSamplerate": "auto",
                    "AudioTrackDRCSlider": 0.0,
                    "AudioTrackGainSlider": 0.0,
                    "AudioTrackQuality": 1.0,
                    "AudioTrackQualityEnable": false
                }
            ],
            "AudioSecondaryEncoderMode": true,
            "AudioTrackSelectionBehavior": "first",
            "ChapterMarkers": true,
            "ChildrenArray": [],
            "Default": true,
            "FileFormat": "av_mkv",
            "Folder": false,
            "FolderOpen": false,
            "InlineParameterSets": false,
            "MetadataPassthrough": true,
            "Mp4iPodCompatible": false,
            "Optimize": false,
            "PictureAllowUpscaling": false,
            "PictureAutoCrop": true,
            "PictureBottomCrop": 0,
            "PictureChromaSmoothCustom": "",
            "PictureChromaSmoothPreset": "off",
            "PictureChromaSmoothTune": "none",
            "PictureColorspaceCustom": "",
            "PictureColorspacePreset": "off",
            "PictureCombDetectCustom": "",
            "PictureCombDetectPreset": "off",
            "PictureCropMode": 0,
            "PictureDARWidth": 713,
            "PictureDeblockCustom": "strength=strong:thresh=20:blocksize=8",
            "PictureDeblockPreset": "off",
            "PictureDeblockTune": "medium",
            "PictureDeinterlaceCustom": "",
            "PictureDeinterlaceFilter": "off",
            "PictureDeinterlacePreset": "",
            "PictureDenoiseCustom": "",
            "PictureDenoiseFilter": "off",
            "PictureDenoisePreset": "",
            "PictureDenoiseTune": "none",
            "PictureDetelecine": "off",
            "PictureDetelecineCustom": "",
            "PictureForceHeight": 0,
            "PictureForceWidth": 0,
            "PictureHeight": 480,
            "PictureItuPAR": false,
            "PictureKeepRatio": true,
            "PictureLeftCrop": 0,
            "PictureModulus": 2,
            "PicturePAR": "auto",
            "PicturePARHeight": 2513,
            "PicturePARWidth": 2560,
            "PicturePadBottom": 0,
            "PicturePadColor": "black",
            "PicturePadLeft": 0,
            "PicturePadMode": "none",
            "PicturePadRight": 0,
            "PicturePadTop": 0,
            "PictureRightCrop": 0,
            "PictureRotate": "angle=0:hflip=0",
            "PictureSharpenCustom": "",
            "PictureSharpenFilter": "off",
            "PictureSharpenPreset": "",
            "PictureSharpenTune": "",
            "PictureTopCrop": 2,
            "PictureUseMaximumSize": false,
            "PictureWidth": 720,
            "PresetDescription": "H.265 video (up to 480p30) and AAC stereo audio, in an MKV container.",
            "PresetDisabled": false,
            "PresetName": "h265_10bit_700x400",
            "SubtitleAddCC": false,
            "SubtitleAddForeignAudioSearch": true,
            "SubtitleAddForeignAudioSubtitle": false,
            "SubtitleBurnBDSub": true,
            "SubtitleBurnBehavior": "foreign",
            "SubtitleBurnDVDSub": true,
            "SubtitleLanguageList": [],
            "SubtitleTrackSelectionBehavior": "none",
            "Type": 1,
            "UsesPictureFilters": true,
            "VideoAvgBitrate": 1050,
            "VideoColorMatrixCodeOverride": 0,
            "VideoEncoder": "nvenc_h265_10bit",
            "VideoFramerate": "29.97",
            "VideoFramerateMode": "pfr",
            "VideoGrayScale": false,
            "VideoHWDecode": 0,
            "VideoLevel": "5.1",
            "VideoMultiPass": true,
            "VideoOptionExtra": "",
            "VideoPreset": "slowest",
            "VideoProfile": "main10",
            "VideoQSVDecode": false,
            "VideoQualitySlider": 7.0,
            "VideoQualityType": 1,
            "VideoScaler": "swscale",
            "VideoTune": "",
            "VideoTurboMultiPass": false,
            "x264Option": "",
            "x264UseAdvancedOptions": false
        }
    ],
    "VersionMajor": 56,
    "VersionMicro": 0,
    "VersionMinor": 0

[zocker-160]

May I ask you what resolution your source files are and what your hardware is?

I can create a preset for you, if you would like to give it a try.

[yayaasd]

so my hardware is GeForce RTX 3050 and i am using h265_nvenc. Having good results with my preset so far.
my input files are from 480p - 1080p --> completely mixed...

[zocker-160]

Any reason why you are using 10bit? Are the source files 10bit?

[yayaasd]

I realized better results in color and dark areas when using 10bit encoding.
was mostly try&error and see where the best result was showing up.

input files are mixed, sometimes 10bit, sometimes h264/h265, really mixed

[zocker-160]

Understood.
So far the preset seems fine to me, the only part that I find a bit weird is that you set the bitrate to a fixed 1050 kbit, which seems very low and is probably not going to work for all types of inputs.

For 1080p for example, 1050 kbit is simply not enough to produce any kind of useful quality in my opinion.

As I mentioned above, I personally would rather opt to using the constant quality setting set to something around 18.

[allenchou217]

my files vary from 360p to 4k, mostly is 1080p h264 mp4, but the bitrate is various, from 3000k to 10000k
my gpu is rtx3080laptop, my cpu is i5-10600K