crypto.generateKeyPair generates error ERR_OSSL_CRYPTO_MALLOC_FAILURE in node 17.2.0

[Raynos]

Version

17.2.0

Platform

Linux raynos-Precision-5530 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

crypto

What steps will reproduce the bug?

Run the following script in node 17.2.0 and node 16.

const crypto = require('crypto')
crypto.generateKeyPair('rsa', {
        modulusLength: 4096,
        publicKeyEncoding: {
          type: 'spki',
          format: 'pem'
        },
        privateKeyEncoding: {
          type: 'pkcs8',
          format: 'pem',
          cipher: 'aes-256-cbc',
          passphrase: ''
        }
      }, (err, publicKey, privateKey) => {
        console.log('err', err)

        // cb(null, { publicKey, privateKey })
      })

Node 16 has no error, node 17 has an err

The err is

[Error: error:078C0100:common libcrypto routines::malloc failure] {
  opensslErrorStack: [ 'error:078C0100:common libcrypto routines::malloc failure' ],
  library: 'common libcrypto routines',
  reason: 'malloc failure',
  code: 'ERR_OSSL_CRYPTO_MALLOC_FAILURE'
}

How often does it reproduce? Is there a required condition?

every time on linux and mac.

What is the expected behavior?

I expected node 17 to not have a regression

What do you see instead?

I see an error.

[Error: error:078C0100:common libcrypto routines::malloc failure] {
  opensslErrorStack: [ 'error:078C0100:common libcrypto routines::malloc failure' ],
  library: 'common libcrypto routines',
  reason: 'malloc failure',
  code: 'ERR_OSSL_CRYPTO_MALLOC_FAILURE'
}

Additional information

No response

[panva]

Not sure if it's intended or not, but the error only occurs when the private key encryption passphrase is empty, when one is provided this works as expected, likewise when no encryption is requested it's fine.

cc @nodejs/crypto

[Raynos]

I'm using this in my test suite, setting passphrase to a string like 'hello' does indeed allow me to run my test suite on both node 16 and node 17.

[RaisinTen]

I think this is a bug in OpenSSL, so I sent a report with my findings: openssl/openssl#17506

[RaisinTen]

Also, cc @tniessen since you had worked on #35914

[RaisinTen]

My fix landed in the OpenSSL repo: openssl/openssl#17507! Will cherry-pick it to https://github.com/quictls/openssl tomorrow.

[mhdawson]

@RaisinTen was this cherry picked over?

[RaisinTen]

@mhdawson I had sent the cherry-pick PR to the quictls repo - quictls/openssl#75 but I'm yet to get a review from the maintainers.

[mhdawson]

Thanks for the update, was just wondering if it was progressing.,

[RaisinTen]

Sent a cherry-pick pr to Node.js anyways because my change has already landed in the openssl repo - #42319