Backport 3.6: psa_generate_key_custom

[gilles-peskine-arm]

Migrate from psa_generate_key_ext() and psa_key_derivation_output_key_ext() to psa_generate_key_custom() and psa_key_derivation_output_key_custom():

• Define the new functions.
• Change references in the documentation to point to the new functions.
• Keep the old functions for backward compatibility with Mbed TLS 3.6.0. Document them as deprecated, but don't formally deprecate them because we don't add formal deprecations in an LTS branch.
• Stop declaring the old functions when building as C++. This is necessary to resolve Flexible array members are not standard C++ #9020.

The API now matches the current state of ARM-software/psa-api#194.

Resolves #9020.

I started with the 3.6 PR because that's the hardest part where we care about backward compatibility. In development, since we're working on a new major version, we can just remove the old function.

PR checklist

• changelog provided
• development psa_generate_key_custom #9446
• framework no
• 3.6 here
• 2.28 backport N/A
• tests provided

[tom-cosgrove-arm]

Initial pass

[valeriosetti]

Thanks for the replies and the fix.
LGTM :)

[eleuzi01]

LGTM, thanks

[tom-cosgrove-arm]

ABI-API check failure expected.

The Windows-2013-Release-x64-cmake-retarget failure is unrelated:

[2024-07-29T20:07:22.580Z] 2024-07-29 20:07:18,033 - VS2013 Release x64 Retargeted cmake - ERROR - Cloning into 'C:/builds/workspace/mbed-tls-pr-head_PR-9235-head/worktrees/tmps982v0p8/framework'...
[2024-07-29T20:07:22.580Z] fatal: early EOF
[2024-07-29T20:07:22.580Z] fatal: The remote end hung up unexpectedly
[2024-07-29T20:07:22.580Z] fatal: index-pack failed
[2024-07-29T20:07:22.580Z] error: RPC failed; curl 18 transfer closed with outstanding read data remaining
[2024-07-29T20:07:22.580Z] fatal: clone of 'https://github.com/Mbed-TLS/mbedtls-framework' into submodule path 'C:/builds/workspace/mbed-tls-pr-head_PR-9235-head/worktrees/tmps982v0p8/framework' failed
[2024-07-29T20:07:22.580Z] Failed to clone 'framework'. Retry scheduled
[2024-07-29T20:07:22.580Z] Cloning into 'C:/builds/workspace/mbed-tls-pr-head_PR-9235-head/worktrees/tmps982v0p8/framework'...
[2024-07-29T20:07:22.580Z] fatal: early EOF
[2024-07-29T20:07:22.580Z] fatal: The remote end hung up unexpectedly
[2024-07-29T20:07:22.580Z] fatal: index-pack failed
[2024-07-29T20:07:22.580Z] error: RPC failed; curl 18 transfer closed with outstanding read data remaining
[2024-07-29T20:07:22.580Z] fatal: clone of 'https://github.com/Mbed-TLS/mbedtls-framework' into submodule path 'C:/builds/workspace/mbed-tls-pr-head_PR-9235-head/worktrees/tmps982v0p8/framework' failed
[2024-07-29T20:07:22.580Z] Failed to clone 'framework' a second time, aborting
[2024-07-29T20:07:22.580Z] 
[2024-07-29T20:07:22.580Z] 2024-07-29 20:07:18,035 - VS2013 Release x64 Retargeted cmake - ERROR - Checking out worktree failed, aborting
[2024-07-29T20:07:22.580Z] Traceback (most recent call last):
[2024-07-29T20:07:22.580Z]   File "windows_testing.py", line 222, in get_clean_worktree_for_git_reference
[2024-07-29T20:07:22.580Z]     check=True
[2024-07-29T20:07:22.580Z]   File "C:\Python36\lib\subprocess.py", line 418, in run
[2024-07-29T20:07:22.580Z]     output=stdout, stderr=stderr)
[2024-07-29T20:07:22.580Z] subprocess.CalledProcessError: Command '['git', 'submodule', 'update', '--init']' returned non-zero exit status 1.

[tom-cosgrove-arm]

(minor, related to PR title) it's not really a backport, it's normal development, just against the 3.6 branch, not the development branch. There will be a (quite different!) forward port - effectively a different fix - for development

[gilles-peskine-arm]

Rebased for the all.sh breakdown. The previous version is in https://github.com/gilles-peskine-arm/mbedtls/tree/psa_generate_key_custom-3.6-2

git range-diff origin/mbedtls-3.6 4acd8c91c04e914ddc6d3efd105d873500464bdc 14b87f6318f3aa0e9b02866618cf471bf7934825

[gilles-peskine-arm]

@tom-cosgrove-arm Technically, yes, this is the original PR and the development version is a forward port. But once both PR are up, it doesn't make a difference any longer. And for gatekeeping, it's easier if the title convention is uniform.

[gilles-peskine-arm]

By the way, framework/scripts/generate_psa_wrappers.py --log stdout was already broken in 3.6.0, so fixing that is out of scope. I'll probably repair it as par of the fix to #9447.

[Tachi107]

Hi all, commit 52504f8, which got merged with this pull request, removed (renamed) psa_key_production_parameters_are_default from the ABI. I seem to understand that this was intended, but why? The symbol was exposed in a header file, so was it really "internal" as mentioned in the commit message?

This is currently holding back upgrading to the latest MbedTLS LTS release in Debian.

[gilles-peskine-arm]

@Tachi107 psa_key_production_parameters_are_default was not declared in a public header, so it was ambiguously not part of the ABI of 3.6.0.

Our build scripts are not set to hide symbols that are not exposed in public headers because the way to do that depends on the toolchain, and we don't like to maintain toolchain-dependent code. That being said, we have considered adding MBEDTLS_EXPORTED annotations to all declarations of public functions, which could help if you want to declare which symbols are to be exported.