Multithreaded access to PSA slots - stateful approach

[AndrzejKurek]

Protect access to key slot data & metadata by a global mutex.
Introduce a state machine to handle psa key slot states in a more robust way.
Split key destruction into two parts to be able to delay the destruction once no readers are active.

State graph:

                │
           ┌────▼─────┐  ┌──────────┐
      ┌────┤  EMPTY   ◄──┤DESTROYING│
      │    └─┬────┬──▲┘  └──▲──▲────┘
┌─────▼────┐ │ ┌──▼──┴────┐ │  │
│ CREATING │ │ │  WIPING  ├─┘  │
└──┬─┬─────┘ │ └──────▲───┘    │
   │ │  ┌────▼─────┐  ├────────┘
   │ └──►  UNUSED  ├──┤
   │    └──▲────┬──┘  │
   │    ┌──┴────▼──┐  │
   │  ┌─┤ READING  ├──┤
   │  │ └─▲────────┘  │
   │  └───┘           │
   └──────────────────┘ 

State transitions from:
Empty state:
-> Wiping: cleaning slots in psa_wipe_all_key_slots: psa_crypto_init.
-> Unused: importing an existing key that does not require creation: psa_load_persistent_key_into_slot, psa_load_builtin_key_into_slot.
-> Creating: psa_import_key, psa_copy_key, psa_key_derivation_output_key, psa_generate_key.

Creating state:
-> Wiping: psa_fail_key_creation.
-> Destroying: UNUSED; TODO - could/should it happen?
-> Unused: psa_finish_key_creation from psa_import_key, psa_copy_key, psa_key_derivation_output_key, psa_generate_key.

Unused state:
-> Reading: Any operation that needs to read the key material. Copying, exporting, signing, verifying, enc/dec... via psa_get_and_lock_key_slot_with_policy.
-> Wiping: psa_purge_key, psa_close_key, but also psa_get_empty_key_slot if there's an unused persistent key.
-> Destroying: psa_destroy_key.

Reading state:
-> Reading: another reader added via psa_get_and_lock_key_slot_with_policy.
-> Unused: psa_unlock_key_slot by the last reader.
-> Wiping: psa_purge_key, psa_close_key;
-> Destroying: psa_destroy_key.

Wiping state:
-> Destroying: psa_destroy_key.
-> Empty: psa_wipe_key from psa_close_key, psa_purge_key, but also delayed wiping when the last reader is unlocked in psa_unlock_key_slot. Failures in psa_get_and_lock_key_slot, psa_get_empty_key_slot.

Destroying state:
-> Empty: psa_finish_key_destruction (also calls psa_wipe_key), psa_destroy_key if there are no readers, and if there was - unlocking the last one via psa_unlock_key_slot.

Open questions:

• If a destruction is scheduled (but there are some active readers), should a second destruction call return a new error, for example PSA_ERROR_PENDING?
• Should calling psa_open_key on a key slot that has pending destruction/wiping fail?
• Is it possible to go from PSA_STATE_CREATING to PSA_STATE_DESTROYING?
• Should psa_wipe_key_slot not work if the key slots are in use? Currently, calling psa_wipe_all_key_slots will wipe all key slots once it can acquire a mutex, regardless of slot states. psa_wipe_key_slot is surrounded by protective logic everywhere apart from psa_wipe_all_key_slots.
• Should the caller be notified, that, when the key slot was unlocked, an additional delayed wiping/destruction was performed? PSA_SUCCESS_KEY_DESTROYED? PSA_SUCCESS_KEY_WIPED?
• Should psa_get_and_lock_key_slot be renamed? In fact this function gets the key slot and tries to transition it to a desired state.

Double, simultaneous key creation test cannot be added without multithreading in tests.

This PR supercedes #5084 and #5226.
Partially fixes #3263.

Status update 2023-03-29

• informally, broadly agreed on design, but have not done formal design review (significant task)
• needs significant work to add tests (probably follow-up PR to add thorough multi-threaded tests)
• interest from multiple partners and internal team

[AndrzejKurek]

Force pushed to trigger the CI.

[AndrzejKurek]

Rebased on top of current development without any conflicts.

[carlaageamundsen]

Hi
What is the plan for this PR?

[AndrzejKurek]

Hi What is the plan for this PR?

A design review was agreed upon in the nearest future, followed up by an assesment of the volume of work needed to finish this task. Any relevant information will be posted in this PR in the process to keep you updated.

[tom-daubney-arm]

We are now converting older PRs to draft PRs where the following conditions are met: They have not been updated in the last 3 months, and they need more than non-trivial work to complete.

[yanesca]

Closing in favour of https://github.com/orgs/Mbed-TLS/projects/1#column-19657300