gaps between app shared memory partitions can waste a lot of space

[andrewboie]

Describe the bug

On systems which require power-of-two size/alignment, we are not ordering the app shared memory partitions in an optimal way, which results in a lot of memory being wasted.

A good example is running the app memory size script from #14120 on the kernel produced by tests/subsys/jwt on mps2_an385 after applying #14115. Each partition is either very small or has good efficiency, but notice the huge gap in between z_libc_partition and k_mbedtls_partition:

$ python3 ~/projects/zephyr4/scripts/appmem_report.py zephyr/zephyr.elf 
Userspace memory partition report:
Partition                     Total       Used        Wasted      Efficiency
----------------------------------------------------------------------------
ztest_mem_partition           0x40        0x34        0xc         81.25%
  gap 0x0         
z_libc_partition              0x20        0x4         0x1c        12.50%
  gap 0x7fa0      
k_mbedtls_partition           0x8000      0x7fe0      0x20        99.90%

total                         0x10000     0x8018      0x7fe8      50.04%

Ordering the app shared memory partitions in reverse order of total size should greatly reduce these gaps.

We don't know the full sizes of these partitions when gen_app_partitions.py runs since the zephyr_prebuilt.elf has not been linked yet, but since we are examining the individual object files we might be able to assemble a total anyway. Or we could somehow figure out a way to get the linker to reorganize these partitions for us, but that might require them being placed in individual sections.

[andrewboie]

If we try to solve this in gen_app_partitions.py, any solution for this will need to consider the memory used not only in .o files scanned, but libraries added via the --library option to gen_app_partitions.py

[galak]

The linker has SORT_BY_ALIGNMENT which might allow us to sort the partitions at link time. Just requires that each partition ends up in its own section. Not sure what the implications or requirements in the linker script are related to that.

I know we need to keep the partitions together, that should be easy enough. @ioannisg had some additional constraints we needed to ensure.

Maybe first step is just to get each partition in its own linker section and ensuring the proper constraints.

[dcpleung]

SORT_BY_ALIGNMENT can only be applied to input sections. With app shared memory, the sorting needs to be done to across different output sections. The issue is due to ARM's MPU alignment requirements, where a 64KB region must be aligned on 64KB. So k_mbedtls_partition in the description has to be aligned at 32KB since its partition is 32KB. Hence there is a big gap between z_libc_partition and k_mbedtls_partition because of all the padding.