Extend `generate-update-metadata()` to read from `/usr` #938

HuijingHei · 2025-05-29T05:53:46Z

Extend generate-update-metadata() to read from /usr

Firstly, check if existing /usr/lib/efi, then get package info
from path like usr/lib/efi/<pkg>/<version>/EFI/.
If the path /usr/lib/efi doesn’t exist, falls back to the legacy
OSTree location /usr/lib/ostree-boot/efi/EFI/, and walkthrough
all files and save them to usr/lib/efi/<pkg>/<version>/EFI/.

See #926

ci: add testing generate-update-metadata in bootc container
Thanks Colin's PR #936

HuijingHei · 2025-06-05T07:03:56Z

Ready for reviewing now.

I did testing under fedora-bootc container, and upgrade grub2-efi-x64 shim-x64 to new, remove the old /usr/lib/bootupd/updates/* and run generate-update-metadata to create metadata.

Build new version according to #926 (comment):
grub2:
https://koji.fedoraproject.org/koji/taskinfo?taskID=133336624
shim:
https://koji.fedoraproject.org/koji/taskinfo?taskID=133341841

Download and prepare the repo

# cat myrepo.repo
[myrepo]
name=My Custom Repository
baseurl=http://localhost:8080/bootloader/
enabled=1
gpgcheck=0

Build patch

# make && make install-all DESTDIR=/srv/bootupd-test/out

Run testing under fedora-bootc container

podman run --privileged -it -v ./:/opt -w /opt -v ./tmp/myrepo.repo:/etc/yum.repos.d/myrepo.repo --net host quay.io/fedora/fedora-bootc:42 bash


bash-5.2# dnf upgrade -y grub2-efi-x64 shim-x64 --repo myrepo

bash-5.2# rpm -ql grub2-efi-x64 shim-x64 | grep usr/lib/efi
/usr/lib/efi/grub2/2.12-34.fc43/EFI/fedora/grubx64.efi
/usr/lib/efi/shim/15.8-4/EFI/BOOT/BOOTX64.EFI
/usr/lib/efi/shim/15.8-4/EFI/BOOT/fbx64.efi
/usr/lib/efi/shim/15.8-4/EFI/fedora/BOOTX64.CSV
/usr/lib/efi/shim/15.8-4/EFI/fedora/mmx64.efi
/usr/lib/efi/shim/15.8-4/EFI/fedora/shim.efi
/usr/lib/efi/shim/15.8-4/EFI/fedora/shimx64.efi

(dnf reinstall -y grub2-efi-x64 shim-x64 --repo myrepo)
bash-5.2# cp -rav out/usr /
bash-5.2# rm -rf /usr/lib/bootupd/updates/*

bash-5.2# /usr/bin/bootupctl backend generate-update-metadata / -vvv
[TRACE bootupd] executing cli
[DEBUG bootupd::ostreeutil] Using dbpath "--dbpath=/usr/lib/sysimage/rpm"
Generated update layout for BIOS: grub2-tools-1:2.12-34.fc43.x86_64
[DEBUG bootupd::ostreeutil] Using dbpath "--dbpath=/usr/lib/sysimage/rpm"
Generated update layout for EFI: grub2-efi-x64-1:2.12-34.fc43.x86_64,shim-x64-15.8-4.x86_64
[DEBUG bootupd::efi] Unmounting

### check the result:
bash-5.2# cat /usr/lib/bootupd/updates/BIOS.json 
{"timestamp":"2025-05-30T03:39:14Z","version":"grub2-tools-1:2.12-34.fc43.x86_64"} 
bash-5.2# cat /usr/lib/bootupd/updates/EFI.json 
{"timestamp":"2025-05-30T06:29:19Z","version":"grub2-efi-x64-1:2.12-34.fc43.x86_64,shim-x64-15.8-4.x86_64"}

cgwalters

Overall looks sane to me, a variety of comments. I don't have anything blocking.

I think we should be testing this scenario in CI, I doubt we are? I am not up to date on things, does it require us pulling in a copr?

cgwalters · 2025-06-09T17:45:08Z

src/efi.rs

        let dest_efidir = component_updatedir(sysroot_path, self);

-        if ostreebootdir.exists() {
+        // New EFI dir /usr/lib/efi


Do you know if this this specific to the new Fedora grub/shim packages? I suspect it is...we may want to somehow make this a build time or even runtime conditional so in theory it's more pluggable/controllable by others.

I'd at least factor it out into a const somewhere that explains where it came from.

Hmm actually, this topic also strongly relates to #766 right?

Thank you so much for the reviewing!

Do you know if this this specific to the new Fedora grub/shim packages? I suspect it is...we may want to somehow make this a build time or even runtime conditional so in theory it's more pluggable/controllable by others.

Yes.

I'd at least factor it out into a const somewhere that explains where it came from.

SGTM.

Hmm actually, this topic also strongly relates to #766 right?

Actually this is related to issue #926 (comment), but we can extend it support #766 too.

cgwalters · 2025-06-09T17:51:53Z

src/efi.rs

+                    Ok(acc)
+                });
+            packagesystem::query_files(sysroot_path, all_files?.into_iter())?
+        } else if ostreebootdir.exists() {


Would it be cleaner perhaps if we checked first for ostreebootdir, and migrated it to usr/lib/efi if that doesn't exist? It should be an error if both exist.

Then we get closer to thinking of the ostreebootdir one as legacy.

cgwalters · 2025-06-09T17:53:00Z

src/efi.rs

+                        })
+                        .collect::<Vec<String>>();
+
+                    Command::new("mv")


I think we have an opportunity here to make usr/lib/efi the standard path actually if it exists, maybe we could make things as simple as the equivalent of ln -sr /usr/lib/efi /usr/lib/bootupd/updates/EFI?

EDIT: Ah I see it's not that simple based on find_all_efi_dirs.

Hmmm...but actually I like the idea of that layout, what if we tried to adopt that as the standard and migrate our current EFI layout to it? (It'd break updates for older bootupd though...without having dual layouts for a while, ug)

Hmmm...but actually I like the idea of that layout, what if we tried to adopt that as the standard and migrate our current EFI layout to it? (It'd break updates for older bootupd though...without having dual layouts for a while, ug)

Yes, you are right, that will break old bootupd. The current EFI.json is like:

# cat EFI.json | jq { "timestamp": "2025-03-27T10:27:15Z", "version": "grub2-efi-x64-1:2.12-28.fc42.x86_64,shim-x64-15.8-3.x86_64" }

We only get /usr/lib/efi/grub2/2.12-34.fc43/EFI (-> grub2-2.12-34.fc43) and /usr/lib/efi/shim/15.8-4/EFI (-> shim-15.8-4) from the path without rpmdb, does this make sense? For silverblue we install both shim-ia32 and shim-x64, it only shows once, we do not care about this if we only concerns the version.

{ "timestamp": "<now>", "version": "grub2-2.12-34.fc43,shim-15.8-4" }

When we do the update, we sync all the files under /usr/lib/bootupd/updates/EFI to /boot/efi/EFI, means we only apply once, but if there are 2 or more directories, we need to sync each EFI directory, any good suggestion for this?

cgwalters · 2025-06-09T17:57:10Z

src/efi.rs

@@ -615,6 +656,29 @@ fn find_file_recursive<P: AsRef<Path>>(dir: P, target_file: &str) -> Result<Vec<
    Ok(result)
 }

+// Find EFI dirs under usr/lib/efi
+// for exmaple: usr/lib/efi/shim/15.8-4/EFI, usr/lib/efi/grub2/2.12-34.fc42/EFI
+fn find_all_efi_dirs(sysroot_lib: &Path) -> Result<Vec<PathBuf>> {


BTW looking at this layout, I think we can now avoid invoking rpm to query the file information for these which would be a huge side benefit.

So again I think this relates to #766 in that perhaps we make this layout our new "API" for adding content in the ESP?

Actually thinking about things here...you know, it probably wouldn't be terribly hard to change what rpm-ostree does to automatically do this instead (via an opt-in). That'd require some coordination but the powerful benefit is we'd effectively automatically "backport" support for /usr/lib/efi even for older OSes which seems like it'd help us a lot here.

Want to clarify this, what we currently do in generate-update-metadata() is move files under legacy ostree /usr/lib/ostree-boot/efi/EFI to /usr/lib/bootupd/updates/EFI, then invoke rpm to query the file information (need to insert the /boot/efi/EFI to get the correct path) and create EFI.json to include the package info

What we want to change is: retrieve /usr/lib/efi and get EFI path, then add the package info from path and create EFI.json. The change might be easy.

What I am concern is for the installation and update, if there are 2 or more directories, we need to change the current logic to sync each EFI directory, instead of only /usr/lib/bootupd/updates/EFI. WDYT?

What we want to change is: retrieve /usr/lib/efi and get EFI path, then add the package info from path and create EFI.json. The change might be easy.

Yeah I think so.

What I am concern is for the installation and update, if there are 2 or more directories, we need to change the current logic to sync each EFI directory, instead of only /usr/lib/bootupd/updates/EFI. WDYT?

Yeah, I think that would make sense. However it would mean we need to bridge between the current list of files in bootupd-state.json vs the split directories.

Anyways in the short term what you're doing here (not changing the payload layout) is probably what we have to do in order to retain backwards compat (i.e. older clients can upgrade).

But after this work lands it'd probably be useful to try to start some work on making /usr/lib/efi style layout be supported; something like bootupctl backend install --format-version=2 as an opt in or so.

start some work on making /usr/lib/efi style layout be supported;

SGTM.

say-paul · 2025-06-10T08:35:17Z

src/efi.rs

+// Find EFI dirs under usr/lib/efi
+// for exmaple: usr/lib/efi/shim/15.8-4/EFI, usr/lib/efi/grub2/2.12-34.fc42/EFI
+fn find_all_efi_dirs(sysroot_lib: &Path) -> Result<Vec<PathBuf>> {
+    const LIBDIRS: &[&str] = &["grub2", "shim"];


Can we have : const LIBDIRS: &[&str] = &["grub2", "shim","."];
So that we can keep in things in /usr/lib/efi/<version>/EFI/<files> , EFI.json metadata can contain the rpm details.

Maybe can remove the limitation and scan all EFI directories like /usr/lib/efi/<name>/<version>/EFI/<files> ?

That can be one method, we would only need to find a name for putting things to esp, which is trivial.

Let's not consider files outside of a <name>/<version> directory here as we need to be able to know "where" they come from / attach they to a package/source, even if it's user specified.

is reliance on <name>/<version> to remove dependency on rpm -qf which does not work when files are copied around?

is reliance on <name>/<version> to remove dependency on rpm -qf which does not work when files are copied around?

Yes, I think it will not work. In future, will rely on that to get meta for package.

HuijingHei · 2025-06-13T08:16:55Z

Look at more about #938 (comment), to clarify:

Firstly checks /usr/lib/efi/(shim|grub)/%{version}-%{release}/EFI/ EFI path exists, then add EFI.json about the "(shim|grub)/%{version}-%{release}"
If the path /usr/lib/efi doesn’t exist, check the legacy OSTree location /usr/lib/ostree-boot/efi/EFI/, use rpm to query the files and translate them like /usr/lib/efi/(shim|grub)/%{version}-%{release}/EFI/, then add EFI.json about the "(shim|grub)/%{version}-%{release}"
For the installed /boot/efi files, use rpm to query the files and translate them to /usr/lib/efi/(firmware)/%{version}-%{release}/EFI/boot/efi/*, also append EFI.json about "(firmware)/%{version}-%{release}", like:

{
  "timestamp": "2025-03-27T10:27:59Z",
  "version": "grub2-2.12-28.fc42,shim-15.8-3,bcm2711-firmware-20250319-1.80f23d4.fc42,bcm2835-firmware-20250319-1.80f23d4.fc42,bcm283x-firmware-20250319-1.80f23d4.fc42,bcm283x-overlays-20250319-1.80f23d4.fc42"
}
bash-5.2# dnf install -y bcm2711-firmware
Package                                                   Arch           Version                                                   Repository                            Size
Installing:
 bcm2711-firmware                                         aarch64        20250319-1.80f23d4.fc42                                   fedora                            10.3 MiB
Installing dependencies:
 bcm2835-firmware                                         aarch64        20250319-1.80f23d4.fc42                                   fedora                            12.1 MiB
 bcm283x-firmware                                         aarch64        20250319-1.80f23d4.fc42                                   fedora                            73.6 KiB
 bcm283x-overlays                                         aarch64        20250319-1.80f23d4.fc42                                   fedora                             1.0 MiB
...
bash-5.2# find /boot/efi/ -type f -exec rpm -qf {} \; 2>/dev/null | sort | uniq
bcm2711-firmware-20250319-1.80f23d4.fc42.aarch64
bcm2835-firmware-20250319-1.80f23d4.fc42.aarch64
bcm283x-firmware-20250319-1.80f23d4.fc42.aarch64
bcm283x-overlays-20250319-1.80f23d4.fc42.aarch64

Consequently, need to change the install to copy each directory to the destination "/boot/efi". WDYT?

say-paul · 2025-06-13T09:50:41Z

@HuijingHei thank you for thinking this through, looks pretty exhaustive and cover most of the use-cases.
Though I would also like to point one more use cases:
Its related to the peculiarities of how rpi boot function works:
It requires the uboot binary to be in esp which means:

[root@192 ~]# rpm -ql uboot-images-armv8 | grep rpi
/usr/share/uboot/rpi_arm64
/usr/share/uboot/rpi_arm64/u-boot.bin

u-boot.bin has to moved to /boot/efi .
Suggestion:
generate-update-metadata to accept any source_location(in this case:/usr/share/uboot/rpi_arm64)
and followed by your suggest method for /boot/efi files.

HuijingHei · 2025-06-13T10:22:09Z

generate-update-metadata to accept any source_location(in this case:/usr/share/uboot/rpi_arm64)

Not sure how to point the file /usr/share/uboot/rpi_arm64/u-boot.bin precisely, or we only care about it? I see there are many u-boot.bin under different directories.

say-paul · 2025-06-13T10:53:34Z

Not sure how to point the file

I see this in cli/bootupd.rs is restricted to / for now:

 pub(crate) fn run_generate_meta(opts: GenerateOpts) -> Result<()> {
        let sysroot = opts.sysroot.as_deref().unwrap_or("/");
        if sysroot != "/" {
            anyhow::bail!("Using a non-default sysroot is not supported: {}", sysroot);
        }
        bootupd::generate_update_metadata(sysroot).context("generating metadata failed")?;
        Ok(())
     }

I see there are many u-boot.bin under different directories.

The package installs the binary for all the boards, but rpi needs it be in /boot/efi

Also want to understand what will happen if: cp /usr/share/uboot/rpi_arm64/u-boot.bin /boot/efi/u-boot.bin is run before generate-update-metadata() is executed, as rpm -qf will show "not owned by any package", will it still be moved to /usr/lib/efi/firmware?

HuijingHei · 2025-06-18T10:12:31Z

The package installs the binary for all the boards, but rpi needs it be in /boot/efi

Also want to understand what will happen if: cp /usr/share/uboot/rpi_arm64/u-boot.bin /boot/efi/u-boot.bin is run before generate-update-metadata() is executed, as rpm -qf will show "not owned by any package", will it still be moved to /usr/lib/efi/firmware?

I think probably no, as it can not find the correct path.

say-paul · 2025-06-20T17:00:49Z

@HuijingHei will it be possible to add /usr/lib/efi/(grub|shim|firmware) to the parser as per the #935

Firstly, check if existing `/usr/lib/efi`, then get package info from path like `usr/lib/efi/<pkg>/<version>/EFI/`. If the path `/usr/lib/efi` doesn’t exist, falls back to the legacy OSTree location `/usr/lib/ostree-boot/efi/EFI/`, and walkthrough all files and save them to `usr/lib/efi/<pkg>/<version>/EFI/`. See coreos#926

HuijingHei · 2025-06-23T08:01:48Z

@HuijingHei will it be possible to add /usr/lib/efi/(grub|shim|firmware) to the parser as per the #935

Yes, I think so.

Thanks Colin's PR coreos#936

HuijingHei · 2025-06-25T06:51:29Z

CI failed error:

error: boot data installation failed: installing component EFI: No such file or directory (os error 2)

[2025-06-23T09:02:20.994Z] Traceback (most recent call last):

[2025-06-23T09:02:20.994Z]   File "/run/osbuild/bin/org.osbuild.bootupd", line 82, in <module>

[2025-06-23T09:02:20.994Z]     r = main(_args, _args["options"])

[2025-06-23T09:02:20.994Z]   File "/run/osbuild/bin/org.osbuild.bootupd", line 75, in main

[2025-06-23T09:02:20.994Z]     subprocess.run(cmd, check=True)

[2025-06-23T09:02:20.994Z]     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^

[2025-06-23T09:02:20.994Z]   File "/usr/lib64/python3.13/subprocess.py", line 577, in run

[2025-06-23T09:02:20.994Z]     raise CalledProcessError(retcode, process.args,

[2025-06-23T09:02:20.995Z]                              output=stdout, stderr=stderr)

[2025-06-23T09:02:20.995Z] subprocess.CalledProcessError: Command '['chroot', '/run/osbuild/mounts/ostree/deploy/fedora-coreos/deploy/81e9a6bc0f04476229d4887ac2316146fdee69ecebf8e9fa9b1824508bfa152b.0', '/usr/bin/bootupctl', 'backend', 'install', '--device=/dev/loop0', '--with-static-configs', '/run/osbuild/mounts']' returned non-zero exit status 1.

In this PR, we copy the EFI content from ostree-boot to new /usr/lib/{pkg}/{version}/EFI, not /usr/lib/bootupd/updates, so this failed.
One option is move the EFI content to /usr/lib/bootupd/updates and make bootupd works like before, and change in later PR teach bootupd to use new dir.
Another option is teach bootupd works to adapt the new /usr/lib/{pkg}/{version}/EFI.
WDYT?

HuijingHei requested review from travier and cgwalters May 29, 2025 06:23

HuijingHei force-pushed the generate-metadata-from-usr branch 2 times, most recently from 01b09e0 to be19115 Compare May 29, 2025 07:42

HuijingHei added the do-not-merge/work-in-progress label May 29, 2025

HuijingHei force-pushed the generate-metadata-from-usr branch 5 times, most recently from 54d905f to fa2e65c Compare June 2, 2025 11:57

openshift-merge-robot added the needs-rebase label Jun 2, 2025

HuijingHei force-pushed the generate-metadata-from-usr branch from fa2e65c to 2a8961f Compare June 2, 2025 12:19

openshift-merge-robot removed the needs-rebase label Jun 2, 2025

HuijingHei force-pushed the generate-metadata-from-usr branch 3 times, most recently from ae28fbc to efa3b42 Compare June 5, 2025 06:44

HuijingHei removed the do-not-merge/work-in-progress label Jun 5, 2025

cgwalters approved these changes Jun 9, 2025

View reviewed changes

cgwalters mentioned this pull request Jun 9, 2025

support dynamically extending/adding update payloads #766

Open

say-paul reviewed Jun 10, 2025

View reviewed changes

HuijingHei force-pushed the generate-metadata-from-usr branch from 4cac136 to 7241cdf Compare June 11, 2025 14:24

say-paul mentioned this pull request Jun 13, 2025

Bootc image does not boot on rpi fedora-iot/iot-distro#82

Open

openshift-merge-robot added the needs-rebase label Jun 17, 2025

say-paul marked this pull request as draft June 20, 2025 11:38

openshift-ci bot added the do-not-merge/work-in-progress label Jun 20, 2025

say-paul marked this pull request as ready for review June 20, 2025 11:39

openshift-ci bot removed the do-not-merge/work-in-progress label Jun 20, 2025

say-paul mentioned this pull request Jun 20, 2025

extend-payload-to-esp: extends any src_input_dir to esp #935

Open

HuijingHei force-pushed the generate-metadata-from-usr branch from 7241cdf to 91a345c Compare June 23, 2025 03:47

openshift-merge-robot removed the needs-rebase label Jun 23, 2025

HuijingHei force-pushed the generate-metadata-from-usr branch from 91a345c to 6ef48a0 Compare June 23, 2025 07:52

HuijingHei force-pushed the generate-metadata-from-usr branch from 6ef48a0 to a7cf718 Compare June 23, 2025 08:27

ci: add testing generate-update-metadata in bootc container

067e12b

Thanks Colin's PR coreos#936

HuijingHei force-pushed the generate-metadata-from-usr branch from a7cf718 to 067e12b Compare June 23, 2025 08:47

Extend generate-update-metadata() to read from /usr #938

Are you sure you want to change the base?

Extend generate-update-metadata() to read from /usr #938

Uh oh!

Conversation

HuijingHei commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuijingHei commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

say-paul Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

say-paul Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuijingHei commented Jun 13, 2025

Uh oh!

say-paul commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuijingHei commented Jun 13, 2025

Uh oh!

say-paul commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuijingHei commented Jun 18, 2025

Uh oh!

say-paul commented Jun 20, 2025

Uh oh!

HuijingHei commented Jun 23, 2025

Uh oh!

HuijingHei commented Jun 25, 2025

Uh oh!

Uh oh!

Extend `generate-update-metadata()` to read from `/usr` #938

Extend `generate-update-metadata()` to read from `/usr` #938

HuijingHei commented May 29, 2025 •

edited

Loading

HuijingHei commented Jun 5, 2025 •

edited

Loading

say-paul Jun 10, 2025 •

edited

Loading

say-paul Jun 10, 2025 •

edited

Loading

say-paul commented Jun 13, 2025 •

edited

Loading

say-paul commented Jun 13, 2025 •

edited

Loading