restructure image builds #1913

deitch · 2021-02-17T09:47:24Z

Does two things:

adds LABELs to the dockerfile lfedge/eve indicating where the rootfs and initrd are, so they can be found later
instead of copying installer/ in toto, copy each file/dir as desired

The first part is important for allowing anything to look at the lfedge/eve container and, just by inspecting its config, discover where the artifacts are. It lays more groundwork for baseos in OCI images.

The second part helps keep containers clean, and is a longstanding issue. make for various kinds of images dumps everything - with a filename that reflects the build hash - in dist/<arch>/installer/.

The image we build includes the following components. Here is an example from looking at the tree on my test amd64 env:

~/go/src/github.com/lf-edge/eve$ tree dist/amd64/
dist/amd64/
├── Dockerfile
├── OVMF.fd
├── OVMF_CODE.fd
├── OVMF_VARS.fd
├── build.yml
├── installer
│   ├── EFI
│   │   └── BOOT
│   │       ├── BOOT.pc
│   │       ├── BOOTX64.EFI
│   │       └── grub.cfg
│   ├── boot
│   │   └── u-boot.bin
│   ├── config.img
│   ├── initrd.img
│   ├── persist.img
│   ├── rootfs-0.0.0-master-a8c5dddf-xen-amd64.squash
│   ├── rootfs-0.0.0-volumemgr-all-contenttree-021dbb95-kvm-amd64.squash
│   ├── rootfs-0.0.0-volumemgr-all-contenttree-68a5970b-kvm-amd64.squash
│   ├── rootfs-0.0.0-remove-hash-check-bd3809aa-kvm-amd64.squash
│   ├── rootfs-0.0.0-remove-hash-check-bad19db4-kvm-amd64.squash
│   ├── rootfs-xen.img -> rootfs-0.0.0-master-a8c5dddf-xen-amd64.squash
│   └── rootfs.img -> rootfs-xen.img
├── live.qcow2
├── live.raw
├── rootfs-kvm.yml
├── rootfs-xen.yml
└── runme.sh

This is a first step to cleaning it up, by explicitly adding to the image only those things we need:

OVMF files
EFI
boot
config.img
persist.img
initrd.img
rootfs.img

In the future, we might want to think about having a per-build installer, so that it would be even cleaner. It would look something like this:

~/go/src/github.com/lf-edge/eve$ tree dist/amd64/
dist/amd64/
├── Dockerfile
├── build.yml
├── installer
  ├── master-a8c5dddf-xen-amd64
      ├── EFI
          └── BOOT
            ├── BOOT.pc
            ├── BOOTX64.EFI
            └── grub.cfg
      ├── boot
           └── u-boot.bin
      ├── firmware
            ├── OVMF.fd
            ├── OVMF_CODE.fd
            └── OVMF_VARS.fd
      ├── config.img
      ├── initrd.img
      ├── persist.img
      ├── runme.sh
      └── rootfs.img
  ├── volumemgr-all-contenttree-021dbb95-kvm-amd64
      ├── EFI
          └── BOOT
            ├── BOOT.pc
            ├── BOOTX64.EFI
            └── grub.cfg
      ├── boot
           └── u-boot.bin
      ├── firmware
            ├── OVMF.fd
            ├── OVMF_CODE.fd
            └── OVMF_VARS.fd
      ├── config.img
      ├── initrd.img
      ├── runme.sh
      └── rootfs.img
├── rootfs-kvm.yml
└── rootfs-xen.yml

Is this duplicative? Sure it is. But it duplicates everything in the git-ignored dist/ tree, and shows very clearly everything you build that gets installed. It also has the benefit of sending docker context only the necessary files, not every single image that is there, making the builds slower as time goes on.

If people think this is better day one, I am happy to rework this PR to arrange the above.

eriknordmark

This looks reasonable to me.

rvs

Hey @deitch - we have scripts that expect a rootfs-.squash to be there. Please restore that (but now as a symlink). I think the scripts will be happy with that (not sure if that ever gets extracted -- but we can deal with it later).

deitch · 2021-02-18T04:59:02Z

That's fine, we can do that.

But which scripts? I hunted down dependencies before proposing this, and didn't find them. Obviously I missed them, but I would like to know which ones. I will add comments into the PR as well.

rvs · 2021-02-18T05:22:16Z

But which scripts?

Test scripts -- some of them in Eden -- some of them you don't get to see

deitch · 2021-02-18T05:25:05Z

Also, no comments on the proposed changed installer structure?

deitch · 2021-02-18T05:26:21Z

What exactly do they depend upon? Just having a file named *.squash? Can they just look for the final file?

rvs · 2021-02-18T05:27:41Z

Also, no comments on the proposed changed installer structure?

I think we need to discuss it in a more 1-1 fashion

rvs · 2021-02-18T05:28:52Z

What exactly do they depend upon? Just having a file named *.squash? Can they just look for the final file?

They depend on the version being embedded in the filename -- hence the exact rootfs-.squash link needs to be restored.

deitch · 2021-02-18T05:44:25Z

We can do that, but again, that's not a great pattern. Don't we have the version on the image tag, or at least the image metadata?

deitch · 2021-02-18T07:30:55Z

I think we need to discuss it in a more 1-1 fashion

Actually, we might need to discuss it now. See this:

we have scripts that expect a rootfs-.squash to be there. Please restore that (but now as a symlink).

It isn't quite that simple. The old one just copied everything over as:

COPY installer /bits

This lets the dockerfile not care what the actual filename is, which can change (it includes the branch name and the hash and hypervisor and arch, etc.).

That also, unfortunately, is what caused it to install every build ever made that still lingers in the installer/ directory.

I cannot think of a sane way to pass the name of the targeted build file to the docker build. Theoretically, we could use docker build args, but linuxkit doesn't support them for the same reason that docker originally didn't: reproducible builds. If you can change the arg, you can change the build, but nothing on the filesystem is changed, so it isn't reproducible.

We can debate the merits of that policy - I certainly have, and see both sides of it - but it isn't getting changed between now and tomorrow.

Given the above, I think my proposal to have a per-build installer directory actually may be the easiest solution here. I would just modify it slightly to fit your requirements:

├── Dockerfile
├── build.yml
├── installer
  ├── master-a8c5dddf-xen-amd64
      ├── EFI
          └── BOOT
            ├── BOOT.pc
            ├── BOOTX64.EFI
            └── grub.cfg
      ├── boot
           └── u-boot.bin
      ├── firmware
            ├── OVMF.fd
            ├── OVMF_CODE.fd
            └── OVMF_VARS.fd
      ├── config.img
      ├── initrd.img
      ├── persist.img
      ├── runme.sh
      ├── rootfs-master-a8c5dddf-xen-amd64.squash -> rootfs.img
      └── rootfs.img
  ├── volumemgr-all-contenttree-021dbb95-kvm-amd64
      ├── EFI
          └── BOOT
            ├── BOOT.pc
            ├── BOOTX64.EFI
            └── grub.cfg
      ├── boot
           └── u-boot.bin
      ├── firmware
            ├── OVMF.fd
            ├── OVMF_CODE.fd
            └── OVMF_VARS.fd
      ├── config.img
      ├── initrd.img
      ├── runme.sh
      ├── rootfs-volumemgr-all-contenttree-021dbb95-kvm-amd64.squash -> rootfs.img
      └── rootfs.img
├── rootfs-kvm.yml
└── rootfs-xen.yml

deitch · 2021-02-18T07:32:00Z

The above makes it really easy for the dockerfile, as it goes back to installing just the needed bits, but it installs them by blindly taking from a build-specific installer directory. Makefile, which already knows all of the correct filenames and such, can set it up fairly easily.

What do you both say? Any reason not to make the change that way?

rvs · 2021-02-19T05:11:29Z

Yup -- I like the latest proposal @deitch -- can you please push relevant commits to this PR?

deitch · 2021-02-19T10:30:00Z

The installer directory has been completely restructured. I still am working on making sure that make run, make installer and make live work, but make eve is in good shape.

@rvs I think you know this the best, so can you review? Specifically, I want to make sure we didn't miss any interesting use cases, e.g. certain HV like kvm or adam combinations.

deitch · 2021-02-19T10:32:27Z

Particularly, this does simplify quite a bit in the dist/ directory. It is instantly clear all of the parts of a build, as they are segregated by directory.

deitch · 2021-02-19T10:42:11Z

OK, I think I have those now too. Good for someone to take a look.

deitch · 2021-02-19T10:57:34Z

To make the changes clear, here is a similar tree structure. Ran make eve and make live and make installer.

dist/
└── amd64
    ├── 0.0.0-dockerfile-labels-fd9533a2
        ├── Dockerfile
        ├── build.yml
        ├── installer
        │   ├── EFI
        │   │   └── BOOT
        │   │       ├── BOOT.pc
        │   │       ├── BOOTX64.EFI
        │   │       └── grub.cfg
        │   ├── boot
        │   │   └── u-boot.bin
        │   ├── config.img
        │   ├── firmware
        │   │   ├── OVMF.fd
        │   │   ├── OVMF_CODE.fd
        │   │   └── OVMF_VARS.fd
        │   ├── initrd.img
        │   ├── persist.img
        │   ├── rootfs-0.0.0-dockerfile-labels-fd9533a2-xen-amd64.squash -> rootfs.img
        │   └── rootfs.img
        ├── live.qcow2
        ├── live.raw
        ├── rootfs-xen.yml
        └── runme.sh
    └── 0.0.0-dockerfile-labels-fd9533a2-dirty-2021-02-19.10.42
        ├── Dockerfile
        ├── build.yml
        ├── installer
        │   ├── EFI
        │   │   └── BOOT
        │   │       ├── BOOT.pc
        │   │       ├── BOOTX64.EFI
        │   │       └── grub.cfg
        │   ├── boot
        │   │   └── u-boot.bin
        │   ├── config.img
        │   ├── firmware
        │   │   ├── OVMF.fd
        │   │   ├── OVMF_CODE.fd
        │   │   └── OVMF_VARS.fd
        │   ├── initrd.img
        │   ├── persist.img
        │   ├── rootfs-0.0.0-dockerfile-labels-fd9533a2-dirty-2021-02-19.10.42-xen-amd64.squash -> rootfs.img
        │   └── rootfs.img
        ├── live.qcow2
        ├── live.raw
        ├── rootfs-xen.yml
        └── runme.sh

rvs

LGTM

rvs · 2021-02-19T21:24:02Z

Hey @deitch -- this looks much better now. Still, the following target seems to break:

make HV=kvm 
make HV=kvm ACCEL=y run-installer-raw

rvs

Hey @deitch -- this looks much better now, but while testing it I uncovered two problems:

If you repository is considered dirty by git then the version will change every time you run build or run targets which means that a sequence of the following commands will fail make live && make run
It seems that a versioned rootfs file is gone from installer folder and I'd like to have it around for now

I think for both of these -- it is just a simple matter of an extra symlink (as we've agree I think). One to be put under dist/<ARCH> (to always point at the latest version so that run commands can use that) and the other one under dist/<ARCH>/<VERSION/installer/rootfs-<VERSION>.squash

I do realize that the later is a bit redundant -- but lets keep it for compatibility reasons for now -- we can drop it early in 6.x cycle.

Once these two changes are in (plus a tiny nit I spotted above) we can merge this

pkg/eve/Dockerfile.in

deitch · 2021-02-21T08:33:07Z

If you repository is considered dirty by git then the version will change every time you run build or run targets which means that a sequence of the following commands will fail make live && make run

I saw that, but it should be an issue with the existing mainline as well (pre-this-PR), because it uses the ROOTFS_VERSION in the filename. But I will run it and check again.

deitch · 2021-02-21T08:34:08Z

Oh I see, make run relied on the live.qcow2 file. Symlinks will fix all of this. Easy enough.

Signed-off-by: Avi Deitcher <[email protected]>

deitch · 2021-02-21T09:12:03Z

OK, this looks good for that first issue now @rvs . Every time you do a build - make live, make eve, make rootfs - it creates a symlink current -> <build_dir> that is used by make run.

deitch · 2021-02-21T09:13:04Z

It seems that a versioned rootfs file is gone from installer folder and I'd like to have it around for now

I don't know what you mean by this. I just ran make live (same with make eve) and I get this:

dist/
└── amd64
    ├── 0.0.0-dockerfile-labels-dfc20df2
    │   ├── Dockerfile
    │   ├── build.yml
    │   ├── installer
    │   │   ├── EFI
    │   │   │   └── BOOT
    │   │   │       ├── BOOT.pc
    │   │   │       ├── BOOTX64.EFI
    │   │   │       └── grub.cfg
    │   │   ├── boot
    │   │   │   └── u-boot.bin
    │   │   ├── config.img
    │   │   ├── firmware
    │   │   │   ├── OVMF.fd
    │   │   │   ├── OVMF_CODE.fd
    │   │   │   └── OVMF_VARS.fd
    │   │   ├── initrd.img
    │   │   ├── persist.img
    │   │   ├── rootfs-0.0.0-dockerfile-labels-dfc20df2-xen-amd64.squash -> rootfs.img
    │   │   └── rootfs.img
    │   ├── live.qcow2
    │   ├── live.raw
    │   ├── rootfs-xen.yml
    │   └── runme.sh
    └── current -> /home/ubuntu/go/src/github.com/lf-edge/eve/dist/amd64/0.0.0-dockerfile-labels-dfc20df2

That symlink is there.

rvs

LGTM. Thanks for taking care of this so quickly @deitch

deitch · 2021-02-21T19:18:55Z

Definitely! Happy to see this moving ahead.

deitch requested review from eriknordmark and rvs as code owners February 17, 2021 09:47

eriknordmark approved these changes Feb 17, 2021

View reviewed changes

rvs requested changes Feb 18, 2021

View reviewed changes

deitch force-pushed the dockerfile-labels branch from c1bda30 to 500cfa1 Compare February 19, 2021 10:28

deitch changed the title ~~copy only specific bits and add label~~ restructure image builds Feb 19, 2021

deitch force-pushed the dockerfile-labels branch from 500cfa1 to 0ffbf19 Compare February 19, 2021 10:31

deitch force-pushed the dockerfile-labels branch from 0ffbf19 to fd9533a Compare February 19, 2021 10:41

rvs approved these changes Feb 19, 2021

View reviewed changes

rvs closed this Feb 19, 2021

rvs reopened this Feb 19, 2021

rvs requested changes Feb 19, 2021

View reviewed changes

pkg/eve/Dockerfile.in Outdated Show resolved Hide resolved

deitch force-pushed the dockerfile-labels branch from fd9533a to 94ca2f4 Compare February 21, 2021 08:31

deitch force-pushed the dockerfile-labels branch from 94ca2f4 to 48792d3 Compare February 21, 2021 08:46

copy only specific bits and add label

dfc20df

Signed-off-by: Avi Deitcher <[email protected]>

deitch force-pushed the dockerfile-labels branch from 48792d3 to dfc20df Compare February 21, 2021 08:46

rvs approved these changes Feb 21, 2021

View reviewed changes

rvs merged commit bccc421 into lf-edge:master Feb 21, 2021

deitch deleted the dockerfile-labels branch February 22, 2021 07:44

deitch mentioned this pull request Mar 1, 2021

Need test for baseos image upgrade from within OCI image lf-edge/eden#539

Open

giggsoff mentioned this pull request Mar 2, 2021

[WIP] fix build and run for dirty source tree #1952

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

restructure image builds #1913

restructure image builds #1913

deitch commented Feb 17, 2021

eriknordmark left a comment

rvs left a comment

deitch commented Feb 18, 2021

rvs commented Feb 18, 2021

deitch commented Feb 18, 2021

deitch commented Feb 18, 2021

rvs commented Feb 18, 2021

rvs commented Feb 18, 2021

deitch commented Feb 18, 2021

deitch commented Feb 18, 2021

deitch commented Feb 18, 2021

rvs commented Feb 19, 2021

deitch commented Feb 19, 2021

deitch commented Feb 19, 2021

deitch commented Feb 19, 2021

deitch commented Feb 19, 2021

rvs left a comment

rvs commented Feb 19, 2021

rvs left a comment

deitch commented Feb 21, 2021

deitch commented Feb 21, 2021

deitch commented Feb 21, 2021

deitch commented Feb 21, 2021

rvs left a comment

deitch commented Feb 21, 2021

restructure image builds #1913

restructure image builds #1913

Conversation

deitch commented Feb 17, 2021

eriknordmark left a comment

Choose a reason for hiding this comment

rvs left a comment

Choose a reason for hiding this comment

deitch commented Feb 18, 2021

rvs commented Feb 18, 2021

deitch commented Feb 18, 2021

deitch commented Feb 18, 2021

rvs commented Feb 18, 2021

rvs commented Feb 18, 2021

deitch commented Feb 18, 2021

deitch commented Feb 18, 2021

deitch commented Feb 18, 2021

rvs commented Feb 19, 2021

deitch commented Feb 19, 2021

deitch commented Feb 19, 2021

deitch commented Feb 19, 2021

deitch commented Feb 19, 2021

rvs left a comment

Choose a reason for hiding this comment

rvs commented Feb 19, 2021

rvs left a comment

Choose a reason for hiding this comment

deitch commented Feb 21, 2021

deitch commented Feb 21, 2021

deitch commented Feb 21, 2021

deitch commented Feb 21, 2021

rvs left a comment

Choose a reason for hiding this comment

deitch commented Feb 21, 2021