add Ubuntu Impish Indri

[vicamo]

While there is no ubuntu core image built for Impish yet, this should be a draft before fully verified.

[tianon]

It's open for development now, so maybe builds are coming soon. 🤓

[muktupavels]

Some Impish images are now available - https://partner-images.canonical.com/core/impish/20210512/.

[tianon]

In the past, the lack of a current folder symlink has been due to build issues -- @philroche that's probably something you're already aware of (and possibly even folks are looking into), but just in case, here's an explicit ping. 😄

[tianon]

per @mwhudson's prodding, I started looking at what might be a minimal change to start consuming from the brand-new https://partner-images.canonical.com/oci/ for impish (and stole this commit): https://github.com/tianon/docker-brew-ubuntu-core/compare/oci-tarballs?w=1

[tianon]

(One major blind spot I see so far is that we don't have something like unpacked/build-info.txt to tell us which serial current corresponds to 😄)

[muktupavels]

Now there is also current folder for Impish.

[vicamo]

Tested all architectures locally. Works for me.

[toabctl]

@tianon ubuntu wants to switch to use the tarballs from https://partner-images.canonical.com/oci/ . I'll do 2 more changes before we can use that:

• the filenames will get a prefix ubuntu-. So https://partner-images.canonical.com/oci/impish/current/impish-oci-amd64-root.tar.gz will be https://partner-images.canonical.com/oci/impish/current/ubuntu-impish-oci-amd64-root.tar.gz
• i'll add the unpacked/build-info.txt similar the file in core so the mechanism to get the serial can be reused.

Btw. the ubuntu-oci code will be backported/SRUed (see https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1926732) so the tarballs will be available soon for older releases, too.

[toabctl]

@tianon ubuntu wants to switch to use the tarballs from https://partner-images.canonical.com/oci/ . I'll do 2 more changes before we can use that:

• the filenames will get a prefix ubuntu-. So https://partner-images.canonical.com/oci/impish/current/impish-oci-amd64-root.tar.gz will be https://partner-images.canonical.com/oci/impish/current/ubuntu-impish-oci-amd64-root.tar.gz
• i'll add the unpacked/build-info.txt similar the file in core so the mechanism to get the serial can be reused.

Both changes are now implemented and https://partner-images.canonical.com/oci/impish/current/ has them

@tianon is there anything else we need for switching import to the oci tarballs?

[vicamo]

@tianon ubuntu wants to switch to use the tarballs from https://partner-images.canonical.com/oci/ . I'll do 2 more changes before we can use that:

• the filenames will get a prefix ubuntu-. So https://partner-images.canonical.com/oci/impish/current/impish-oci-amd64-root.tar.gz will be https://partner-images.canonical.com/oci/impish/current/ubuntu-impish-oci-amd64-root.tar.gz
• i'll add the unpacked/build-info.txt similar the file in core so the mechanism to get the serial can be reused.

Both changes are now implemented and https://partner-images.canonical.com/oci/impish/current/ has them

@tianon is there anything else we need for switching import to the oci tarballs?

current solution is available for all supported architectures, will oci images support them as well?

[toabctl]

@tianon ubuntu wants to switch to use the tarballs from https://partner-images.canonical.com/oci/ . I'll do 2 more changes before we can use that:

• the filenames will get a prefix ubuntu-. So https://partner-images.canonical.com/oci/impish/current/impish-oci-amd64-root.tar.gz will be https://partner-images.canonical.com/oci/impish/current/ubuntu-impish-oci-amd64-root.tar.gz
• i'll add the unpacked/build-info.txt similar the file in core so the mechanism to get the serial can be reused.

Both changes are now implemented and https://partner-images.canonical.com/oci/impish/current/ has them
@tianon is there anything else we need for switching import to the oci tarballs?

current solution is available for all supported architectures, will oci images support them as well?

yes. in fact the only real change is moving the modificaitons currently done in the Dockerfile to the build process of the OCI tarball.

[tianon]

current solution is available for all supported architectures, will oci images support them as well?

yes. in fact the only real change is moving the modificaitons currently done in the Dockerfile to the build process of the OCI tarball.

Are you saying that impish only supports amd64 + arm64, or that more tarballs are "coming soon" in that OCI folder?

@tianon is there anything else we need for switching import to the oci tarballs?

I'd love to see some kind of response to https://code.launchpad.net/~toabctl/livecd-rootfs/+git/livecd-rootfs-1/+merge/401596/comments/1059820:

So, the files we currently add to the Ubuntu images are really terse because they're added in a Dockerfile -- in the Debian image, we're creating much more verbose files such that each of these comes with a pretty long-winded explanation of why they exist and what problem they're solving: https://github.com/debuerreotype/debuerreotype/blob/0.12/scripts/debuerreotype-minimizing-config

$ docker run --rm debian:buster-slim wc -l /etc/apt/apt.conf.d/docker-autoremove-suggests
16 /etc/apt/apt.conf.d/docker-autoremove-suggests
$ docker run --rm debian:buster-slim grep -vE '^$|^#' /etc/apt/apt.conf.d/docker-autoremove-suggests | wc -l
1

That script also does some rudimentary detection for whether to bother with /sbin/initctl based on whether upstart even exists in our target distribution.

I'd love to see these scripts incorporate more of that (and even better if we could collaborate on those in debuerreotype instead of duplicating the effort so that both distros can benefit from any improvements).

(I've played with running "debuerreotype-minimizing-config" directly in that Ubuntu image build, and I was able to get it working pretty well overall with one minor tweak to debuerreotype replacing debuerreotype-chroot with standard chroot; see #119 (comment) for where I did that experimentation.)

Just reproducing the current brain-dead implementation in tarball form in a separate script that's even less transparent to users of the image feels like two steps backwards IMO. 🤔

Directly related to that, what can we do to improve the discoverability of how these tarballs get created? I'm pretty well steeped in this process, and even I have a hard time finding the scripts that build them when I have a question or want to consider contributing to them (which is frankly something I wouldn't even feel comfortable attempting, given how strange Launchpad MRs are and not having any idea how to test my changes).

[toabctl]

current solution is available for all supported architectures, will oci images support them as well?

yes. in fact the only real change is moving the modificaitons currently done in the Dockerfile to the build process of the OCI tarball.

Are you saying that impish only supports amd64 + arm64, or that more tarballs are "coming soon" in that OCI folder?

More tarballs will be there soon (in the next days). There will be the same archs that we have for the core tarballs.

@tianon is there anything else we need for switching import to the oci tarballs?

I'd love to see some kind of response to https://code.launchpad.net/~toabctl/livecd-rootfs/+git/livecd-rootfs-1/+merge/401596/comments/1059820:

I'll comment here now.

So, the files we currently add to the Ubuntu images are really terse because they're added in a Dockerfile -- in the Debian image, we're creating much more verbose files such that each of these comes with a pretty long-winded explanation of why they exist and what problem they're solving: https://github.com/debuerreotype/debuerreotype/blob/0.12/scripts/debuerreotype-minimizing-config

$ docker run --rm debian:buster-slim wc -l /etc/apt/apt.conf.d/docker-autoremove-suggests
16 /etc/apt/apt.conf.d/docker-autoremove-suggests
$ docker run --rm debian:buster-slim grep -vE '^$|^#' /etc/apt/apt.conf.d/docker-autoremove-suggests | wc -l
1

That script also does some rudimentary detection for whether to bother with /sbin/initctl based on whether upstart even exists in our target distribution.
I'd love to see these scripts incorporate more of that

Same here and I would be happy to include these changes into livecd-rootfs.

(and even better if we could collaborate on those in debuerreotype instead of duplicating the effort so that both distros can benefit from any improvements).

I wonder how that could look like. One of the goals moving the code from the Dockerfile to livecd-rootfs is to not rely on "external" processes so Ubuntu can release OCI images when needed to different registries.
Currently Ubuntu does create the core tarball, then waits for reviews, waits for the publish of "official" dockerhub images and then use these images to distribute to other registries (eg. AWS pub registry).
What we want is releasing on dockerhub & other registries in parallel (so no longer relying on the image on dockerhub).

(I've played with running "debuerreotype-minimizing-config" directly in that Ubuntu image build, and I was able to get it working pretty well overall with one minor tweak to debuerreotype replacing debuerreotype-chroot with standard chroot; see #119 (comment) for where I did that experimentation.)

Just reproducing the current brain-dead implementation in tarball form in a separate script that's even less transparent to users of the image feels like two steps backwards IMO.

I agree that the whole process how the OCI images are created is complicated.

Directly related to that, what can we do to improve the discoverability of how these tarballs get created? I'm pretty well steeped in this process, and even I have a hard time finding the scripts that build them when I have a question or want to consider contributing to them (which is frankly something I wouldn't even feel comfortable attempting, given how strange Launchpad MRs are and not having any idea how to test my changes).

I also had a very hard time to understand how the whole pipeline for creating a OCI image is.
Maybe adding a post to https://discourse.ubuntu.com/ and describing the process how the images are produced would help? And also how things can be tested?
In the end it's:

• create OCI (same for core, just different url) tarball with livefs-rootfs in launchpad: https://launchpad.net/~cloud-images-release-managers/+livefs/ubuntu/impish/ubuntu-oci
• test/release the tarballs (that's in an internal jenkins instance at Canonical)

If you want to change something, do a MP for livecd-rootfs (https://launchpad.net/ubuntu/+source/livecd-rootfs). I think that's the usual workflow for changing a Ubuntu .deb package
For local testing, there is a tool that can build livefs: https://github.com/ubuntu-bartenders/ubuntu-old-fashioned/ . The scripts here are helpful as a start: https://github.com/ubuntu-bartenders/ubuntu-old-fashioned/tree/master/scripts/ubuntu-bartender

[toabctl]

Are you saying that impish only supports amd64 + arm64, or that more tarballs are "coming soon" in that OCI folder?

@tianon all archs are now there: https://partner-images.canonical.com/oci/impish/current/

[tianon]

In the end it's:

• create OCI (same for core, just different url) tarball with livefs-rootfs in launchpad: https://launchpad.net/~cloud-images-release-managers/+livefs/ubuntu/impish/ubuntu-oci
• test/release the tarballs (that's in an internal jenkins instance at Canonical)

If you want to change something, do a MP for livecd-rootfs (https://launchpad.net/ubuntu/+source/livecd-rootfs). I think that's the usual workflow for changing a Ubuntu .deb package
For local testing, there is a tool that can build livefs: https://github.com/ubuntu-bartenders/ubuntu-old-fashioned/ . The scripts here are helpful as a start: https://github.com/ubuntu-bartenders/ubuntu-old-fashioned/tree/master/scripts/ubuntu-bartender

This is useful data -- this is the first I've ever seen https://discourse.ubuntu.com, so I'm not sure that's super discoverable for folks who aren't already deep enough to figure out where things are. I was thinking something more like some extra detail/links on https://hub.docker.com/_/ubuntu (which is updated via https://github.com/docker-library/docs/blob/master/ubuntu/content.md).

See for example the "How It's Made" section of https://hub.docker.com/_/debian.

all archs are now there: https://partner-images.canonical.com/oci/impish/current/

Nice! I've updated https://github.com/tianon/docker-brew-ubuntu-core/compare/oci-tarballs?w=1 with this list 👍

(Which I suppose I should probably push to this PR now 😅)

[tianon]

With that latest push, an example generated impish/Dockerfile now looks like this:

FROM scratch
ADD ubuntu-impish-oci-amd64-root.tar.gz /
CMD ["bash"]

[tianon]

@toabctl I assume you're ready to pull the trigger here and start publishing impish from these new OCI tarballs? or is there more you wanted to do to/with them first?

[mwhudson]

This all looks great to me. I agree https://github.com/docker-library/docs/blob/master/ubuntu/content.md could do with an update, @toabctl do you want to work on that or should I start something?

[toabctl]

@toabctl I assume you're ready to pull the trigger here and start publishing impish from these new OCI tarballs? or is there more you wanted to do to/with them first?

Nothing left from my side. I'm fine with publishing now the impish images. And there is nothing left I want to do.

[toabctl]

This all looks great to me. I agree https://github.com/docker-library/docs/blob/master/ubuntu/content.md could do with an update, @toabctl do you want to work on that or should I start something?

@mwhudson I also agree with updating content.md . I have a public holiday today, so feel free to start something if you want. If not, I'll have a look tomorrow.

[toabctl]

Update for content.md is docker-library/docs#1967 .