OCI authentication support

[stgraber]

Currently our interactions with OCI registries doesn't support authentication.

We should add support for this, at minimum supporting HTTP basic auth encoded type credentials in the OCI registry URL. Possibly also supporting auth helpers on the client side, though that part gets a bit weird as it will prevent refreshes on the server side.

[cory-chang]

Hi @stgraber! @pi2chen and I are interested in working on this issue as part of Professor Chidambaram's Virtualization class. Could you assign this issue to us?

Looking forward to working in this repo, thanks!

[stgraber]

Done. Same as the other two, I'll add @pi2chen once he comments.

This is another issue that's not marked as Easy as we haven't quite determined how far we should take this right now.

I had a chat about some of that with @xnox and others before and in the Docker world, there is the concept of a credential helper than can be hooked into the registry interaction process, basically allowing for a temporary token of some kind to be retrieved and used to interact with a private registry.

We may be able to use something like this too, basically extending the config in ~/.config/incus/config.yml to indicate that the registry requires authentication and then have it use the helper to get that token.

This should be sufficient for the CLI as it will be able to use basically the same logic as Docker at that point. It would be more of a problem for other clients, like our web UI, where there's no access to such an helper, but that will be a problem for another day :)

@xnox any chance you can give us a few more details on the common way this is handled, and if possible at all, provide us either with a private registry we can test against (including credentials) or some pointers on how to setup a local private registry with the most common authentication mechanism so we can validate on that instead.

[xnox]

a cred helper just needs to emit json for ServerURL Username Secret, it can be a shell script that echos that.

the most trivial private registry is like k3d registy create.

To put it behind auth, one needs to front that with nginx password auth.

So a basic task could be:

1. install k3d
2. start k3d localhost registry with k3d registry create
3. install nginx
4. configure nginx, to have https self signed / snakeoil cert (or letsencrypt cert)
5. add basic username / password auth in nginx, and reverse proxy on localhost to said k3d registry
6. create "docker-credential-helper-myprivateregistry" that echos json of the requested server url, and that static username and password
7. configure docker to use that credhelper shell script for your private registry
8. test that with docker push/pull to that private registry works

Note, that this is a bit insecure, as there is no mechanism to enforce ACL => meaning if people can find that exposed registry, they can push gigabytes of data to it, and instantly use it as a dropbox to share files.

An alternative way, is to use authenticated access to dockerhub => meaning login to docker hub, and use creds to access it. The best way to test that it works, is to start off with invalid username/password, which docker pull should prevent from using. Then you know invalid auth is blocking access. If that gets rejected via incus too; swap in valid credentials => and it should then just work correctly.

@stgraber is this what you were looking for?

[xnox]

for webui

• provide input field for a token
• ask users to run the credhelper locally to get the json
• copy-paste json, and use it

Basically a slightly user-assisted api call; cause regular incus client would just pass the json to the daemon to unpack and use in the calls, until expiry.

[xnox]

Tokens can be valid for like 1h, 6h, or be long lived - for example a year. Thus when client sends a token, the daemon can store it, and keep on using. Once it stops working, create errors/messages asking the client to provide a fresh one (as in renewed) token.

[stgraber]

for webui

* provide input field for a token

* ask users to run the credhelper locally to get the json

* copy-paste json, and use it

Basically a slightly user-assisted api call; cause regular incus client would just pass the json to the daemon to unpack and use in the calls, until expiry.

Yeah, that's what I was thinking. We'd just offer the auth token field and ask that they submit it that way.

[stgraber]

Okay. I've gotten myself a private repository on my Google Workspace account.

The way this works is that I'm provided with a docker-credential-gcloud binary which supports a get argument and reads the domain from stdin (in my case northamerica-northeast1-docker.pkg.dev).

This then outputs a JSON with two fields:

• Username
• Secret

Internally, we're using skopeo for the image retrieval, so that's what will need to be able to consume those fields. We don't want to have the secrets be passed as arguments as that would allow anyone on the system to see them. So instead we need to jump through a few hoops.

So the result is that we need to:

• Extend the syntax in ~/.config/incus/config.yml to allow specifying a credentials_helper. That's in shared/cliconfig/remote.go
• Make it possible to set this through incus remote add with a new --credentials-helper option. That's in cmd/incus/remote.go
• Use the credentials_helper option in GetImageServer when in the OCI case. That's still in shared/cliconfig/remote.go. What should be done here is combine the existing remote.Addr with the Username and Secret values you get by running the credentials_helper, encoding it as the username and password field in the URL. So if the registry is https://docker.io and credentials helper gives Username=foo Secret=bar, this becomes https://foo:bar@docker.io. The net.url package can help do this cleanly.
• Have our invocation of skopeo inspect and skopeo copyhandle those credentials. This is inclient/oci_images.go`.

For that last point, here is a quick proof of concept I did here which works for me:

diff --git a/client/oci_images.go b/client/oci_images.go
index 72745a4512..d39f46d8f5 100644
--- a/client/oci_images.go
+++ b/client/oci_images.go
@@ -3,6 +3,7 @@ package incus
 import (
        "compress/gzip"
        "context"
+       "encoding/base64"
        "encoding/json"
        "fmt"
        "io"
@@ -360,6 +361,38 @@ func (r *ProtocolOCI) GetImageAlias(name string) (*api.ImageAliasesEntry, string
                }
        }
 
+       // Parse the URL.
+       uri, err := url.Parse(r.httpHost)
+       if err != nil {
+               return nil, "", err
+       }
+
+       creds, err := json.Marshal(map[string]any{"auths": map[string]any{fmt.Sprintf("%s://%s", uri.Scheme, uri.Host): map[string]string{"auth": base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s", uri.User.String())))}}})
+       if err != nil {
+               return nil, "", err
+       }
+
+       uri.Scheme = "docker"
+       uri.User = nil
+
+       authFile, err := os.CreateTemp("", "incus_client_auth_")
+       if err != nil {
+               return nil, "", err
+       }
+
+       defer authFile.Close()
+       defer os.Remove(authFile.Name())
+
+       err = authFile.Chmod(0o600)
+       if err != nil {
+               return nil, "", err
+       }
+
+       _, err = fmt.Fprintf(authFile, "%s", creds)
+       if err != nil {
+               return nil, "", err
+       }
+
        // Get the image information from skopeo.
        stdout, _, err := subprocess.RunCommandSplit(
                context.TODO(),
@@ -367,7 +400,8 @@ func (r *ProtocolOCI) GetImageAlias(name string) (*api.ImageAliasesEntry, string
                nil,
                "skopeo",
                "inspect",
-               fmt.Sprintf("%s/%s", strings.ReplaceAll(r.httpHost, "https://", "docker://"), name))
+               "--authfile", authFile.Name(),
+               fmt.Sprintf("%s/%s", uri.String(), name))
        if err != nil {
                logger.Debug("Error getting image alias", logger.Ctx{"name": name, "stdout": stdout, "stderr": err})
                return nil, "", err

Obviously this isn't clean code. We need to pull out the logic to run skopeo into a function we can share for both the skopeo inspect and skopeo copy and we'll need to only do the authFile logic if uri.User != nil.

With all that done, we'll have support for credentials helpers in the command line without having to do any API level change within Incus as we'll just use our existing ability to pass through HTTP authentication credentials through the URLs. Then when it comes time to support that in the UI, the UI can take the credentials JSON, parse it and include the values in the URL it sends to the server, easy enough to do (out of scope for this issue though, we only care about CLI here).

[stgraber]

To test this, you can:

• Install the Google SDK, this will give you the gcloud and docker-credential-gcloud commands
• Create a creds.json file with the content of https://dl.stgraber.org/creds-incus-1700.json
• Run gcloud auth login --cred-file=login.json
• Confirm that the login worked by running echo https://northamerica-northeast1-docker.pkg.dev | docker-credential-gcloud get

If that gets you one of those credentials JSON with a Username and Secret field, then you're all set to work on and test this.

Adding the registry should then be done by using the URL https://northamerica-northeast1-docker.pkg.dev/stgraber-1525358518329/test-registry. It contains a single image, alpine:latest.

With this feature fully implemented, I'd expect to be able to do:

incus remote add oci-stgraber https://northamerica-northeast1-docker.pkg.dev/stgraber-1525358518329/test-registry --credentials-helper=docker-credential-gcloud
incus image info oci-stgraber:alpine:latest
incus launch oci-stgraber:alpine:latest foo

[pi2chen]

Thank you @stgraber! Feel free to assign me to this issue.

Looking forward to working on it!