feat: Support docker image manifest v2 schema1 scanning

[jopemachine]

Overview

This PR adds support for docker image manifest V2 schema1 for legacy system.
(Fix https://github.com/lablup/giftbox/issues/719)

About docker image manifest V2 schema1 refer to below link:
https://github.com/openshift/docker-distribution/blob/master/docs/spec/manifest-v2-1.md

Test

Manually tested it by following the steps below.

1. Create a private docker registry for testing.

❯ docker run -d -p 5000:5000 --restart always --name registry registry:2

The registry returns a content type of application/vnd.docker.distribution.manifest.v1+prettyjws in the response in the /manifests API.

2. Create a fixture file for testing this registry and apply.

{
    "container_registries": [
        {
            "id": "abc42a05-4471-41fa-8772-10bf6452c7d9",
            "registry_name": "localhost",
            "url": "http://localhost:5000",
            "type": "docker",
            "project": "stable"
        }
    ]
}

❯ ./backend.ai mgr fixture populate ./fixtures/manager/example-container-registries-dockerhub.json

3. Tag and push an image

Let's assume there is a locally built image named stable/bai-python here.

❯ docker tag stable/bai-python:3.11 localhost:5000/stable/bai-python:3.11
❯ docker push localhost:5000/stable/bai-python:3.11

4. Rescan images

Scan the added container registry.

❯ ./backend.ai mgr image rescan localhost
2024-09-06 03:54:58.965 INFO ai.backend.manager.models.image [164836] Scanning kernel images from the registry "localhost"
2024-09-06 03:54:58.968 INFO ai.backend.manager.container_registry.base [164836] rescan_single_registry()
2024-09-06 03:54:58.971 INFO ai.backend.manager.container_registry.base [164836] _scan_image()
2024-09-06 03:54:58.974 WARNING ai.backend.manager.container_registry.base [164836] Docker image manifest v1 is deprecated.
2024-09-06 03:54:58.977 INFO ai.backend.manager.container_registry.base [164836] Scanned image - stable/bai-python:3.11/x86_64 (sha256:3a7829f911d5c601af3273f21df405b289acb51c35d90ac661a845f3a4a4a9f6)
2024-09-06 03:54:58.990 INFO ai.backend.manager.container_registry.base [164836] Updated image - localhost/stable/bai-python:3.11/x86_64 (sha256:3a7829f911d5c601af3273f21df405b289acb51c35d90ac661a845f3a4a4a9f6)

Check that the image has been correctly updated in the database with the following command.

❯ ./backend.ai mgr dbshell
2024-09-06 04:38:30.349 INFO ai.backend.manager.cli [212600] using the db container backendai-backendai-half-db-1 ...
psql (16.3)
Type "help" for help.

backend=# SELECT * FROM images
backend-# ORDER BY created_at DESC
backend-# LIMIT 1;
                  id                  |               name               | project |       image       |          created_at           | tag  | registry  |             registry_id              | architecture |                              config_digest                               | size_bytes | is_local |  type   | accelerators |                                                                                                      labels                                                                                                       |                                   resources                                   
--------------------------------------+----------------------------------+---------+-------------------+-------------------------------+------+-----------+--------------------------------------+--------------+--------------------------------------------------------------------------+------------+----------+---------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------
 e8925a8a-1251-4935-b995-ffd797ce337c | localhost/stable/bai-python:3.11 | stable  | stable/bai-python | 2024-09-06 03:54:58.979416+00 | 3.11 | localhost | abc42a05-4471-41fa-8772-10bf6452c7d9 | x86_64       | sha256:3a7829f911d5c601af3273f21df405b289acb51c35d90ac661a845f3a4a4a9f6  |  409172485 | f        | COMPUTE |              | {"ai.backend.base-distro": "ubuntu18.04", "ai.backend.features": "uid-match batch query", "ai.backend.kernelspec": "1", "ai.backend.runtime-path": "/usr/local/bin/python3", "ai.backend.runtime-type": "python"} | {"cpu": {"max": null, "min": "1"}, "mem": {"max": null, "min": "1073741824"}}
(1 row)

Checklist: (if applicable)

• Milestone metadata specifying the target backport version
• Mention to the original issue

[jopemachine]

• feat: Allow empty image name #2831
• feat: Replace image rescan command's --local flag with the local type ContainerRegistry record #2665
• feat: Support AWS ECR Private Container Registry based on PostgreSQL #2624
• feat: Support AWS ECR Public Container Registry based on PostgreSQL #2623
• feat: Support GitLab Container Registry based on PostgreSQL #2622
• feat: Support GitHub Container Registry based on PostgreSQL #2621
• feat: Support docker image manifest v2 schema1 scanning #2815 👈
• refactor: Base ContainerRegistry's scan_tag and implement MEDIA_TYPE_DOCKER_MANIFEST type handling #2620
• feat: Implement ID based client workflow to ContainerRegistry graphql API #2615
• feat: Add project column to the images table and refactor ImageRef related logics #2707
• feat: Migrate container registry config storage from Etcd to PostgreSQL #1917
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @jopemachine and the rest of your teammates on Graphite

[kyujin-cho]

Merge activity

• Sep 30, 11:06 AM EDT: Graphite rebased this pull request after merging its parent, because this pull request is set to merge when ready.