Why can the image be pulled but no such manifest is displayed during manifest inspect? ——Docker Buildx Attestations inspection note

Have you ever encountered such a situation, for a mirror, it can be pulled normally:

 pad ~ # docker pull knatnetwork/github-runner-amd64:focal-2.301.1 focal-2.301.1: Pulling from knatnetwork/github-runner-amd64 846c0b181fff: Pull complete 588b3eef3b63: Pull complete 189ea0ac146f: Pull complete 4f4fb700ef54: Pull complete 546945707c6e: Pull complete 71464c2d54c9: Pull complete 1c4efc443e6a: Pull complete 21bbc223ea9a: Pull complete Digest: sha256:6b5b4aa94f8c1e781785e831d18d7ccc1a0de7d70d63b1afd4df3cce27ddd53f Status: Downloaded newer image for knatnetwork/github-runner-amd64:focal-2.301.1 docker.io/knatnetwork/github-runner-amd64:focal-2.301.1

But if you want to inspect its manifest, you will find no such manifest .

 pad ~ # docker manifest inspect knatnetwork/github-runner-amd64:focal-2.301.1 no such manifest: docker.io/knatnetwork/github-runner-amd64:focal-2.301.1

How could I have such a damn problem?

In April 2022, I open sourced GitHub Runner. The related article is: Open Source Github Actions Self-Hosted Runner , because the Image of this Runner is built on GitHub Actions, and in order to provide multi-architecture support (ARM64 and AMD64) And in order to ensure the construction speed, the whole construction work is divided into the following steps:

In the first stage, two Runners are opened at the same time to build the images of knatnetwork/github-runner-amd64:focal-2.301.1 and knatnetwork/github-runner-arm64:focal-2.301.1 respectively
After the above two Runners are completed, they are merged into a Multi-Arch image called knatnetwork/github-runner:focal-2.301.1 by manipulating the manifest

There has been no problem doing this until a few days ago when I encountered the first error when merging the mirror in the last step: https://github.com/knatnetwork/github-runner/actions/runs/3954481625/jobs/6776296661

 failed to put manifest docker.io/knatnetwork/github-runner:focal-2.301.1: errors: manifest blob unknown: blob unknown to registry

Strange, is it because some steps in GitHub have not been upgraded?

Thinking of seeing a bunch of The set-output command is deprecated and will be disabled soon. before, I tried to upgrade docker/login-action and docker/build-push-action , etc., and then retriggered the task, the result was still in An error was reported when merging images, but this time the content of the error is not the same, it is:

 Run docker manifest create knatnetwork/github-runner:focal-2.301.1 --amend knatnetwork/github-runner-amd64:focal-2.301.1 --amend knatnetwork/github-runner-arm64:focal-2.301.1 docker.io/knatnetwork/github-runner-amd64:focal-2.301.1 is a manifest list

Based on personal experience, if the same piece of code was able to run before, but suddenly cannot run now, in this case, the following possibilities are generally possible:

The Docker version of the GitHub Runner environment has changed
What’s changed in docker/login-action and docker/build-push-action , or what’s changed in the components (such as buildx) used by these steps
Something went wrong with DockerHub/GHCR

Let’s rule out the last possibility first, because after two days of retrying, we found that the problem is still there, and we haven’t seen a lot of feedback about the unavailability of these two services, so only the first two possibilities are left.

GitHub Runner Docker

First check if there is a problem caused by a Breaking change in Docker. The last successful Action is: https://github.com/knatnetwork/github-runner/actions/runs/3736662591, in the debugging information:

 Client: Version: 20.10.21+azure-2 API version: 1.41 Go version: go1.18.9 Git commit: baeda1f82a10204ec5708d5fbba130ad76cfee49 Built: Tue Oct 25 17:53:02 UTC 2022 OS/Arch: linux/amd64 Context: default Experimental: true Server: Engine: Version: 20.10.21+azure-2 API version: 1.41 (minimum version 1.12) Go version: go1.18.9 Git commit: 3056208812eb5e792fa99736c9167d1e10f4ab49 Built: Tue Oct 25 11:44:15 2022 OS/Arch: linux/amd64 Experimental: false

Action started by the first failure: https://github.com/knatnetwork/github-runner/actions/runs/3954481625/jobs/6776269393 , in the debug information:

 Client: Version: 20.10.22+azure-1 API version: 1.41 Go version: go1.18.9 Git commit: 3a2c30b63ab20acfcc3f3550ea756a0561655a77 Built: Thu Dec 15 15:37:38 UTC 2022 OS/Arch: linux/amd64 Context: default Experimental: true Server: Engine: Version: 20.10.22+azure-1 API version: 1.41 (minimum version 1.12) Go version: go1.18.9 Git commit: 42c8b314993e5eb3cc2776da0bbe41d5eb4b707b Built: Thu Dec 15 22:17:04 2022 OS/Arch: linux/amd64 Experimental: false

It seems that there are indeed some version upgrades, but after reading https://docs.docker.com/engine/release-notes/#201022 , I found that there is basically only a little patch, and there is no update that is enough to cause this problem.

So now the pressure comes to the second one, that is, “what changes have been made in docker/login-action and docker/build-push-action , or what changes have been made in the components (such as buildx) used by these steps”.

Manifest

Before continuing to investigate, let’s take a look at the situation of the above error report, why the image can be pulled, but the manifest cannot be seen, don’t you need to read the manifest before pulling the image?

The command that Docker uses to view the manifest is docker manifest inspect , but this command does not have an option similar to -v for debugging, so if you see no such manifest , then you have no way of knowing what is wrong behind it, but considering the manifest It’s just a JSON file, so there must be a Docker Hub API that can be queried, so I immediately sent a script online:

 #!/bin/sh ref = " ${ 1 :- library /ubuntu: latest } " sha = " ${ ref #*@ } " if [ " $sha " = " $ref " ] ; then sha = "" fi wosha = " ${ ref %%@* } " repo = " ${ wosha %:* } " tag = " ${ wosha ##*: } " if [ " $tag " = " $wosha " ] ; then tag = "latest" fi api = "application/vnd.docker.distribution.manifest.v2+json" apil = "application/vnd.docker.distribution.manifest.list.v2+json" token = $( curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository: ${ repo } :pull" \ | jq -r '.token' ) curl -H "Accept: ${ api } " -H "Accept: ${ apil } " \ -H "Authorization: Bearer $token " \ -s "https://registry-1.docker.io/v2/ ${ repo } /manifests/ ${ sha :- $tag } " | jq .

Source: https://stackoverflow.com/questions/57316115/get-manifest-of-a-public-docker-image-hosted-on-docker-hub-using-the-docker-regi

Then I found a normal image and tried it. The output is similar to this, which is consistent with the result of using docker manifest inspect :

 { "mediaType" : "application/vnd.docker.distribution.manifest.v2+json" , "schemaVersion" : 2 , "config" : { "mediaType" : "application/vnd.docker.container.image.v1+json" , "digest" : "sha256:19bf2d0d0a8aaf27988db772ff6ba4044405447535762bfc9ba451d0d84f0a18" , "size" : 4995 }, "layers" : [ { "mediaType" : "application/vnd.docker.image.rootfs.diff.tar.gzip" , "digest" : "sha256:846c0b181fff0c667d9444f8378e8fcfa13116da8d308bf21673f7e4bea8d580" , "size" : 28576882 }, ... { "mediaType" : "application/vnd.docker.image.rootfs.diff.tar.gzip" , "digest" : "sha256:74b36662af5e651ae3390a6cf13fcaa8fca08fea5bd711ddbed60bf9e5924654" , "size" : 932 } ] }

So I immediately took a look at the problematic image, and the result is this:

 { "errors" : [ { "code" : "MANIFEST_UNKNOWN" , "message" : "OCI index found, but accept header does not support OCI indexes" } ] }

From the OCI Image Index Specification document, we know that there are many types of manifests. We generally use application/vnd.docker.distribution.manifest.v2+json . If it is a multi-arch image, the output may look like this:

 { "manifests" : [ { "digest" : "sha256:93d5a28ff72d288d69b5997b8ba47396d2cbb62a72b5d87cd3351094b5d578a0" , "mediaType" : "application/vnd.docker.distribution.manifest.v2+json" , "platform" : { "architecture" : "amd64" , "os" : "linux" }, "size" : 528 }, { "digest" : "sha256:176bc6c6e93528f4b729fae1f8dbd70b73861264dba3a3f64c49c92e1f42a5aa" , "mediaType" : "application/vnd.docker.distribution.manifest.v2+json" , "platform" : { "architecture" : "s390x" , "os" : "linux" }, "size" : 528 } ], "mediaType" : "application/vnd.docker.distribution.manifest.list.v2+json" , "schemaVersion" : 2 }

Here its format is application/vnd.docker.distribution.manifest.list.v2+json , which is why the following two headers are included in the request in the script above.

 api="application/vnd.docker.distribution.manifest.v2+json" apil="application/vnd.docker.distribution.manifest.list.v2+json"

But here, according to the prompt OCI index found , we guessed that the actual manifest format might not match the above two, so we added the following two new Headers, and explicitly defined that we also accept application/vnd.oci.image.index.v1+json this format:

 api_old="application/vnd.oci.image.manifest.v1+json" api_oldi="application/vnd.oci.image.index.v1+json"

Soon, we saw that the problematic image can also be returned, and the data is as follows:

 { "mediaType" : "application/vnd.oci.image.index.v1+json" , "schemaVersion" : 2 , "manifests" : [ { "mediaType" : "application/vnd.oci.image.manifest.v1+json" , "digest" : "sha256:73809677ff2aff4bee611f1da7cdc9b8825c5729d2aab4c88b683cfa0e5fc7f0" , "size" : 1817 , "platform" : { "architecture" : "amd64" , "os" : "linux" } }, { "mediaType" : "application/vnd.oci.image.manifest.v1+json" , "digest" : "sha256:f47cf60d8b8da4e0f5040071b78ddb41f0ae160da6b1be7ddcba03a5c0bf9b3d" , "size" : 567 , "annotations" : { "vnd.docker.reference.digest" : "sha256:73809677ff2aff4bee611f1da7cdc9b8825c5729d2aab4c88b683cfa0e5fc7f0" , "vnd.docker.reference.type" : "attestation-manifest" }, "platform" : { "architecture" : "unknown" , "os" : "unknown" } } ] }

This is very interesting, I use:

 - name : Build and push AMD64 Version  
    uses : docker/build-push-action@v2  
    with :  
      context : ./amd64/  
      file : ./amd64/Dockerfile  
      platforms : linux/amd64  
      push : true  
      tags : | knatnetwork/github-runner-amd64:focal-$

Why is the manifests of the built image an array (like a multi-arch image), and the second platform is still unknown?

So it should also be caused by this reason: docker.io/knatnetwork/github-runner-amd64:focal-2.301.1 is a manifest list This error is reported, and the operation manifest to merge the mirror cannot merge two multi-Arch mirrors.

but why?

attestation manifest

In the above output, we saw a key message: "vnd.docker.reference.type": "attestation-manifest" After searching, we saw this document: Attestation storage | Docker Documentation

Buildkit supports creating and attaching attestations to build artifacts. These attestations can provide valuable information from the build process, including, but not limited to: SBOMs, SLSA Provenance, build logs, etc.

Oh? Is it a Buildkit thing?

So I started to check the last successful Buildx version and found that it was:

 github.com/docker/buildx 0.9.1+azure-2 ed00243a0ce2a0aee75311b06e32d33b44729689

Look again at the Buildx version that failed for the first time:

 github.com/docker/buildx 0.10.0+azure-1 876462897612d36679153c3414f7689626251501

The version has been upgraded from 0.9.1 to 0.10.0. At this time, review the following passage in the Release Note of docker/build-push-action :

Buildx v0.10 enables support for a minimal SLSA Provenance attestation, which requires support for OCI-compliant multi-platform images. This may introduce issues with registry and runtime support (eg GCR and Lambda). You can optionally disable the default provenance attestation functionality using provenance: false.

We soon learned that the problem here is that Buildx has added this thing called SLSA Provenance attestation by default since 0.10, which is the "vnd.docker.reference.type": "attestation-manifest" at the bottom of the manifest we saw content, this has no effect on directly built Multi-Arch images, and generally has no effect on single-architecture images (although an error will be reported during docker manifest inspect ), but once there are multiple parallel builds like mine, When the merge operation of the manifest is operated later, it will cause errors like docker.io/knatnetwork/github-runner-amd64:focal-2.301.1 is a manifest list .

If you want to know more about Build attestations, you can start reading from Docker’s documentation: Build attestations | Docker Documentation , which is simply divided into SBOM and Provenance:

Build attestations describe how an image was built, and what it contains. The attestations are created at build-time by BuildKit, and become attached to the final image as metadata.

Two types of build annotations are available:

Software Bill of Material (SBOM): list of software artifacts that an image contains, or that were used to build the image.

Provenance: how an image was built.

Now that the problem is clear, the idea of solving the problem is also clear. Just add the following two lines to docker/build-push-action :

 provenance: false sbom: false

After the build, we confirmed again through the script and found that the manifest was normal, as follows:

 { "mediaType" : "application/vnd.docker.distribution.manifest.v2+json" , "schemaVersion" : 2 , "config" : { "mediaType" : "application/vnd.docker.container.image.v1+json" , "digest" : "sha256:82da6a4f14803932bfece329e5d2592b74dbbb65a3c493bb6b459fb8b3a082ff" , "size" : 4995 }, "layers" : [ { "mediaType" : "application/vnd.docker.image.rootfs.diff.tar.gzip" , "digest" : "sha256:846c0b181fff0c667d9444f8378e8fcfa13116da8d308bf21673f7e4bea8d580" , "size" : 28576882 }, ... { "mediaType" : "application/vnd.docker.image.rootfs.diff.tar.gzip" , "digest" : "sha256:8b5ad40966565f7a972b30cf9494aa3600645350952d99f1d442c143a03d2650" , "size" : 932 } ] }

As for the manifest blob unknown: blob unknown to registry problem encountered at the beginning, the guess is that the merged image needs to be under one repo, and the logic should be:

knatnetwork/github-runner-amd64:latest and knatnetwork/github-runner-arm64:latest cannot be merged
knatnetwork/github-runner:latest-amd64 and knatnetwork/github-runner:latest-arm64 can be merged

However, it seems that there is no way to explain why it was possible to do this before. If any readers know, please point it out in the comment area.

Summary

To sum up, in order to solve the above two problems, I made the following adjustments respectively:

Explicitly disabled provenance and sbom output in docker/build-push-action
Change the image of amd64 and arm64 to the different tag output of the same repo

At the same time, I came to a conclusion: If you want to adjust the image by operating the manifest later, you must pay attention to this new feature of buildx, or explicitly disable it, or consider modifying your Dockerfile. Try to build Multi through buildx at one time -Mirror of Arch.

References

This article is transferred from https://nova.moe/docker-attestation/
This site is only for collection, and the copyright belongs to the original author.