Bundle inspection and generation utilities

[haydentherapper]

Description

Inspired by @codysoyland's https://github.com/codysoyland/sigstore-bundle-upgrade and building on the upcoming bundle support (#3139), it would be great to have utilities to work with or create bundles outside of signing and verification workflows. This could include:

cosign bundle upgrade --version 0.3 --in-place foo.sigstore.json
cosign bundle synthesize --certificate foo.crt --signature foo.sig --attestation foo.json --out foo.sigstore.json
~~cosign bundle verify --identity foo foo.sigstore.json~~
cosign bundle inspect foo.sigstore.json
cosign bundle export --format protobuf --output foo.sigstore.pb foo.sigstore.json
cosign bundle export-certificate --output foo.crt foo.sigstore.json

In order:

• Upgrading a bundle version, resolving new requirements
• Merging detached verification material into a bundle
• Verifying that a bundle contains a specified identity (@codysoyland, did you have something else in mind for this? If this is effectively verify-blob without artifact verification, I'd skip this one, or at least rename)
• Provide user-readable bundle output (machine readable should just do cat foo.sigstore.json | jq)
• Export json bundle as protobuf (and should support proto -> json as well)
• Decompose bundle into detached verification material

Discussion: https://sigstore.slack.com/archives/C0440BFT43H/p1721088432507969

[codysoyland]

(re: cosign bundle verify)

@codysoyland, did you have something else in mind for this? If this is effectively verify-blob without artifact verification, I'd skip this one, or at least rename

Yeah, this one is probably redundant with existing verify commands, so it can be omitted.

[steiza]

Okay, so merging in the content from #3855, what about if we start with:

cosign bundle create

Assembles a protobuf ("new") bundle. Has flags similar to `verify-blob` and
`verify-blob-attestation`. Includes whatever information is not empty
strings.

    --attestations='':
        path to attestation FILE. Conflicts with --signature.

    --bundle='':
	path to old bundle FILE to convert to new bundle

    --certificate='':
	path to the public certificate. The certificate will be verified
	against the Fulcio roots.

    --out='':
	path to bundle FILE to output

    --rfc3161-timestamp='':
	path to RFC3161 timestamp FILE

    --sct='':
	path to a detached Signed Certificate Timestamp, formatted as a
	RFC6962 AddChainResponse struct. If a certificate contains an SCT,
	verification will check both the detached and embedded SCTs.

    --signature='':
	signature content or path or remote URL. Conflicts with
        --attestation.

Can we just assume this will only output new protobuf bundles? I think that's okay - I'm not sure why someone would be looking to take detached materials and create an old bundle. Or do we need a --new-bundle-format flag?

Then for trusted roots we can do something like:

cosign trusted-root create

Defaults to output PGI trusted root, obtained via TUF. Has flags similar to
`verify-blob` and `verify-blob-attestation`. Includes whatever information
is not empty strings (e.g. if you don't want a timestamp authority in your
trusted root, leave off `--timestamp-certificate-chain`). 

    --ca-intermediates='':
	path to a file of intermediate CA certificates in PEM format which
	will be needed when building the certificate chains for the signing
	certificate. The flag is optional and must be used together with
	--ca-roots, conflicts with --certificate-chain.

    --ca-roots='':
	path to a bundle file of CA certificates in PEM format which will be
	needed when building the certificate chains for the signing
	certificate. Conflicts with --certificate-chain.

    --certificate-chain='':
	path to a list of CA certificates in PEM format which will be needed
	when building the certificate chain for the signing certificate. Must
	start with the parent intermediate CA certificate of the signing
	certificate and end with the root certificate. Conflicts with
	--ca-roots and --ca-intermediates.

    --key='':
	path to the public key file, KMS URI or Kubernetes Secret

    --out='':
	path to trusted root FILE to output

    --rekor-url='https://rekor.sigstore.dev':
	address of rekor STL server

    --timestamp-certificate-chain='':
	path to PEM-encoded certificate chain file for the RFC3161 timestamp
	authority. Must contain the root CA certificate. Optionally may
	contain intermediate CA certificates, and may contain the leaf TSA
	certificate if not present in the timestamp

If this interface looks reasonable, I can start implementing!

[haydentherapper]

LGTM! A few details:

--sct can be dropped, we are deprecating detached SCTs (sigstore/fulcio#1499). For --bundle, we should note this is how an inclusion proof/SET is provided. One open question is do we want to fetch an inclusion proof is one is not present, for this and/or the cosign bundle upgrade case? If we're removing get-proof-by-hash, then I'd lean towards no.

Can we just assume this will only output new protobuf bundles?

Yes!

For cosign trusted-root create, we need to take in the Rekor public key too. I wouldn't rely on rekor-url providing you the public key (we should remove that functionality from the log). You also might need a few more details like signature algorithm and URLs for the CA and timestamp authority.

[woodruffw]

To tack on: another useful set of subcommand(s) here would be for interacting with PEP 740 attestation and provenance objects, which are semantically compatible with Sigstore bundles but have their own structure (to make them mesh with Python packaging land).

As a rough sketch, some things that would be useful:

# maybe this could be another `bundle synthesize` variant?
# convert an entire provenance object into >=1 sigstore bundles
cosign bundle convert-pep740 --provenance foo-1.2.3.tar.gz.provenance

# convert a single attestation object into its corresponding bundle
cosign bundle convert-pep740 --attestation foo-1.2.3.tar.gz.publish.attestation

The inverse (bundle to PEP 740) might also be useful, although this is already covered within standard uploading/signing flows.

[wking]

cosign bundle upgrade ... and cosign bundle create ... can help with old -> new conversion. Which is great if your consumer prefers the new version (and is maybe unaware of the old version) but your producer is still generating old-version signatures. But there could also be situations where a producer is generating only new-version signatures, and older consumers unaware of the new format or discovery mechanisms need the old format and discovery mechanisms. You could bridge from the new-version producer to old-version consumers with a new -> old converter. Is that direction in-scope for this issue? If not, does it deserve its own issue?

[steiza]

But there could also be situations where a producer is generating only new-version signatures, and older consumers unaware of the new format or discovery mechanisms need the old format and discovery mechanisms.

So where we're at today is that all of the Sigstore client libraries produce and consume the (new) protobuf bundle format, and cosign is the only client that produces or consumes the old signature format.

Cosign already has support for consuming the new bundle format (as of https://github.com/sigstore/cosign/releases/tag/v2.4.0 for blobs and as of last week for containers.

Cosign also has support for signing with the bundle format with --new-bundle-format, although it is not yet the default.

So the next step will be to switch cosign to default to the new bundle format, and let people specify a flag to verify the old signature format. At that point, all produces and consumers will default to using the new bundle format.

In this scenario, I don't think there's a reason why we need a new -> old converter. You would have to upgrade the version of cosign you were using in order to get that functionality, at which point you'd already have a version of cosign that can work with both formats.

[wking]

In this scenario, I don't think there's a reason why we need a new -> old converter. You would have to upgrade the version of cosign you were using in order to get that functionality, at which point you'd already have a version of cosign that can work with both formats.

There can be middle layers in the signature/bundle-distribution pipeline between the original producer and the end-consumer. Say you have an original-producer generating new-style bundles. And an end-consumer that only understands old-style signatures. Either the original-producer, the end-consumer, or some third party could run a new -> old converter to produce signatures the old consumer could find and understand until the end-consumer was able to update to tooling that understood the new bundles. As one example of this, you could have a mirroring team that pulled in new-style bundles from the public internet and walked them into a disconnected/restricted-network environment. The mirroring team might know that some of the local tools only understood old-style signatures, and the mirror team could run a new -> old converter to set up signatures that the older local tools could find and trust.

Some formats are just incompatible, and while new -> old converters would help with ecosystem transitions, they might not always be possible. I'm not clear enough on what goes into an old-style signature vs. what goes into a new-style bundle to know if new -> old conversion is possible. But if it is possible, I think it's worth consideration, to help folks bridge the transition. But it doesn't have to get discussed in this issue, if you think it deserves a separate issue. And even if it is possible, obviously Sigstore/Cosign maintainers are free to say things like "someone else is free to do this kind of thing, but we don't have time ourselves" or "we're happy to review and accept pulls to $REPO" or whatever the maintainer position happens to be.