Call for Action: Strengthening Paketo Buildpacks Against Upstream Vulnerabilities

[ubermensch37]

Hello Community

I'm advocating for the adoption of Paketo Buildpacks within our organization. We are transitioning from Dockerfiles to buildpacks to achieve a more secure and standardized image creation process. To automate our strict security standards, we use Snyk scanning into our CI/CD pipeline and enforce a zero-vulnerability policy for all image releases.

However, scans have revealed vulnerabilities in the base images used by some Paketo Buildpacks (latest and all versions). I'm opening this discussion to review these findings and collaboratively explore effective mitigation strategies.

Key Points

• Enhanced Security with Buildpacks:
  We are moving away from Dockerfiles in favor of buildpacks to improve security and standardize our build process.

• Zero-Vulnerability Policy:
  Our CI/CD pipeline uses Snyk scanning to ensure that no vulnerabilities are present in the released images.

• Upstream Challenges:
  Some vulnerabilities originate from upstream dependencies in the base images. These issues fall under Paketo Buildpacks’ security policy, which recommends reporting such vulnerabilities to the upstream maintainers.

Reproduction Steps

1. Build the Image:
   Run the following command (using any Paketo Builder, for example):

   pack build buildpack-archetype:fulllatest --builder paketobuildpacks/builder-jammy-full

2. Scan the Image:
   Execute the Snyk container test:

   snyk container test buildpack-archetype:fulllatest

Vulnerability Breakdown

Detailed Findings (Examples):

• tiff/libtiff5 (CVE-2024-6716)

  • Layer: OS Packages
  • Severity: High
  • Description: Out-of-bounds write vulnerability.

• systemd/libsystemd0 (CVE-2023-7008)

  • Layer: OS Packages
  • Severity: High
  • Description: Potential memory corruption vulnerability.

• openssl (CVE-2024-41996)

  • Layer: OS Packages
  • Severity: Critical
  • Description: Denial of service vulnerability.

...and additional issues affecting packages such as sqlite, shadow, samba, pcre, openjpeg2, openexr, ncurses, libzstd, libpng1.6, libgcrypt20, krb5, intel-mediasdk, imagemagick, gnupg2, glibc, gcc-12, flex, dbus, curl, coreutils, cairo, wget, pixman, pam, libxml2, libtheora, libheif, libde265, libcap2, ffmpeg, avahi, freetype, among others.

Call for Community Action

Versioning Strategies:

• How can we best identify and select Paketo Buildpack versions that incorporate the most secure base images?
• What is the typical turnaround time for base image updates to integrate upstream security fixes?

Mitigating Upstream Risks:

• What strategies can we implement to minimize vulnerabilities inherited from upstream dependencies?
• Are there alternative base images or buildpack configurations that offer a more secure starting point?

Best Practices for a Secure Paketo Workflow:

• What tools, configurations, or best practices can help us maintain our zero-vulnerability policy with Paketo Buildpacks?
• How can we proactively detect and address potential vulnerabilities before they impact our builds?

I believe that a collaborative discussion on these points is essential to fortify the security of Paketo Buildpacks and promote their wider adoption. I look forward to hearing your insights and experiences.

Thank you!

[dmikusa]

enforce a zero-vulnerability policy for all image releases

You might get this with Paketo stacks when they are newer/first released, but as time goes on there will be CVEs that don't get patched.

It's not something that we, Paketo, can control. We base on Ubuntu images and that is just the way they handle CVEs. They do not guarantee they will patch everything, so over time you will start to see low severity issues in the stack that don't get patched.

There are no guarantees with OSS software, but this is what we strive for with the Paketo stacks.

https://paketo.io/docs/concepts/stacks/#when-are-paketo-stacks-updated

This has come up before and what you'd need to do is create your own custom stack.

https://buildpacks.io/docs/for-platform-operators/how-to/build-inputs/stack/

You can then choose to base off an upstream image that has a security policy that meets your needs (i.e. upstream has a zero-vulnerability policy). Most Paketo buildpacks will run on any Linux distro with reasonable glibc versions, but we don't test other distributions so you would want to do some testing. The farther you get from an Ubuntu/Debian base, the more likely you might run into issues. I have run Java apps on a UBI base image though and that worked fine for me.

If that sounds like too much work, then I would suggest looking for a company or consultant that can provide these features for you.

How can we best identify and select Paketo Buildpack versions that incorporate the most secure base images?

All of the Paketo buildpacks are meant to run on the Paketo stacks (tiny/base/full). You can create custom stacks and in many cases Paketo buildpacks will run on those as well, so long as there's a suitable glibc version. That's not a guarantee though, we don't test against any other stacks so that testing is up to you.

What is the typical turnaround time for base image updates to integrate upstream security fixes?

https://paketo.io/docs/concepts/stacks/#when-are-paketo-stacks-updated

but it's important to remember that ultimately that depends on when Ubuntu/upstream patches.

What strategies can we implement to minimize vulnerabilities inherited from upstream dependencies?

If the upstream stack isn't sufficient, then you'd need to make your own stack. There's not really any way to patch stacks on the fly with buildpacks.

Are there alternative base images or buildpack configurations that offer a more secure starting point?

You can make a custom stack/base image. Aside from Paketo, the main other CNB offerings I've seen are Heroku's and Google's.

What tools, configurations, or best practices can help us maintain our zero-vulnerability policy with Paketo Buildpacks?

Being honest, I don't know that you're going to be able to do this. Get to zero-vulnerabilities. You can try but it's likely to be an uphill battle.

The only idea that comes to mind would be if you were to purchase a support contract from Ubuntu and they could provide you access to upstream Ubuntu images where they patch all of the vulnerabilities. If you can get an Ubuntu base image that has zero-vulnerabilities, then layering Paketo on top of that should not introduce vulnerabilities (or if it does, those are ones that we can patch). You'd still need to make a custom stack, but you could take our automation and re-run them with your zero-vuln base image. That would give you the best compatibility with Paketo buildpacks because you'd still be using an Ubuntu base image.

Hope that helps!