Apps with huge amount of files possible lead to K8s node DoS

[BernhardDenner]

What happened?

We are packaging spring boot based apps, which also serve static UI content files. Due to the nature of UI dev we now have some apps that have a huge amount of static files that are served via the spring-boot app. Not sure if this is an anti-pattern or not but this situation was the root cause for the outage of some of our K8s nodes.

The reason for this was quite obvious once we have investigated the issue. The buildpack puts the a file listing of the app code as metadata to the resulting image label io.buildpacks.lifecycle.metadata. As a result of this, the image and later then the K8s pod metadata get quite huge. In our case the image metadata grows to around 3.2MB in size. When we now run multiple such pods on the same K8s node the overall metadata of all pods can get quite huge. And exactly this is the reason for forcing the K8s node to go down.

So, this is my current understanding what leads to this situation:
The K8s kubelet component responsible for retrieving the current state of all containers (part of PLEG) uses gRPC to communicate with the container runtime. This gRPC has a maximum message size of 16MB. If now the metadata of all containers on a node grow beyond this 16MB limit, kubelet can't query the state of its containers any more, finally leading to an unusable K8s node. The error message that we see in this situation is:

... code = ResourceExhausted desc = grpc: trying to send message larger than max (21922673 vs. 16777216)

See here the corresponding K8s issue: kubernetes/kubernetes#63858

A restart of kubelet or even a reboot of the node does not fix this situation, as the containers with the huge overall metadata are still there. The only way to get out of this situation is to manually remove some of the containers started from such a packeto buildpack created image.

From checking the buildpack sources I understand that on

spring-boot/boot/web_application_type.go

Line 38 in d085524

expected["files"], err = sherpa.NewFileListing(applicationPath)

the file listing is added to the image metadata, but without any filtering. What I do not understand: why is this required?

I also see quite a huge amount of on the data on other lables, e.g. io.paketo.stack.packages. In the end this strategy could be a problem when adding data unconditionally.

I would consider this also to be a security issue, since once could quickly take down a cf-for-k8s cluster.

How to reproduce

• pack a java Spring-boot application jar with lots of static files (~12K or more) with spring-boot buildpack
• run the resulting image on a K8s node multiple times

Additional info

• What were you attempting to do?

Push several Spring-Boot applications with an huge amount of static files via CF-for-K8s. All is packed to a single jar file. So I'm doing a cf push -p app.jar.

The same behavior can be triggered when scaling such an application horizontally.

• What did you expect to happen?

All pushed applications are running and K8s cluster is in good conditions ;)

• What was the actual behavior? Please provide log output, if possible.

Some K8s nodes go out of service (NotReady). Kubelet logs start looking like this

E0511 11:27:39.173420    3892 kuberuntime_container.go:382] getKubeletContainers failed: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (21922673 vs. 16777216)
E0511 11:27:39.173460    3892 generic.go:205] GenericPLEG: Unable to retrieve pods: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (21922673 vs. 16777216)
E0511 11:27:40.943159    3892 remote_runtime.go:312] ListContainers with filter &ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (21922673 vs. 16777216)
E0511 11:27:40.943249    3892 kuberuntime_container.go:382] getKubeletContainers failed: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (21922673 vs. 16777216)
E0511 11:27:40.943273    3892 generic.go:205] GenericPLEG: Unable to retrieve pods: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (21922673 vs. 16777216)
E0511 11:27:42.722286    3892 remote_runtime.go:312] ListContainers with filter &ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (21922673 vs. 16777216)
E0511 11:27:42.722338    3892 kuberuntime_container.go:382] getKubeletContainers failed: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (21922673 vs. 16777216)
E0511 11:27:42.722352    3892 generic.go:205] GenericPLEG: Unable to retrieve pods: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (21922673 vs. 16777216)

Build Configuration

• What platform (pack, kpack, tekton buildpacks plugin, etc.) are you
  using? Please include a version.
  CF-for-k8s v3.0.0, kpack 0.2.2

log output of the staging process ...

 build                                                                                                                                                                                                                                      
 build Paketo CA Certificates Buildpack 2.1.0                                                                                                                                                                                               
 build   https://github.com/paketo-buildpacks/ca-certificates                                                                                                                                                                               
 build   Launch Helper: Contributing to layer                                                                                                                                                                                               
 build     Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper                                                                                                                                          
 build                                                                                                                                                                                                                                      
 build Paketo BellSoft Liberica Buildpack 7.1.0                                                                                                                                                                                             
 build   https://github.com/paketo-buildpacks/bellsoft-liberica                                                                                                                                                                             
 build   Build Configuration:                                                                                                                                                                                                               
 build     $BP_JVM_VERSION              11              the Java version                                                                                                                                                                    
 build   Launch Configuration:                                                                                                                                                                                                              
 build     $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation                                                                                                                                                  
 build     $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation                                                                                                                                  
 build     $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation                                                                                                                                         
 build     $JAVA_TOOL_OPTIONS                           the JVM launch flags                                                                                                                                                                
 build   BellSoft Liberica JRE 11.0.10: Contributing to layer                                                                                                                                                                               
 build     Downloading from http://cf-buildpack-offline-dependencies-svc.offline-deps:8080/bellsoft-jre11.0.10+9-linux-amd64.tar.gz                                                                                                         
 build     Verifying checksum                                                                                                                                                                                                               
 build     Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jre                                                                                                                                                                     
 build     Adding 6 container CA certificates to JVM truststore                                                                                                                                                                             
 build     Writing env.launch/BPI_APPLICATION_PATH.default                                                                                                                                                                                  
 build     Writing env.launch/BPI_JVM_CACERTS.default                                                                                                                                                                                       
 build     Writing env.launch/BPI_JVM_CLASS_COUNT.default                                                                                                                                                                                   
 build     Writing env.launch/BPI_JVM_SECURITY_PROVIDERS.default                                                                                                                                                                            
 build     Writing env.launch/JAVA_HOME.default                                                                                                                                                                                             
 build     Writing env.launch/MALLOC_ARENA_MAX.default                                                                                                                                                                                      
 build   Launch Helper: Contributing to layer                                                                                                                                                                                               
 build     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/active-processor-count                                                                                                                                        
 build     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/java-opts                                                                                                                                                     
 build     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/link-local-dns                                                                                                                                                
 build     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator                                                                                                                                             
 build     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/openssl-certificate-loader                                                                                                                                    
 build     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-configurer                                                                                                                                 
 build     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-classpath-9                                                                                                                                
 build   JVMKill Agent 1.16.0: Contributing to layer                                                                                                                                                                                        
 build     Downloading from http://cf-buildpack-offline-dependencies-svc.offline-deps:8080/jvmkill-1.16.0-RELEASE.so                                                                                                                        
 build     Verifying checksum                                                                                                                                                                                                               
 build     Copying to /layers/paketo-buildpacks_bellsoft-liberica/jvmkill                                                                                                                                                                   
 build     Writing env.launch/JAVA_TOOL_OPTIONS.append                                                                                                                                                                                      
 build     Writing env.launch/JAVA_TOOL_OPTIONS.delim                                                                                                                                                                                       
 build   Java Security Properties: Contributing to layer                                                                                                                                                                                    
 build     Writing env.launch/JAVA_SECURITY_PROPERTIES.default                                                                                                                                                                              
 build     Writing env.launch/JAVA_TOOL_OPTIONS.append                                                                                                                                                                                      
 build     Writing env.launch/JAVA_TOOL_OPTIONS.delim                                                                                                                                                                                       
 build                                                                                                                                                                                                                                      
 build Paketo Executable JAR Buildpack 5.0.0                                                                                                                                                                                                
 build   https://github.com/paketo-buildpacks/executable-jar                                                                                                                                                                                
 build   Class Path: Contributing to layer                                                                                                                                                                                                  
 build     Writing env/CLASSPATH.delim                                                                                                                                                                                                      
 build     Writing env/CLASSPATH.prepend                                                                                                                                                                                                    
 build   Process types:                                                                                                                                                                                                                     
 build     executable-jar: java org.springframework.boot.loader.JarLauncher (direct)                                                                                                                                                        
 build     task:           java org.springframework.boot.loader.JarLauncher (direct)                                                                                                                                                        
 build     web:            java org.springframework.boot.loader.JarLauncher (direct)                                                                                                                                                        
 build                                                                                                                                                                                                                                      
 build Paketo Spring Boot Buildpack 4.1.0                                                                                                                                                                                                   
 build   https://github.com/paketo-buildpacks/spring-boot                                                                                                                                                                                   
 build   Launch Helper: Contributing to layer                                                                                                                                                                                               
 build     Creating /layers/paketo-buildpacks_spring-boot/helper/exec.d/spring-cloud-bindings                                                                                                                                               
 build   Spring Cloud Bindings 1.7.0: Contributing to layer                                                                                                                                                                                 
 build     Downloading from http://cf-buildpack-offline-dependencies-svc.offline-deps:8080/spring-cloud-bindings-1.7.0.jar                                                                                                                  
 build     Verifying checksum                                                                                                                                                                                                               
 build     Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings                                                                                                                                                           
 build   Web Application Type: Contributing to layer                                                                                                                                                                                        
 build     Servlet web application detected                                                                                                                                                                                                 
 build     Writing env.launch/BPL_JVM_THREAD_COUNT.default                                                                                                                                                                                  
 build   Image labels:                                                                                                                                                                                                                      
 build     org.opencontainers.image.title                                                                                                                                                                                                   
 build     org.opencontainers.image.version                                                                                                                                                                                                 
 build     org.springframework.boot.spring-configuration-metadata.json                                                                                                                                                                      
 build     org.springframework.boot.version                                                                                                                                                                                                 
 build stream closed                                                                                                                                                                                                                        
 export Adding layer 'paketo-buildpacks/ca-certificates:helper'                                                                                                                                                                             
 export Adding layer 'paketo-buildpacks/bellsoft-liberica:helper'                                                                                                                                                                           
 export Adding layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'                                                                                                                                                         
 export Adding layer 'paketo-buildpacks/bellsoft-liberica:jre'                                                                                                                                                                              
 export Adding layer 'paketo-buildpacks/bellsoft-liberica:jvmkill'                                                                                                                                                                          
 export Adding layer 'paketo-buildpacks/executable-jar:classpath'                                                                                                                                                                           
 export Adding layer 'paketo-buildpacks/spring-boot:helper'                                                                                                                                                                                 
 export Adding layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'                                                                                                                                                                  
 export Adding layer 'paketo-buildpacks/spring-boot:web-application-type'                                                                                                                                                                   
 export Adding 1/1 app layer(s)                                                                                                                                                                                                             
 export Adding layer 'launcher'                                                                                                                                                                                                             
 export Adding layer 'config'                                                                                                                                                                                                               
 export Adding layer 'process-types'                                                                                                                                                                                                        
 export Adding label 'io.buildpacks.lifecycle.metadata'                                                                                                                                                                                     
 export Adding label 'io.buildpacks.build.metadata'                                                                                                                                                                                         
 export Adding label 'io.buildpacks.project.metadata'                                                                                                                                                                                       
 export Adding label 'org.opencontainers.image.title'                                                                                                                                                                                       
 export Adding label 'org.opencontainers.image.version'                                                                                                                                                                                     
 export Adding label 'org.springframework.boot.spring-configuration-metadata.json'                                                                                                                                                          
 export Adding label 'org.springframework.boot.version'                                                                                                                                                                                     
 export Setting default process type 'web'                                                                                                                                                                                                  

• What buildpacks are you using? Please include versions.

 detect 5 of 18 buildpacks participating                                                                                                                                                                                                    
 detect paketo-buildpacks/ca-certificates   2.1.0                                                                                                                                                                                           
 detect paketo-buildpacks/bellsoft-liberica 7.1.0                                                                                                                                                                                           
 detect paketo-buildpacks/executable-jar    5.0.0                                                                                                                                                                                           
 detect paketo-buildpacks/dist-zip          4.0.0                                                                                                                                                                                           
 detect paketo-buildpacks/spring-boot       4.1.0

• What builder are you using? If custom, can you provide the output from pack inspect-builder <builder>?

cf-default-builder, with stack: bionic-stack (CF-for-k8s default configuration)

• Can you provide a sample app or relevant configuration (buildpack.yml,
  nginx.conf, etc.)?

CF-for-k8s default configuration.

Checklist

• I have included log output.
• The log output includes an error message.
• I have included steps for reproduction.

[dmikusa]

Thanks for raising this up. I need to look into this more, but I suspect that these labels are part of the BOM (Bill of Materials) that gets generated. This is an auditing mechanism that provides information about the container and can be used for things like security scanners, license scanners, etc...

I'll update you when I have some more info. Thanks.

[BernhardDenner]

@dmikusa-pivotal do I understand correctly that this metadata is not required from a functional perspective? i.e. we could happy live without that metadata and do not loose any functionality? (Thinking of workaround to now)

[mancave]

@dmikusa-pivotal, the runtime security app (https://falco.org/) also marks the image insecure based on the tags

[dmikusa]

@dmikusa-pivotal do I understand correctly that this metadata is not required from a functional perspective? i.e. we could happy live without that metadata and do not loose any functionality? (Thinking of workaround to now)

Still digging into this. It may also be related to how the app is tracking changes & caching layers. That would be functional, but also much easier to fix. I'll share more details shortly.

[dmikusa]

the runtime security app (https://falco.org/) also marks the image insecure based on the tags

Can you expand on this? I'm not familiar with the site you listed. What doesn't it like about the tags?

[MG2R]

I'm seeing similar problems, not necessarily because of a huge number of files, but because of the listing of all available boot options. The docker describe of a single Java container built by buildpacks has a size of 528619 bytes (half a megabyte). 32 of these containers and I'm hitting the gRPC limits in K8S. This particular container is a worker that processes messages from a queue, so we run quite a lot of them in parallel.

513174 bytes of that message size are for the labels, with the org.springframework.boot.spring-configuration-metadata.json label being responsible for 416176 bytes of that.

[jsimao71]

We are also having the issue with the docker images built from spring boot apps. This is a major blocker for us, as we are preparing our system to go production with cf4ks. Thus, fixing this issue is a requirement for us to be able to go live.
@dmikusa-pivotal can you provide us some feedback: 1 - are you already working on the issue or plan to start working o the issue soon? 2 - when can we expect to have a fix or a workaround?! 3 - If the answer to 1-2 is negative or uncertain, what hints can you provide us to try to fix the issue on our own (and possibly contributed with a PR).

[dmikusa]

UPDATE

1. I have adjusted the code so that io.buildpacks.lifecycle.metadata does not grow in proportion to the number of files in the app. Previously, the buildpack was storing a list of all of the files & file metadata in the layer's metadata. It was doing this to detect when there is a change in the application, that way it could reevaluate the layer. Instead of storing the actual data, the change I made is now going to take a sha256 hash of all the data and store just the hash. This should have the same effect.

   The hash is being stored in the same location within the metadata as the list of files was previously being stored. This should have the effect that on the next build, it overwrites the list of files with the hash, which should shrink the size of the label. tl;dr - if you have existing images, once the fix has been released rebuild your images and the label size should shrink.

2. There is a second label called "org.springframework.boot.spring-configuration-metadata.json", which is quite large as well. It's about 500K for my test app. This label is storing a merged list of all of the configuration options available to the Spring Boot application. I'm still trying to understand the backstory on this one. I don't see the buildpack itself using this information, more just publishing it for consumption by someone using the image. For example, if your operations team wanted to see what configuration options were exposed by the application, they could read this label and get a full list. I could see tooling, dashboards or other maybe even reporting as other possible consumers of this data.

   I need to investigate this a bit more, but right now my plan is to expose an option to skip adding these labels. It would be an env variable that you set & the buildpack will just bypass generating & adding the label. I feel like it needs to be an opt-out setting because a.) making it opt-in would change the default behavior (i.e. be a breaking change) and that might affect people using these labels and b.) I hope Kubernetes removes or raises this limit in the future so this becomes less of an issue.

[dmikusa]

UPDATE

I have a PR that will be merged shortly to address the issues raised up here. It implements #1 from my previous note as described.

It has a different implementation for item #2 though. Rather than add flags to turn on/off these labels, we have just removed the label. The rationale being that if 95% of people turn them off, you can't build any sort of useful tooling around these labels. Thus we just removed them. The exception to this is for Spring DataFlow applications. I believe that the DataFlow UI is using these labels and so we can't remove them in this case without breaking functionality elsewhere.

[dmikusa]

OK v4.3.0 has this: https://github.com/paketo-buildpacks/spring-boot/releases/tag/v4.3.0. That version should get picked up by the multi buildpacks and builders shortly.

[imitbn]

Spring Cloud Data Flow Server uses org.springframework.cloud.dataflow.spring-configuration-metadata.json label instead of org.springframework.boot.spring-configuration-metadata.json , that is much smaller:
https://github.com/spring-cloud/spring-cloud-dataflow/blob/main/spring-cloud-dataflow-configuration-metadata/src/main/java/org/springframework/cloud/dataflow/configuration/metadata/BootApplicationConfigurationMetadataResolver.java#L99

I tried to override the label by adding --buildpack paketo-buildpacks/image-labels --env 'BP_IMAGE_LABELS=org.springframework.boot.spring-configuration-metadata.json=""', but other labels org.springframework.* disappear.

The only workaround for overriding (since labels can't be removed) that I found is building a new image based on the image created by buildpacks:
echo "FROM my-base-buildpacks-image" | docker build --label org.springframework.boot.spring-configuration-metadata.json="" -t "my-image" -
But it's pretty inefficient.

@dmikusa, сould you reconsider adding flags to exclude labels?

[dmikusa]

@imitbn Can you open a new issue for this? I think we can discuss adding an opt-out. Technically it wouldn't be hard. I think the reason we left this in is that there is software outside of buildpacks that is reading the label. If we just removed it that would be a breaking change. Anyway, we can discuss the impact of adding an opt-out on a new issue. Thanks

[imitbn]

@dmikusa Is it supposed to be a part of just spring-boot buildpack or more general (like tiny-builder/base-builder/full-builder)?

[dmikusa]

It's specific to this buildpack, so you can open the issue here.