Terminating due to java.lang.OutOfMemoryError: Metaspace

[GIREESH-P]

we are seeing some of our java 21 spring-boot applications with an error message as
"Terminating due to java.lang.OutOfMemoryError: Metaspace"

When we checked the application logs on KOB, we saw the Loaded Class Count: 28339 and subsequently based on the JVM memory calculator they are assigning -XX:MaxMetaspaceSize=174185K for the application.

[sbil-asset-api-ge1-f665f55c5-xbbdp] Adding $JAVA_OPTS to $JAVA_TOOL_OPTIONS
[sbil-asset-api-ge1-f665f55c5-xbbdp] Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx1266938K -XX:MaxMetaspaceSize=174185K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 1953124K, Thread Count: 250, Loaded Class Count: 28339, Headroom: 0%)

[sbil-asset-api-ge1-f665f55c5-xbbdp] Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -Dspring.profiles.active=cloud,ge1,security-oauth2,master -Dinfo.version=26.0205.0-2502241229-373718fd -XX:MaxDirectMemorySize=10M -Xmx1266938K -XX:MaxMetaspaceSize=174185K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

However, this metaspace is insufficient and the application crashes as soon as we receive load on the application.

For the same application deployed on PCF, it shows the Loaded Classes: 35530 and it has a higher -XX:MaxMetaspaceSize=219050K

2025-02-25T12:12:57.375+05:30 [STG/0] [OUT] [1m[31m----->[0m[22m Downloading [1m[34mOpen JDK Like Memory Calculator[0m[22m [34m3.13.0_RELEASE[0m from https://java-buildpack.cloudfoundry.org/memory-calculator/jammy/x86_64/memory-calculator-3.13.0-RELEASE.tar.gz [3m[32m(found in cache)[0m[23m
2025-02-25T12:12:58.321+05:30 [STG/0] [OUT] Loaded Classes: 35530, Threads: 250

Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:MaxMetaspaceSize=219050K -Dspring.profiles.active=cloud,ge1,security-oauth2,master -Dinfo.version=26.0205.0-2502241229-373718fd -XX:MaxDirectMemorySize=10M -Xmx1222074K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

So we tried to update the -XX:MaxMetaspaceSize=219050K through JAVA_OPTS variable for KOB application and it is working as expected. So we wanted to know how we can optimally manage the memory parameters rather than having to manually manage them for each of our micro services.

[dmikusa]

There is only so much that the memory calculator can do when it comes to picking a metaspace size, but we believe there is a relationship between class count and required metaspace size that seems to hold up reasonably well in a lot of cases; that's why the memory calculator counts classes. This basic premise is the same in Paketo & the old CF JBP.

The fact you're saying the class count with Paketo is different from using the old CF JBP is unexpected. They are different code bases, though, so it's not guaranteed to be the same, but they are doing basically the same thing.

My suggestion would be to look into your application and try to see why it's loading a different class count. Memory calculator will go through all of your code and JAR files (recursively), counting anything with one of the following extensions: ".class", ".classdata", ".clj", ".groovy", ".kts". It's a little tedious, but you can do the same and try to figure out if what it's counting is reasonable. It will also count and include classes in the JRE/JDK, although that number is pretty constant.

Aside from that, you could enable the -verbose flag for the JVM. That will print where it loads classes from. Watch that and see what classes are being loaded into metaspace. You could count and compare with the count detected by the memory calculator. You can also look for things that wouldn't be counted, like dynamic class generation. It is possible to create new classes on the fly in Java code and the memory calculator can't do anything to detect that.

Lastly, for the rare cases where the memory calculator is off, we do recommend just adjusting the metaspace size as you've done. If you're seeing this wrong for many of your apps, that would be unexpected. From observation, we see the default works well most of the time (think 80/20 or 90/10). If you're seeing it wrong on many of your apps, IMO, that's worth investigating. Either something in the apps is triggering higher than expected metaspace usage, or there's a bug in the memory calculator, and we're missing some classes with your application.

[andrej-urvantsev]

@GIREESH-P I think we had a similar problem in one service, but there is a way to adjust how memory is calculated before startup.

You can set BPL_JVM_CLASS_ADJUSTMENT environment variable in container to "125%" for example.

Read here for details: https://github.com/paketo-buildpacks/bellsoft-liberica?tab=readme-ov-file#configuration

Also here: paketo-buildpacks/spring-boot#235 (comment)