jmx_exporter report wrong TotalPhysicalmemorySize

[zhashuyu]

I have a java app which running in Kubernetes, and config resource limit, here,

resources:
  limits:
    cpu: 2
    memory: 4Gi
  requests:
    cpu: 2
    memory: 4Gi

when I deploy jmx_exporter whith java agent mode, the metrics show here,

java_lang_OperatingSystem_AvailableProcessors 2.0
java_lang_OperatingSystem_TotalPhysicalMemorySize 5.0475278336E10

The jmx_exporter show AvailableProcessors = 2.0 , the same size in pod resources config, but the total physical memory size = 47G (5.0475278336E10) , which is way big than pod resources config (4G).
The question is : does jmx_exporter acknowledge the existence of kubernetes pod resources config , and why cpu and memory show different result.

env:

• java: openjdk 1.8.0-242
• java app command line: java -javaagent:./jmx_prometheus_javaagent-1.2.0.jar=8989:jmx_exporter_config.yaml -Xms2048m -Xmx3072m xxx.jar
• jmx_prometheus_javaagent: 1.2.0
• kubernetes: 1.21.6
• docker: 20.10.7
• kubernetes node resource: memory 47G / cpu 16vCore
• jmx_exporter_config.yaml:

rules: 
- pattern: ".*"

[dhoard]

Java 8 incorrectly reports the memory of the container host and not the container memory m. This is a well-documented Java 8 issue when running in a container.

Java 11+ correctly reports the container memory.

[zhashuyu]

But according this: https://www.baeldung.com/java-docker-jvm-heap-size , java 1.8.0.242 does support container.

The JVM is pretty good at determining appropriate default [memory settings](https://www.baeldung.com/java-stack-heap).

In the past, [the JVM was not aware of the memory and CPU allocated to the container](https://developers.redhat.com/blog/2017/03/14/java-inside-docker/). So, Java 10 introduced a new setting: +UseContainerSupport (enabled by default) to fix the [root cause](https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8146115), and developers backported the fix to Java 8 in [8u191](https://www.oracle.com/technetwork/java/javase/8u191-relnotes-5032181.html#JDK-8146115). The JVM now calculates its memory based on the memory allocated to the container.

[dhoard]

@zhashuyu The value is being retrieved from the com.sun.management:type=OperatingSystem MBean, which is only included in Oracle (or OpenJDK-based distributions that include the com.sun.management extensions).

I believe the JVM options -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap are required to be set, even with versions post 1.8.0_191.

Can you set them and retest?

[zhashuyu]

I've try :

1. openjdk 1.8.0-242 with options -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap, no luck
2. oralce jdk 1.8.0-211 with options -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap, no luck

info

[boomer@boonix boomer]$ kubectl -n ae2f exec -it test-84b686f6ff-8strv /bin/bash
[root@test-84b686f6ff-8strv home]# curl -s http://test.k8s.local/metrics | grep java_lang_OperatingSystem_TotalPhysicalMemorySize
# HELP java_lang_OperatingSystem_TotalPhysicalMemorySize java.lang:name=null,type=OperatingSystem,attribute=TotalPhysicalMemorySize
# TYPE java_lang_OperatingSystem_TotalPhysicalMemorySize untyped
java_lang_OperatingSystem_TotalPhysicalMemorySize 3.3564966912E10
[root@test-84b686f6ff-8strv home]# grep -i memtotal /proc/meminfo
MemTotal:       32778288 kB
[root@test-84b686f6ff-8strv home]# cat /sys/fs/cgroup/memory/memory.limit_in_bytes
4294967296
[root@test-84b686f6ff-8strv home]# /opt/jdk1.8.0_211/bin/java -XX:+PrintFlagsFinal -version | grep -Ei "maxheapsize|maxram"
    uintx DefaultMaxRAMFraction                     = 4                                   {product}
    uintx MaxHeapSize                              := 1073741824                          {product}
 uint64_t MaxRAM                                    = 137438953472                        {pd product}
    uintx MaxRAMFraction                            = 4                                   {product}
   double MaxRAMPercentage                          = 25.000000                           {product}
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

[zhashuyu]

I've try openjdk11.0.6+10 ，still no luck.

info

[root@test-84b686f6ff-8strv home]# curl -s http://test.k8s.local/metrics | grep -i 'jvm_runtime_info\|java_lang_OperatingSystem_TotalPhysicalMemorySize'
java_lang_OperatingSystem_TotalPhysicalMemorySize 3.3564966912E10
jvm_runtime_info{runtime="OpenJDK Runtime Environment",vendor="Oracle Corporation",version="11.0.6+10-LTS"} 1

[dhoard]

The value is retrieved from the com.sun.management:type=OperatingSystem MBean and presented as returned.

This is a JVM issue that can't be resolved by the JMX Exporter.