Understanding Memory Allocation + Debug memory Leak

[XciD]

Details

We have migrated one of our top nodejs application from pm2 to containers + kubernetes.
This migration gave us way more information about our running code and we have fixe multiple memory leak related to dependencies.

After multiple iteration, we successfully have the heap quite stabilized but we seems to still having a leak.
A chart is worth a thousand words:

As you can see, the heap sounds normal, but the external memory have a good leak representation.

Node is running with --max-heap-size=768 option

What I dont understand is this error:

<--- Last few GCs --->

[1:0x69e3660] 45788793 ms: Mark-Compact (reduce) 716.8 (728.9) -> 716.8 (728.9) MB, 610.99 / 0.00 ms  (average mu = 0.474, current mu = 0.000) last resort; GC in old space requested
[1:0x69e3660] 45789396 ms: Mark-Compact (reduce) 716.8 (728.9) -> 716.8 (728.9) MB, 602.47 / 0.00 ms  (average mu = 0.305, current mu = 0.001) last resort; GC in old space requested


<--- JS stacktrace --->

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

If I understand the gc trace correctly, the heap is at 716MB, but why not on the graph ?
Also, how to find a leak on the external memory ?

Node.js version

v20.3.1

Example code

No response

Operating system

Linux 5.10.178-162.673.amzn2.x86_64 #1 SMP Mon Apr 24 23:34:06 UTC 2023 x86_64 GNU/Linux

Scope

runtime

Module and version

Not applicable.

[preveen-stack]

cc @nodejs/v8

[gireeshpunathil]

If I understand the gc trace correctly, the heap is at 716MB, but why not on the graph ?

you are right, the heap usage is not correctly reflected in the graph. which tool is that?

Also, how to find a leak on the external memory ?

it is hard. but this is a rough sketch of what you can do:

• take a heap dump
• load it in chrome dev tools
• navigate through all array buffers and shared array buffers in the heap
• find sizes of each and sum those up
• see if that tallies with the external memory size.
• if yes and you are not happy with that, walk through the dominator tree and see where those are allocated.
• if no, look out for presence of native modules and see what those are allocating.

if you want to take that path and have questions on any steps let me know!

[github-actions]

It seems there has been no activity on this issue for a while, and it is being closed in 30 days. If you believe this issue should remain open, please leave a comment.
If you need further assistance or have questions, you can also search for similar issues on Stack Overflow.
Make sure to look at the README file for the most updated links.

[hamza1022]

Just change the node version to 21.7.3 it will resolve your issue.Thanks me later .....!

[avivkeller]

This issue didn't get a response to @gireeshpunathil's suggestion, and would've stale closed. If you disagree with this, please reopen :-)