Using a fast approach to see the heap
Sometimes taking a heap dump will be too long for the production environment, on this cases, it is possible to wait for the load to reduce and then take the heap dump.
However, it is possible to use the quick java diagnostics tool to see the some quick information. Basic usage is just run jcmd, and then see the processes running get the name or PID, then jcmd PID help. To see the operations available, in JDK 11+ there are so many operations that can be done, like jcmd PID VM.metaspace
If one does `GC.class_histogram`, which does not depend of +UnlockDiagnosticVMOptions, it can see the list of instances and retention based on the heap usage, example <— like seriously you don’t need to use it
GC.class_stats$ jdk-11.0.1/bin/jcmd 1568 GC.class_histogram
1568:
num #instances #bytes class name
1: 2725548 87217536 java.util.HashMap$Node
2: 101237 56724056 [B
3: 2706313 43301008 org.infinispan.server.some.Class <--- some clas takes 433k bytes, so then 43mb
4: 13443 17843296 [Ljava.util.HashMap$Node;
5: 21725 16866280 [Ljava.lang.Object;
6: 8658 5679648 io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue
7: 62530 5015664 [C
8: 79376 2540032 java.util.concurrent.ConcurrentHashMap$NodeSo we can see that the class `org.infinispan.server.some.Class` takes 2706313bytes, so 27mb of the heap, very easily.
This is a very powerful and pretty simple, you can use jcmd to get a heap dump with: `jcmd PID GC.heap_dump` but then you need to set a tool to analyse the Heap itself, like MAT.
Of course this quick usage is not for beginners, you need to know a bit of your application/stack so then one can see how it gets some data. But for a quick investigation it is pretty useful.
JCMD GC histogram
Just be careful with the histogram ( I will write a blog post specifically about it) but basically the histogram would only show memory directly used by those classes. Not all the other memory retained in their graph. So for instance, you’d see a lot of individual objects, but wouldn’t see that one particular collection is using 90% of the heap –> can lead to false positives. But it is really helpful for those scenarios that heap is not that possible and you have a clear knowledge of what you doing and can investigate the code if necessary. Histogram will perform Full GC operation btw.
The example below shows that, you think is the actual object that is taking 244 mb but there is a major object used by Infinispan/DataGrid for cache processing that holds it:
num #instances #bytes class name (module)
-------------------------------------------------------
1: 505686 856288432 [B (java.base@11.0.12)
2: 808426 806075760 [Ljava.lang.Object; (java.base@11.0.12)
4: 765492 24495744 io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache <-------- 244 mb
5: 333316 15999168 java.util.concurrent.locks.StampedLock (java.base@11.0.12)
6: 394052 15762080 java.util.WeakHashMap$Entry (java.base@11.0.12)
7: 138136 12155968 io.netty.buffer.PooledUnsafeDirectByteBuf
8: 309118 9891776 io.netty.util.Recycler$DefaultHandle
9: 386970 9287280 java.lang.String (java.base@11.0.12)
10: 115820 9264848 [Ljava.util.HashMap$Node; (java.base@11.0.12)
Total 7081360 2396918072 <------------------------------ 2.4 gbI will be presenting some quick jcmd usage in a conference: TDC. This is my second time presenting at the TDC and I’m very glad to share my knowledge on jcmd and live gc observations. In 2014 went to Florianopolis to present about EEG but now I will be presenting in Porto Alegre, RS, Brazil:
TDC 2020 Porto Alegre
I will add slides/presentation here and some extra comments right after, of course.
Update: I’ve been using more and more jcmd PID VM.info particularly for containers’ investigation context. It is really good and useful and it brings the native allocation as well. I will do a blog post about it.
Overhead de Melo 