Oracle memory troubleshooting using analysis on heapdumps, part 2
In the article oracle memory troubleshooting using analysis on heap dumps I introduced heap_analyze.awk.
The reason the tool exists is because I am using it myself. Therefore, I ran into additional things that I wanted the tool to do. I added some stuff, which is that significant, that I decided to make another blogpost to introduce the new features.
1. Percentages
In order to get an idea of the relative size of the summarised topic, I added a percentage. For example:
top 5 allocation by total size per alloc reason per heap ================================================================================================== heap alloc_reason #chunks total_size % -------------------------------------------------------------------------------------------------- sga heap(1,0) perm 19 182947456 72 sga heap(1,0) 43 14488960 5 sga heap(1,0) SO private sga 18 14284168 5 sga heap(1,0) KGLHD 5787 4318400 1 sga heap(1,0) KSFD SGA I/O b 1 4190416 1
2. Enhanced perm (permanent) memory descriptions
It seems that for PGA heap dumps, sometimes there is a description for memory area’s that are perm (permanent memory, memory allocated for the lifetime of the process). This is how that’s visible in the dump:
PERMANENT CHUNKS:
Chunk 7fcbcad6c020 sz= 20632 perm "perm " alo=20632
7fcbcad6c020 sz= 20632 cpmlst "callback hsn "
I must say I don’t know what ‘cpmlst’ means, so if anyone knows or has a good guess, please let me know. However, the two addresses and sizes are an exact match, so I now change the alloc_reason for the cpmlst text.
This is helpful because there is a quite some memory allocated as perm. Sadly, this is not done for SGA dumps.
3. Excel mode
If you want to store and compare different dumps, one way of doing that is pasting the output in Microsoft excel. Once you set the ‘Text To Columns’ option to space as a separator, it will put the information in its own cells. But there are a few problems with that, one of them is that the heap names and alloc_reasons can have spaces in them, so that the placement of the figures can vary. I created excel mode for that.
If you set excel mode (set the variable excel_mode to 1 on line 5):
#!/bin/awk -f
BEGIN {
printf "Heap dump analyzer v0.2 by Frits Hoogland\n";
group_sum_nr=5;
excel_mode=1;
}
In this mode, the horizontal lines (with ‘-‘ and ‘=’) are omitted in the output, and all spaces are changed to underscores, so a table stays consistent when pasted in excel:
heap alloc_reason #chunks total_size % sga_heap(1,0) perm___________ 19 182947456 72 sga_heap(1,0) _______________ 43 14488960 5 sga_heap(1,0) SO_private_sga_ 18 14284168 5 sga_heap(1,0) KGLHD__________ 5787 4318400 1 sga_heap(1,0) KSFD_SGA_I/O_b_ 1 4190416 1
4. New table which shows memory by type
Another way of looking at memory in a heap is by grouping it by type. This allows you to very quickly see if a certain type of chunk is dominating a heap:
heap type #chunks min_size max_size total_size % -------------------------------------------------------------------------------------------------- top call heap free 7 16408 65512 390064 99 top call heap recreate 2 992 992 1984 0 top call heap perm 2 120 904 1024 0
This is an overview of the top call heap of a session that is not active, and therefore most of it should be empty, which is true for this dump.
Once again, get the awk script here: https://gitlab.com/FritsHoogland/oracle_memory_analyze/blob/master/heap_analyze.awk
Frits Hoogland Weblog 