This blogpost is about Oracle database and grid infrastructure software homes, which patches should be applied to which homes, and what it then looks like. This is fully documented by MyOracleSupport notes, but you will see that with version 18 and up this is unclear.

I keep a script-set that automatically installs and patches the Oracle database software and creates a database. This script-set is called vagrant-builder, and it can install any version with any PSU applied between 11.2.0.2 up to 19.5, which is the latest PSU of the latest version, with a few exceptions: for 11.2.0.2 and 11.2.0.3 I only created an install for the base version and the latest PSU for the database, and version 12.1.0.1 is left out entirely.

I recently reviewed my installs and verified everything is carried out correctly. First a simple overview of what I think should be applied on the database and grid infrastructure install:

Version  Grid               Database
-------- ------             ----------
11.2.0.2 -                  DB PSU
11.2.0.3 -                  DB PSU+OJVM
11.2.0.4 GI PSU+JDBC patch  DB PSU+OJVM
12.1.0.2 GI PSU+JDBC patch  DB PSU+OJVM
12.2.0.1 GI PSU             DB PSU+OJVM
18       GI PSU             DB PSU+OJVM
19       GI PSU             DB PSU+OJVM

(‘-‘ means not investigated)
My idea of what should be applied is based on MOS note 1929745.1: Oracle recommended patches.

Grid patches, JDBC patch
The JDBC patch for grid infrastructure 11.2.0.4 and 12.1.0.2 is a patch that updates java classes. Therefore the patch is a generic one, the java classes do not contain operating system dependent machine code. The table in the MOS note also show differences for the JDBC patch between = januari 2014 = july 2016 and higher. No surprise there. It’s especially easy because all versions in premier support as of the date of this blogpost do not need the JDBC patch.

Grid patches, GI PSU
For the GI PSU, there are a lot of patches that contain the GI PSU, because outside of the GI patch itself, there are also combination patches that for example contain both the GI and the DB PSU. I like to keep it as simple as I can. Therefore, I stick to MOS note 2118136.2: Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), bundle patches, Patchsets and Base releases, and look at the following:
– versions 11.2.0.4/12.1.0.2: Oracle Database PSU, SPU(CPU), Bundle Patches (Versions 12.1 & lower), version (11.2.0.4/12.1.0.2), GI PSU column.
– versions 12.2.0.1 and up: Oracle Database Updates, version (12.2.0.1,18.0.0.0,19.0.0.0), GI Update column.

Database patches, DB PSU
Here too there are multiple patches that can be used to apply the DB PSU, and I stick with MOS note 2118136.2: Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), bundle patches, Patchsets and Base releases, and look at the following:
– versions 11.2.0.4/12.1.0.2: Oracle Database PSU, SPU(CPU), Bundle Patches (Versions 12.1 & lower), version (11.2.0.4/12.1.0.2), PSU column.
– versions 12.2.0.1 and up: Oracle Database Updates, version (12.2.0.1,18.0.0.0,19.0.0.0), DB Update column.

Database patches, OJVM
There are multiple MOS documents talking about the database JavaVM patch, and there are multiple patches, but here I stick with MOS note 2118136.2: Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), bundle patches, Patchsets and Base releases once again, and look at the following:
– OJVM Update/PSU/Bundle Patches, 11.2.0.4/12.1.0.2/12.2.0.1/18.0.0.0/19.0.0.0, OJVM Update.

Okay. So we got the table above that is based on MOS note 1929745.1, and we got all the patches organised in MOS note 2118136.2. So that’s nice and simple, right?

Well, not entirely…

As I said, I was checking up on the latest PSU installs. This is the ‘opatch lspatches’ overview of 12.1.0.2 and 12.2.0.1:

-- 12.1.0.2
GI 190716
---------
23727148;
29509318;OCW PATCH SET UPDATE 12.1.0.2.190716 (29509318)
29494060;Database Patch Set Update : 12.1.0.2.190716 (29494060)
29423125;ACFS PATCH SET UPDATE 12.1.0.2.190716 (29423125)
26983807;WLM Patch Set Update: 12.1.0.2.180116 (26983807)
DB 190716
---------
29774383;Database PSU 12.1.0.2.190716, Oracle JavaVM Component (JUL2019)
29494060;Database Patch Set Update : 12.1.0.2.190716 (29494060)
-- 12.2.0.1
GI 191015
---------
30138470;Database Oct 2019 Release Update : 12.2.0.1.191015 (30138470)
30122828;ACFS OCT 2019 RELEASE UPDATE 12.2.0.1.0 (30122828)
30122814;OCW OCT 2019 RELEASE UPDATE 12.2.0.1.191015 (30122814)
30093408;TOMCAT RELEASE UPDATE 12.2.0.1.0(ID:190208.0920) (30093408)
26839277;DBWLM RELEASE UPDATE 12.2.0.1.0(ID:170913) (26839277
DB 191015
---------
30133625;OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)
30138470;Database Oct 2019 Release Update : 12.2.0.1.191015 (30138470)

For the database (DB), we see two patches, which is exactly what is expected.
– The database patch, which is called release update with one and patch set update with the other.
– The OJVM update which also named differently between the two versions, but very much recognisable as such.
I think it’s bad that the names vary, but this is totally expected.

For the grid infrastructure (GI), we see 5 patches in both situations, but these are not the same patches between the homes!
– The unnamed patch in the 12.1.0.2 home is the JDBC patch, which should only be applied to 12.1.0.2, not to higher versions.
– The OCW patch (oracle clusterware).
– The database patch.
– The ACFS patch. The version indication in the patch name changed.
– The WLM/DBWLM patch. Sadly the name changed, and the name with 12.2.0.1 is actually quite useless, I can’t tell the actual version, I have to look up the patch number.
– Starting from version 12.2.0.1, there is a tomcat installation in the grid home, as this patch indicates. The name here is not helpful because it doesn’t indicate the actual version, like with the DBWLM patch.

So, outside of in my opinion bad naming, and a weird inclusion of a competing product of Oracle (Tomcat versus Weblogic), this still follows the rules of logic.

Now let’s look at the same output for version 18 and 19:

--18
GI 18.8
30116128;ACFS RELEASE UPDATE 18.8.0.0.0 (30116128)
30113775;OCW RELEASE UPDATE 18.8.0.0.0 (30113775)
30112122;Database Release Update : 18.8.0.0.191015 (30112122)
30093398;TOMCAT RELEASE UPDATE 18.0.0.0.0 (30093398)
28655963;DBWLM RELEASE UPDATE 18.4.0.0.0 (28655963)
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
DB 18.8
30133603;OJVM RELEASE UPDATE: 18.8.0.0.191015 (30133603)
30112122;Database Release Update : 18.8.0.0.191015 (30112122)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
--19
GI 19.5
30125133;Database Release Update : 19.5.0.0.191015 (30125133)
30122167;ACFS RELEASE UPDATE 19.5.0.0.0 (30122167)
30122149;OCW RELEASE UPDATE 19.5.0.0.0 (30122149)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)
DB 19.5
30128191;OJVM RELEASE UPDATE: 19.5.0.0.191015 (30128191)
30125133;Database Release Update : 19.5.0.0.191015 (30125133)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

I must say that the naming in general looks more consistent, that is a good thing!

For the database (DB) we see THREE patches (I suspected 2):
– The database release update patch, which nicely reports its version.
– The OVM patch, which also reports its version.
– This is weird. One of the grid infrastructure patches, the “OCW” patch, is applied to the database home. In fact, this is applied to the base release. Because it’s not a patch that is documented to be needed to be applied, this will sit at this version and never be updated. However, this unexpected patch is consistently applied to the base release for both version 18 and 19.

For the grid home (GI) we see an inconsistent number of patches (!) between 18.8 and 19.5. This is partly as expected, but I found an weird patch applied too. Let’s go over the patches:
– The ACFS patch is totally expected and appropriately named.
– The OCW patch is expected and appropriately named too.
– The Database Release Update patch is expected and appropriately named.
– The TOMCAT patch is expected. I don’t understand why it can’t have the RU numbering, but at least it’s consistent between 18 and 19.
– The DBWLM patch is only applied to the 18 home. As far as I understand, this is how it’s supposed to be, DBWLM is not regularly updated like the other ones above, so it’s okay to have an older version of it with the other patches, and if there isn’t a patch to apply, it can simply not be there, like with version 19 in this case.
– The OJVM patch puzzles me. I don’t know what to think of it. Also, it’s only applied to the base release of version 18, not to the base release of version 19. This, very much like the OCW patch, will never be updated. But I just don’t understand, this patches the java virtual machine in the database, which for GI is the ASM instance, for which, as far as I know, the java virtual machine isn’t used.

Conclusion.
The naming of the patches as visible with “opatch lspatches” has certainly improved with version 18 and higher. Still it would be helpful if the grid infrastructure tomcat patch would follow the same naming of the other patches.

I am in doubts about the inclusion of two patches in the base releases of versions 18 and 19:
The OJVM patch inclusion in the base release of grid infrastructure of version 18 only.
The OCW patch inclusion in the base release of the database of version 18 and 19.

Addendum.
After debating this on twitter and with my colleagues, I found that my OCW assumptions were incorrect. The grid infrastructure patch versions 12.2 and up reasonably clearly describes that the OCW sub-patch that is part of the GI PSU/RU patch should be applied to the database home too (if cluster ware is used for that home). So that means that if you got another version of the database home than the grid infrastructure home and it is used with grid infrastructure, you should download the grid infrastructure PSU/RU patch and apply the OCW sub patch to the oracle database home, because the OCW patch is not in the database PSU/RU patch.
Because the database home patch itself is also in the grid infrastructure PSU/RU patch, I see no reason to bother downloading the database patch, and now only download and use the grid infrastructure PSU/RU patch, because that contains the database home patch as well as the OCW patch.

Addendum 2.
The OCW patch being installed into the grid infrastructure home and additionally in the database home turns out to be a change with PSU 11.2.0.4.5. Before that, the OCW patch did exist with the grid infrastructure home patch, but couldn’t be applied to the database home. This change was documented in the patch readme, but wasn’t really heavily marketed by Oracle.

Addendum 3.
Despite OCW being documented as being additionally applied to the database home for 11.2.0.4(.5) and higher, in my tests it was not possible to apply the 11.2.0.4 OCW patch to a database home for any 11.2.0.4 PSU; opatch fails with a dependency problem. It does succeed starting from 12.1.0.2 (12.1.0.1 not tested) and higher.

I have to say that when looking in the 11.2.0.4 OCW patch metadata, it says ‘rac’, so the OCW patch might succeed when the home is explicitly installed for a cluster database. I tested with grid infrastructure installed for a single machine (“siha”).

This blogpost is about an inconsistency I found in the X$ tables X$KQFTA and X$KQFCO. This is very specific. If you don’t care about that, you can skip this post.

The Oracle database’s “dynamic performance views” are views that are prefixed with “GV$” and “V$”. The “G” with “GV$” stands for “global” and gets you the results from its “V$” equivalent for all instances. In most cases, but that’s not the purpose of this blog. In most cases, the “V$” version simply is the “GV$” view with inst_id (instance id, used in RAC to specify the instance id) set to the current instance. I’ll refer to both simply as “V$” for simplicity.

The “V$” views are mostly build on top of “X$” tables. The “X$” tables are Oracle internal tables, and not officially supported, therefore you should use the “V$” views.

One of the things that certain “X$” tables expose is information about the “X$” tables and their columns theirselves. These “X$” tables are visible in “X$KQFTA” (kernel query fixed tables tables) for the “X$” tables, and “X$KQFCO” (kernel query fixed tables columns) for the columns of the “X$” tables. I learned about these tables a long time ago from Kyle Hailey, Anjo Kolk and James Morle regarding direct memory attachment for wait events, the reason being the ability to find the memory address for certain “X$” tables to fetch the data these hold directly. We are talking about the 1990s here. In fact, Kyle later put the history of how he learned about it in the “Oracle insights: tales of the oaktable” book.

I gather information about every PSU of every Oracle database version on linux once it comes out. This is how I am able to produce the posts about the difference between a current and a previous PSU. One of the things that I gather is information about X$ tables and their columns. In order to obtain the columns, I use X$KQFTA and X$KQFCO and join these in the way that is publicly known:

select t.kqftanam, c.kqfconam
from x$kqfta t, x$kqfco c
where t.inst_id = 1
and c.inst_id = 1
and t.indx = c.kqfcotab;

And this worked well…

…until I produced a difference report of versions 12.1.0.2.191015 and 12.1.0.2.190716, where the report reported the following difference:

x$ tables columns unique to 12.1.0.2.190716 versus 12.1.0.2.191015

NAME                                                                                                 COLUMN_NAME
---------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------------
X$KQFTVRTTST0                                                                                        ADDR
X$KQFTVRTTST0                                                                                        CHGROWID
X$KQFTVRTTST0                                                                                        CHGROWOP
X$KQFTVRTTST0                                                                                        COL0_0
X$KQFTVRTTST0                                                                                        COL0_1
X$KQFTVRTTST0                                                                                        COL0_2
X$KQFTVRTTST0                                                                                        COL0_3
X$KQFTVRTTST0                                                                                        COL0_4
...and so on...

This could happen, these are columns that seem to have vanished with 12.1.0.2.191015. However, I decided to check it, although I haven’t got access to the 12.1.0.2.191015 PSU, because it’s a PSU version only available for paid extended support.

I looked at the columns of X$KQFTVRTST0 in my current database, which happens to be an ancient 11.2.0.2 database. However, much to my surprise, this gave me radically different columns:

SYS@o112 AS SYSDBA> desc X$KQFTVRTTST0
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 COLA                                                                                                                       VARCHAR2(64)
 COLB                                                                                                                       NUMBER

Okay, something weird is going on here.

First let’s look in which versions X$KQFTVRTTST0 is present:

ORAVER@o112 > @x_table_in_versions.sql

DATABASE_VERSION     NAME
-------------------- --------------------------------------------------------------------------------------------------------------------------------
11.2.0.1.0           X$KQFTVRTTST0
11.2.0.2.0           X$KQFTVRTTST0
11.2.0.2.12          X$KQFTVRTTST0
11.2.0.3.0           X$KQFTVRTTST0
11.2.0.3.15          X$KQFTVRTTST0
11.2.0.4.0           X$KQFTVRTTST0
11.2.0.4.1           X$KQFTVRTTST0
11.2.0.4.160119      X$KQFTVRTTST0
11.2.0.4.160419      X$KQFTVRTTST0
11.2.0.4.160719      X$KQFTVRTTST0
11.2.0.4.161018      X$KQFTVRTTST0
11.2.0.4.170418      X$KQFTVRTTST0
11.2.0.4.170718      X$KQFTVRTTST0
11.2.0.4.170814      X$KQFTVRTTST0
11.2.0.4.180116      X$KQFTVRTTST0
11.2.0.4.180417      X$KQFTVRTTST0
11.2.0.4.180717      X$KQFTVRTTST0
11.2.0.4.190115      X$KQFTVRTTST0
11.2.0.4.191015      X$KQFTVRTTST0
11.2.0.4.2           X$KQFTVRTTST0
11.2.0.4.3           X$KQFTVRTTST0
11.2.0.4.4           X$KQFTVRTTST0
11.2.0.4.5           X$KQFTVRTTST0
11.2.0.4.6           X$KQFTVRTTST0
11.2.0.4.7           X$KQFTVRTTST0
11.2.0.4.8           X$KQFTVRTTST0
12.1.0.2.0           X$KQFTVRTTST0
12.1.0.2.1           X$KQFTVRTTST0
12.1.0.2.160119      X$KQFTVRTTST0
12.1.0.2.160419      X$KQFTVRTTST0
12.1.0.2.160719      X$KQFTVRTTST0
12.1.0.2.161018      X$KQFTVRTTST0
12.1.0.2.170117      X$KQFTVRTTST0
12.1.0.2.170418      X$KQFTVRTTST0
12.1.0.2.170718      X$KQFTVRTTST0
12.1.0.2.170814      X$KQFTVRTTST0
12.1.0.2.171017      X$KQFTVRTTST0
12.1.0.2.180116      X$KQFTVRTTST0
12.1.0.2.180417      X$KQFTVRTTST0
12.1.0.2.181016      X$KQFTVRTTST0
12.1.0.2.190115      X$KQFTVRTTST0
12.1.0.2.190416      X$KQFTVRTTST0
12.1.0.2.190716      X$KQFTVRTTST0
12.1.0.2.191015      X$KQFTVRTTST0
12.1.0.2.3           X$KQFTVRTTST0
12.1.0.2.4           X$KQFTVRTTST0
12.1.0.2.5           X$KQFTVRTTST0
12.2.0.1.0           X$KQFTVRTTST0
12.2.0.1.170620      X$KQFTVRTTST0
12.2.0.1.170718      X$KQFTVRTTST0
12.2.0.1.170814      X$KQFTVRTTST0
12.2.0.1.171017      X$KQFTVRTTST0
12.2.0.1.180116      X$KQFTVRTTST0
12.2.0.1.180417      X$KQFTVRTTST0
12.2.0.1.180717      X$KQFTVRTTST0
12.2.0.1.181016      X$KQFTVRTTST0
12.2.0.1.190115      X$KQFTVRTTST0
12.2.0.1.190416      X$KQFTVRTTST0
12.2.0.1.190716      X$KQFTVRTTST0
12.2.0.1.191015      X$KQFTVRTTST0
18.3                 X$KQFTVRTTST0
18.4                 X$KQFTVRTTST0
18.5                 X$KQFTVRTTST0
18.6                 X$KQFTVRTTST0
18.7                 X$KQFTVRTTST0
18.8                 X$KQFTVRTTST0
19.2                 X$KQFTVRTTST0
19.3                 X$KQFTVRTTST0
19.4                 X$KQFTVRTTST0
19.5                 X$KQFTVRTTST0

So, X$KQFVRTTST0 appeared with 11.2.0.1. Now let’s take a look at how the columns look like in X$KQFVRTTST0 looks in version 11.2.0.1:

ORAVER@o112 > @x_table_cols_in_versions.sql

DATABASE_VERSION	       TABLE_NAME		      COLUMN_NAME
------------------------------ ------------------------------ ----------------------------------------
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_0
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_0
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_0
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_0
11.2.0.1.0		       X$KQFTVRTTST0		      CHGROWOP
11.2.0.1.0		       X$KQFTVRTTST0		      CHGROWID
11.2.0.1.0		       X$KQFTVRTTST0		      COL_OPERATION
11.2.0.1.0		       X$KQFTVRTTST0		      INST_ID
11.2.0.1.0		       X$KQFTVRTTST0		      INDX
11.2.0.1.0		       X$KQFTVRTTST0		      ADDR

These are the columns which were shown in the difference report that said they were removed. Now let’s look at the columns in version 19.5:

ORAVER@o112 > @x_table_cols_in_versions.sql

DATABASE_VERSION               TABLE_NAME                     COLUMN_NAME
------------------------------ ------------------------------ ----------------------------------------
19.5                           X$KQFTVRTTST0                  COL3_9
19.5                           X$KQFTVRTTST0                  COL3_8
19.5                           X$KQFTVRTTST0                  COL3_7
19.5                           X$KQFTVRTTST0                  COL3_6
19.5                           X$KQFTVRTTST0                  COL3_5
19.5                           X$KQFTVRTTST0                  COL3_4
19.5                           X$KQFTVRTTST0                  COL3_3
19.5                           X$KQFTVRTTST0                  COL3_2
19.5                           X$KQFTVRTTST0                  COL3_1
19.5                           X$KQFTVRTTST0                  COL3_0
19.5                           X$KQFTVRTTST0                  COL2_9
19.5                           X$KQFTVRTTST0                  COL2_8
19.5                           X$KQFTVRTTST0                  COL2_7
19.5                           X$KQFTVRTTST0                  COL2_6
19.5                           X$KQFTVRTTST0                  COL2_5
19.5                           X$KQFTVRTTST0                  COL2_4
19.5                           X$KQFTVRTTST0                  COL2_3
19.5                           X$KQFTVRTTST0                  COL2_2
19.5                           X$KQFTVRTTST0                  COL2_1
19.5                           X$KQFTVRTTST0                  COL2_0
19.5                           X$KQFTVRTTST0                  COL1_9
19.5                           X$KQFTVRTTST0                  COL1_8
19.5                           X$KQFTVRTTST0                  COL1_7
19.5                           X$KQFTVRTTST0                  COL1_6
19.5                           X$KQFTVRTTST0                  COL1_5
19.5                           X$KQFTVRTTST0                  COL1_4
19.5                           X$KQFTVRTTST0                  COL1_3
19.5                           X$KQFTVRTTST0                  COL1_2
19.5                           X$KQFTVRTTST0                  COL1_1
19.5                           X$KQFTVRTTST0                  COL1_0
19.5                           X$KQFTVRTTST0                  COL0_9
19.5                           X$KQFTVRTTST0                  COL0_8
19.5                           X$KQFTVRTTST0                  COL0_7
19.5                           X$KQFTVRTTST0                  COL0_6
19.5                           X$KQFTVRTTST0                  COL0_5
19.5                           X$KQFTVRTTST0                  COL0_4
19.5                           X$KQFTVRTTST0                  COL0_3
19.5                           X$KQFTVRTTST0                  COL0_2
19.5                           X$KQFTVRTTST0                  COL0_1
19.5                           X$KQFTVRTTST0                  COL0_0
19.5                           X$KQFTVRTTST0                  CHGROWOP
19.5                           X$KQFTVRTTST0                  CHGROWID
19.5                           X$KQFTVRTTST0                  COL_OPERATION
19.5                           X$KQFTVRTTST0                  CON_ID
19.5                           X$KQFTVRTTST0                  INST_ID
19.5                           X$KQFTVRTTST0                  INDX
19.5                           X$KQFTVRTTST0                  ADDR

These are exactly the same! So the columns for X$KQFTVRTTST0 that I obtained from the database are the same for all versions in my data, but when I look in an actual database using ‘describe’ I see ADDR, INDX, INST_ID and COLA and COLB. So my data seems to be wrong!

My first reaction was to change the columns for X$KQFTVRTST0 for every version to correct this. But then I realised that I had to investigate this deeper to understand the actual problem, there might be more X$ tables involved.

The first thing to do, is to see if I can reproduce the issue in my database. This can be quite easily be done by describing X$KQFTVRTTST0 to see the columns, and then query these using the above query that demonstrates the join between X$KQFTA and X$KQFCO by adding a filter on ‘X$KQFTVRTST0’:

SYS@o112 AS SYSDBA> desc X$KQFTVRTTST0
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 COLA                                                                                                                       VARCHAR2(64)
 COLB                                                                                                                       NUMBER
SYS@o112 AS SYSDBA> l
  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and kqftanam = 'X$KQFTVRTTST0'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$KQFTVRTTST0                  ADDR
X$KQFTVRTTST0                  INDX
X$KQFTVRTTST0                  INST_ID
X$KQFTVRTTST0                  COL_OPERATION
X$KQFTVRTTST0                  CHGROWID
X$KQFTVRTTST0                  CHGROWOP
X$KQFTVRTTST0                  COL0_0
X$KQFTVRTTST0                  COL0_1
X$KQFTVRTTST0                  COL0_2
X$KQFTVRTTST0                  COL0_3
X$KQFTVRTTST0                  COL0_4
X$KQFTVRTTST0                  COL0_5
X$KQFTVRTTST0                  COL0_6
X$KQFTVRTTST0                  COL0_7
X$KQFTVRTTST0                  COL0_8
X$KQFTVRTTST0                  COL0_9
X$KQFTVRTTST0                  COL1_0
X$KQFTVRTTST0                  COL1_1
X$KQFTVRTTST0                  COL1_2
X$KQFTVRTTST0                  COL1_3
X$KQFTVRTTST0                  COL1_4
X$KQFTVRTTST0                  COL1_5
X$KQFTVRTTST0                  COL1_6
X$KQFTVRTTST0                  COL1_7
X$KQFTVRTTST0                  COL1_8
X$KQFTVRTTST0                  COL1_9
X$KQFTVRTTST0                  COL2_0
X$KQFTVRTTST0                  COL2_1
X$KQFTVRTTST0                  COL2_2
X$KQFTVRTTST0                  COL2_3
X$KQFTVRTTST0                  COL2_4
X$KQFTVRTTST0                  COL2_5
X$KQFTVRTTST0                  COL2_6
X$KQFTVRTTST0                  COL2_7
X$KQFTVRTTST0                  COL2_8
X$KQFTVRTTST0                  COL2_9
X$KQFTVRTTST0                  COL3_0
X$KQFTVRTTST0                  COL3_1
X$KQFTVRTTST0                  COL3_2
X$KQFTVRTTST0                  COL3_3
X$KQFTVRTTST0                  COL3_4
X$KQFTVRTTST0                  COL3_5
X$KQFTVRTTST0                  COL3_6
X$KQFTVRTTST0                  COL3_7
X$KQFTVRTTST0                  COL3_8
X$KQFTVRTTST0                  COL3_9

Bingo! Describe tells me about COLA and COLB, which is the actual situation, and querying X$KQFTA and X$KQFCO gives me different, wrong columns.

First let’s investigate if we can find the correct columns in X$KQFCO, and see what the difference is between X$KQFTA.INDX and X$KQFCO.KQFCOTAB:

SYS@o112 AS SYSDBA> select indx from x$kqfta where kqftanam = 'X$KQFTVRTTST0';

      INDX
----------
       841

SYS@o112 AS SYSDBA> select kqfcotab from x$kqfco where kqfconam = 'COLA';

  KQFCOTAB
----------
       842

Aha! So the join is off by one! So, that must mean that if I decrease KQFCOTAB by one, I do get the correct output:

SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab-1
  6* and kqftanam = 'X$KQFTVRTTST0'
  7  /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$KQFTVRTTST0                  ADDR
X$KQFTVRTTST0                  INDX
X$KQFTVRTTST0                  INST_ID
X$KQFTVRTTST0                  COLA
X$KQFTVRTTST0                  COLB

Bingo! The next thing to do is see if this is only a problem for X$KQFTVRTTST0, or a bigger problem. Let’s take the next table based on X$KQFTA.INDX, and see if this shows the same problem:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 842;

KQFTANAM
------------------------------
X$DIAG_ADR_CONTROL
SYS@o112 AS SYSDBA> desc X$DIAG_ADR_CONTROL
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 ADR_PATH_IDX                                                                                                               VARCHAR2(445)
 ADR_HOME                                                                                                                   VARCHAR2(445)
 ADRID                                                                                                                      NUMBER
 SHORTP_POLICY                                                                                                              NUMBER
 LONGP_POLICY                                                                                                               NUMBER
 LAST_MOD_TIME                                                                                                              TIMESTAMP(9) WITH TIME ZONE
 LAST_AUTOPRG_TIME                                                                                                          TIMESTAMP(9) WITH TIME ZONE
 LAST_MANUPRG_TIME                                                                                                          TIMESTAMP(9) WITH TIME ZONE
 ADRDIR_VERSION                                                                                                             NUMBER
 ADRSCHM_VERSION                                                                                                            NUMBER
 ADRSCHMV_SUMMARY                                                                                                           NUMBER
 ADRALERT_VERSION                                                                                                           NUMBER
 CREATE_TIME                                                                                                                TIMESTAMP(9) WITH TIME ZONE
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1   select t.kqftanam, c.kqfconam
  2   from x$kqfta t, x$kqfco c
  3   where t.inst_id = 1
  4   and c.inst_id = 1
  5   and t.indx = c.kqfcotab
  6*  and kqftanam = 'X$DIAG_ADR_CONTROL'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_ADR_CONTROL             ADDR
X$DIAG_ADR_CONTROL             INDX
X$DIAG_ADR_CONTROL             INST_ID
X$DIAG_ADR_CONTROL             COLA
X$DIAG_ADR_CONTROL             COLB

Clearly the columns are wrong, these are the columns of X$KQFTVRTTST0! Let’s subtract X$KQFCO.KQFCOTAB by one:

SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1   select t.kqftanam, c.kqfconam
  2   from x$kqfta t, x$kqfco c
  3   where t.inst_id = 1
  4   and c.inst_id = 1
  5   and t.indx = c.kqfcotab-1
  6*  and kqftanam = 'X$DIAG_ADR_CONTROL'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_ADR_CONTROL             ADDR
X$DIAG_ADR_CONTROL             INDX
X$DIAG_ADR_CONTROL             INST_ID
X$DIAG_ADR_CONTROL             ADR_PATH_IDX
X$DIAG_ADR_CONTROL             ADR_HOME
X$DIAG_ADR_CONTROL             ADRID
X$DIAG_ADR_CONTROL             SHORTP_POLICY
X$DIAG_ADR_CONTROL             LONGP_POLICY
X$DIAG_ADR_CONTROL             LAST_MOD_TIME
X$DIAG_ADR_CONTROL             LAST_AUTOPRG_TIME
X$DIAG_ADR_CONTROL             LAST_MANUPRG_TIME
X$DIAG_ADR_CONTROL             ADRDIR_VERSION
X$DIAG_ADR_CONTROL             ADRSCHM_VERSION
X$DIAG_ADR_CONTROL             ADRSCHMV_SUMMARY
X$DIAG_ADR_CONTROL             ADRALERT_VERSION
X$DIAG_ADR_CONTROL             CREATE_TIME

And this is correct again!

Now let’s verify the highest X$KQFTA.INDX number in the version of this database:

SYS@o112 AS SYSDBA> select max(indx) from x$kqfta;

 MAX(INDX)
----------
       927

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 927;

KQFTANAM
------------------------------
X$DIAG_VTEST_EXISTS

SYS@o112 AS SYSDBA> desc X$DIAG_VTEST_EXISTS
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 ADR_PATH_IDX                                                                                                               VARCHAR2(445)
 ADR_HOME                                                                                                                   VARCHAR2(445)
 PROBLEM_ID                                                                                                                 NUMBER
 PROBLEM_KEY                                                                                                                VARCHAR2(552)
 FIRST_INCIDENT                                                                                                             NUMBER
 FIRSTINC_TIME                                                                                                              TIMESTAMP(9) WITH TIME ZONE
 LAST_INCIDENT                                                                                                              NUMBER
 LASTINC_TIME                                                                                                               TIMESTAMP(9) WITH TIME ZONE
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and kqftanam = 'X$DIAG_VTEST_EXISTS'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_VTEST_EXISTS            ADDR
X$DIAG_VTEST_EXISTS            INDX
X$DIAG_VTEST_EXISTS            INST_ID
X$DIAG_VTEST_EXISTS            ADR_PATH_IDX
X$DIAG_VTEST_EXISTS            ADR_HOME
X$DIAG_VTEST_EXISTS            INCIDENT_ID
X$DIAG_VTEST_EXISTS            PROBLEM_ID
X$DIAG_VTEST_EXISTS            CREATE_TIME
X$DIAG_VTEST_EXISTS            CLOSE_TIME
X$DIAG_VTEST_EXISTS            STATUS
X$DIAG_VTEST_EXISTS            FLAGS
X$DIAG_VTEST_EXISTS            FLOOD_CONTROLLED
X$DIAG_VTEST_EXISTS            ERROR_FACILITY
X$DIAG_VTEST_EXISTS            ERROR_NUMBER
X$DIAG_VTEST_EXISTS            ERROR_ARG1
X$DIAG_VTEST_EXISTS            ERROR_ARG2
X$DIAG_VTEST_EXISTS            ERROR_ARG3
X$DIAG_VTEST_EXISTS            ERROR_ARG4
X$DIAG_VTEST_EXISTS            ERROR_ARG5
X$DIAG_VTEST_EXISTS            ERROR_ARG6
X$DIAG_VTEST_EXISTS            ERROR_ARG7
X$DIAG_VTEST_EXISTS            ERROR_ARG8
X$DIAG_VTEST_EXISTS            SIGNALLING_COMPONENT
X$DIAG_VTEST_EXISTS            SIGNALLING_SUBCOMPONENT
X$DIAG_VTEST_EXISTS            SUSPECT_COMPONENT
X$DIAG_VTEST_EXISTS            SUSPECT_SUBCOMPONENT
X$DIAG_VTEST_EXISTS            ECID
X$DIAG_VTEST_EXISTS            IMPACT
X$DIAG_VTEST_EXISTS            IMPACT_STR
X$DIAG_VTEST_EXISTS            STATUS_STR
X$DIAG_VTEST_EXISTS            ERROR_ARG9
X$DIAG_VTEST_EXISTS            ERROR_ARG10
X$DIAG_VTEST_EXISTS            ERROR_ARG11
X$DIAG_VTEST_EXISTS            ERROR_ARG12

So that’s wrong, let’s try subtracting by one:

SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab-1
  6* and kqftanam = 'X$DIAG_VTEST_EXISTS'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_VTEST_EXISTS            ADDR
X$DIAG_VTEST_EXISTS            INDX
X$DIAG_VTEST_EXISTS            INST_ID
X$DIAG_VTEST_EXISTS            ADR_PATH_IDX
X$DIAG_VTEST_EXISTS            ADR_HOME
X$DIAG_VTEST_EXISTS            PROBLEM_ID
X$DIAG_VTEST_EXISTS            PROBLEM_KEY
X$DIAG_VTEST_EXISTS            FIRST_INCIDENT
X$DIAG_VTEST_EXISTS            FIRSTINC_TIME
X$DIAG_VTEST_EXISTS            LAST_INCIDENT
X$DIAG_VTEST_EXISTS            LASTINC_TIME

And correct again!

So at least starting from X$KQFTA.INDX = 841 up to the highest INDX number, 927, X$KQFCO.KQFCOTAB is off by one.

Starting from there, I decided to test INDX = 1, which was consistent with KQFCOTAB. Then I decided to go down in INDX number starting from 841 being the number of X$KQFTVRTTST0. In my database number 840 is X$KTCNQROW, which was off by one, then I found something weird:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 839;

KQFTANAM
------------------------------
X$SKGXP_MISC

SYS@o112 AS SYSDBA> desc X$SKGXP_MISC
ERROR:
ORA-00942: table or view does not exist
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1      select t.kqftanam, c.kqfconam
  2      from x$kqfta t, x$kqfco c
  3      where t.inst_id = 1
  4      and c.inst_id = 1
  5      and t.indx = c.kqfcotab
  6*    and kqftanam = 'X$SKGXP_MISC'
SYS@o112 AS SYSDBA> /

no rows selected

So the X$ view X$SKGXP_MISC exists in X$KQFTA, but doesn’t exist in reality, nor in X$KQFCO. Please mind this is a separate, different issue from the X$KQFCO.KQFCOTAB inconsistency this blogpost is about. My current thinking is the issue of existing in X$KQFTA and not queryable and (rightfully because of that), not available in X$KQFCO, might be either just a screwup/bug, or the X$ tables being reserved in the X$ table array, but not actually created, maybe because of setting during compilation. Just a hunch, I don’t know.

The same issue of being visible in X$KQFTA and not existing is true for:
838, X$SKGXP_CONNECTION
837, X$SKGXP_PORT
836, X$KSXP_STATS

INDX number 835 is X$KSXPTESTTBL. This table has the X$KQFCO.KQFCOTAB off by one issue:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 835;

KQFTANAM
------------------------------
X$KSXPTESTTBL
SYS@o112 AS SYSDBA> desc X$KSXPTESTTBL
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 KSXPTESTF1                                                                                                                 NUMBER
 KSXPTESTF2                                                                                                                 VARCHAR2(10)
 KSXPTESTF3                                                                                                                 NUMBER
 KSXPTESTF4                                                                                                                 NUMBER
 KSXPTESTF5                                                                                                                 RAW(8)
 KSXPTESTF6                                                                                                                 DATE
 KSXPTESTF7                                                                                                                 TIMESTAMP(3)
 KSXPTESTF8                                                                                                                 TIMESTAMP(3) WITH TIME ZONE
 KSXPTESTF9                                                                                                                 VARCHAR2(10)
 KSXPTESTF10                                                                                                                RAW(2)
 KSXPTESTF11                                                                                                                RAW(10)
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and t.kqftanam = 'X$KSXPTESTTBL'
SYS@o112 AS SYSDBA> /

no rows selected
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab-1
  6* and t.kqftanam = 'X$KSXPTESTTBL'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$KSXPTESTTBL                  ADDR
X$KSXPTESTTBL                  INDX
X$KSXPTESTTBL                  INST_ID
X$KSXPTESTTBL                  KSXPTESTF1
X$KSXPTESTTBL                  KSXPTESTF2
X$KSXPTESTTBL                  KSXPTESTF3
X$KSXPTESTTBL                  KSXPTESTF4
X$KSXPTESTTBL                  KSXPTESTF5
X$KSXPTESTTBL                  KSXPTESTF6
X$KSXPTESTTBL                  KSXPTESTF7
X$KSXPTESTTBL                  KSXPTESTF8
X$KSXPTESTTBL                  KSXPTESTF9
X$KSXPTESTTBL                  KSXPTESTF10
X$KSXPTESTTBL                  KSXPTESTF11

After X$KSXPTESTTBL, INDX number 834 is X$CELL_NAME:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 834;

KQFTANAM
------------------------------
X$CELL_NAME
SYS@o112 AS SYSDBA> desc X$CELL_NAME
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 CELLPATH_CELLROW                                                                                                           VARCHAR2(400)
 CELLHASHVAL_CELLROW                                                                                                        NUMBER
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and t.kqftanam = 'X$CELL_NAME'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$CELL_NAME                    ADDR
X$CELL_NAME                    INDX
X$CELL_NAME                    INST_ID
X$CELL_NAME                    CELLPATH_CELLROW
X$CELL_NAME                    CELLHASHVAL_CELLROW

And with X$CELL_NAME the X$KQFTA.INDX=X$KQFCO.KQFCOTAB link is restored.

So, this means that starting with X$KSXPTESTTBL the X$KQFCO.KQFCOTAB number is off by one. In fact, I tested version 18 (18.4), and found that the value X$KQFTA.INDX is 1217, so much higher, but still with X$KSXPTESTTBL the inconsistency of X$KQFCO.KQFCOTAB being one too high to correctly join with X$KQFTA.INDX.

Conclusion.
This means that based on my current investigations, it seems that starting from Oracle database version 11.2.0.1 up to version 19 and current PSU, starting with the X$KQFTA.INDX value of the table X$KSXPTESTTBL in X$KQFTA, the X$KQFCO.KQFCOTAB value needs to be subtracted by one to get the correct fields from X$KQFCO.KQFCOTAB to join to X$KQFTA.INDX.

A lot of X$ tables are not impacted, the majority of the tables have an X$KQFTA.INDX value lower than X$KSXPTESTTBL, but there are a reasonable amount of tables that are impacted, a lot of them related to ADR (starting with X$DIAG).

The reason for showing an additional oddity with X$SKGXP_MISC, X$SKGXP_CONNECTION, X$SKGXP_PORT and X$KSXP_STATS is that the X$ table where the join problem starts is X$KSXPTESTTBL. All these tables have something to do with cross instance IPC communication (skgxp/ksxp), and a lot of them have the weird property that they don’t exist whilst are visible in X$KQFTA, this might have something to do with X$KQFCO.KQFCOTAB being off by one. Of course this is something I can’t check, because it requires source code access, but I thought it would be good to mention.

It’s interesting to see that the join of X$KQFTA.INDX to X$KQFCO.KQFCOTAB is present in Oracle’s data dictionary with the view V$INDEXED_FIXED_COLUMN:

SQL> select view_definition from v$fixed_view_definition where view_name = 'GV$INDEXED_FIXED_COLUMN';

VIEW_DEFINITION
--------------------------------------------------------------------------------
select c.inst_id,kqftanam, kqfcoidx, kqfconam, kqfcoipo, c.con_id  from x$kqfco
c, x$kqfta t where t.indx = c.kqfcotab and kqfcoidx != 0

Which by now should be clear is not correct for X$ tables with an X$KQFTA.INDX number equal or higher than X$KSXPTESTTBL.

The amount of changes between version 18.7 and 18.8 is truly minimal. There’s one spare parameter renamed to an underscore parameter, which seems to be a back port of functionality created version 19:

DATABASE_VERSION                                                                                     NAME
---------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------
18.8                                                                                                 _ldap_password_oneway_auth
19.2                                                                                                 _ldap_password_oneway_auth
19.3                                                                                                 _ldap_password_oneway_auth
19.4                                                                                                 _ldap_password_oneway_auth
19.5                                                                                                 _ldap_password_oneway_auth

And it seems the DBA view DBA_REGISTRY_BACKPORTS is corrupted with 18.8, there only is a synonym with that name, which throws an ORA 1775: looping chain of synonyms when used. This is the reason it shows as gone in 18.8 (unique to version 18.7).

parameters unique in version 18.7 versus 18.8

NAME
--------------------------------------------------
_one-hundred-and-forty-sixth_spare_parameter

parameters unique in version 18.8 versus 18.7

NAME
--------------------------------------------------
_ldap_password_oneway_auth

parameter values changed isdefault between 18.7 versus 18.8

parameter values unique to 18.7 versus 18.8

parameter values unique to 18.8 versus 18.7

waitevents unique in version 18.7 versus 18.8

waitevents unique in version 18.8 versus 18.7

waitevents changed parameter description between 18.7 versus 18.8

x$ tables unique to 18.7 versus 18.8

x$ tables unique to 18.8 versus 18.7

x$ tables columns unique to 18.7 versus 18.8

x$ tables columns unique to 18.8 versus 18.7

v$ tables unique to 18.7 versus 18.8

v$ tables unique to 18.8 versus 18.7

v$ tables columns unique to 18.7 versus 18.8

v$ tables columns unique to 18.8 versus 18.7

gv$ tables unique to 18.7 versus 18.8

gv$ tables unique to 18.8 versus 18.7

gv$ tables columns unique to 18.7 versus 18.8

gv$ tables columns unique to 18.8 versus 18.7

sysstat statistics unique to 18.7 versus 18.8

sysstat statistics unique to 18.8 versus 18.7

sys_time_model statistics unique to 18.7 versus 18.8

sys_time_model statistics unique to 18.8 versus 18.7

dba tables unique to 18.7 versus 18.8

NAME
----------------------------------------------------------------------------------------------------
DBA_REGISTRY_BACKPORTS

dba tables unique to 18.8 versus 18.7

dba tables columns unique to 18.8 versus 18.7

dba tables columns unique to 18.7 versus 18.8

cdb tables unique to 18.7 versus 18.8

NAME
----------------------------------------------------------------------------------------------------
CDB_REGISTRY_BACKPORTS

cdb tables unique to 18.8 versus 18.7

cdb tables column unique to 18.7 versus 18.8

cdb tables column unique to 18.8 versus 18.7

And here are the differences in symbols (c functions).
The amount of functions added and removed is not huge this time (remember last time littecms was added).
The same java/domain functions as seen with version 19 have been removed, and the same PLSQL codegen layer functions, plus a singe DBMS_SODA function. There are some other functions that do not align with 19 removed.

The same batch of DBMS_SODA related functions are added, plus a lot of functions different than version 19.

code symbol names unique in version 18.7 versus 18.8

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
joncl_at_domains                                             (jon)cl_at_domains                                           java ??
joncl_domains                                                (jon)cl_domains                                              java ??
joncl_set_domains                                            (jon)cl_set_domains                                          java ??
joncl_snor_domains                                           (jon)cl_snor_domains                                         java ??
joniul_at_domains                                            (jon)iul_at_domains                                          java ??
joniul_domains                                               (jon)iul_domains                                             java ??
joniul_set_domains                                           (jon)iul_set_domains                                         java ??
joniul_snor_domains                                          (jon)iul_snor_domains                                        java ??
jonndl_at_domains                                            (jon)ndl_at_domains                                          java ??
jonndl_domains                                               (jon)ndl_domains                                             java ??
jonndl_set_domains                                           (jon)ndl_set_domains                                         java ??
jonndl_snor_domains                                          (jon)ndl_snor_domains                                        java ??
jonoce_at_domains                                            (jon)oce_at_domains                                          java ??
jonoce_domains                                               (jon)oce_domains                                             java ??
jonoce_set_domains                                           (jon)oce_set_domains                                         java ??
jonoce_snor_domains                                          (jon)oce_snor_domains                                        java ??
jonosd_at_domains                                            (jon)osd_at_domains                                          java ??
jonosd_domains                                               (jon)osd_domains                                             java ??
jonosd_set_domains                                           (jon)osd_set_domains                                         java ??
jonosd_snor_domains                                          (jon)osd_snor_domains                                        java ??
jonvcl_at_domains                                            (jon)vcl_at_domains                                          java ??
jonvcl_domains                                               (jon)vcl_domains                                             java ??
jonvcl_set_domains                                           (jon)vcl_set_domains                                         java ??
jonvcl_snor_domains                                          (jon)vcl_snor_domains                                        java ??
kestsInitTCB                                                 (kest)sInitTCB                                               kernel event SQL manageability tokenize (?) ??
kglsim_get_bucket_info                                       (kglsim)_get_bucket_info                                     kernel generic library cache management simulator (library cache advise)  ??
kgskltyp                                                     (kgsk)ltyp                                                   kernel generic service resource manager ??
krvxdsr                                                      (krvx)dsr                                                    kernel redo recovery extract ??
pdzdM60_Make_RBST                                            (pdz)dM60_Make_RBST                                          PLSQL code execution codegen infrastructure ??
pdzdM66_Insert_At_Root                                       (pdz)dM66_Insert_At_Root                                     PLSQL code execution codegen infrastructure ??
qcsoPreNameres_ProcOpt                                       (qcso)PreNameres_ProcOpt                                     query compile semantic analysis (parser) OBJECTS ??
qsodasqlGetInsertOneSQL                                      (qsoda)sqlGetInsertOneSQL                                    query system DBMS_SODA ??

code symbol names unique in version 18.8 versus 18.7

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
audIsSenCol                                                  (aud)IsSenCol                                                audit ??
audOptTab                                                    (aud)OptTab                                                  audit ??
audstwlk                                                     (aud)stwlk                                                   audit ??
kcfis_clear_lob                                              (kcfis)_clear_lob                                            kernel cache file management intelligent storage ??
kclcsb                                                       (kcl)csb                                                     kernel cache lock manager/buffer cache ??
kds_update_turbo_scan_pivot_statistics                       (kds)_update_turbo_scan_pivot_statistics                     kernel data seek/scan ??
kesdtcbResetDiagEvent                                        (kes)dtcbResetDiagEvent                                      kernel event SQL manageability ??
kfdDskComputeHashForRow                                      (kfd)DskComputeHashForRow                                    kernel automatic storage management disk ??
kfdp_chkSufOlnFGAvl                                          (kfdp)_chkSufOlnFGAvl                                        kernel automatic storage management disk PST ??
kfioSrMsgBuf_ack                                             (kfio)SrMsgBuf_ack                                           kernel automatic storage management translation I/O layer ??
kfkcrRefresh                                                 (kfk)crRefresh                                               kernel automatic storage management KFK ??
kfnFreeKfnpnmMem                                             (kfn)FreeKfnpnmMem                                           kernel automatic storage management networking subsystem ??
kgfvTermSkvol                                                (kgf)vTermSkvol                                              kernel generic ASM ??
kjznhcdn                                                     (kjzn)hcdn                                                   kernel lock management DIAG node layer ??
kjznhcds                                                     (kjzn)hcds                                                   kernel lock management DIAG node layer ??
kkopinlpInt                                                  (kko)pinlpInt                                                kernel compile optimizer ??
kkquIsCorrTransInList                                        (kkqu)IsCorrTransInList                                      kernel compile query  subquery unnesting ??
knlpmaxfmsgsize                                              (knl)pmaxfmsgsize                                            kernel replication streams replication ??
kpuxsoCurPrefetchFree                                        (kpu)xsoCurPrefetchFree                                      kernel programmatic interface user ??
kpuxsoDocContentSet                                          (kpu)xsoDocContentSet                                        kernel programmatic interface user ??
kpuxsoOperInit                                               (kpu)xsoOperInit                                             kernel programmatic interface user ??
krb_devtype_chk                                              (krb)_devtype_chk                                            kernel redo backup/restore ??
krvxgtf                                                      (krvx)gtf                                                    kernel redo recovery extract ??
krvxrsr_SkipRedo                                             (krvx)rsr_SkipRedo                                           kernel redo recovery extract ??
kzan_set_ddl_lock_timeout                                    (kza)n_set_ddl_lock_timeout                                  kernel security audit  ??
kzckmWalletInArchive                                         (kzck)mWalletInArchive                                       kernel security Code-Based Access Control encryption support ??
kzckmrOpenRootOnPlug                                         (kzck)mrOpenRootOnPlug                                       kernel security Code-Based Access Control encryption support ??
kzckmrXmlVldTDEKeys                                          (kzck)mrXmlVldTDEKeys                                        kernel security Code-Based Access Control encryption support ??
kzckmrXmlVldTDEKeysFile                                      (kzck)mrXmlVldTDEKeysFile                                    kernel security Code-Based Access Control encryption support ??
kzckmrXmlVldTDEMetaHSM                                       (kzck)mrXmlVldTDEMetaHSM                                     kernel security Code-Based Access Control encryption support ??
pfrcrt                                                       (pfr)crt                                                     PLSQL Code Execution runtime ??
qsodasqlGetContentDefine                                     (qsoda)sqlGetContentDefine                                   query system DBMS_SODA ??
qsodasqlGetCreatedOnDefine                                   (qsoda)sqlGetCreatedOnDefine                                 query system DBMS_SODA ??
qsodasqlGetInsertSQL                                         (qsoda)sqlGetInsertSQL                                       query system DBMS_SODA ??
qsodasqlGetKeyDefine                                         (qsoda)sqlGetKeyDefine                                       query system DBMS_SODA ??
qsodasqlGetLastModDefine                                     (qsoda)sqlGetLastModDefine                                   query system DBMS_SODA ??
qsodasqlGetMediaDefine                                       (qsoda)sqlGetMediaDefine                                     query system DBMS_SODA ??
qsodasqlGetVersionDefine                                     (qsoda)sqlGetVersionDefine                                   query system DBMS_SODA ??
qsodasqlPrefetchPrepare                                      (qsoda)sqlPrefetchPrepare                                    query system DBMS_SODA ??
qsodasqlSelectDocFieldsSQL                                   (qsoda)sqlSelectDocFieldsSQL                                 query system DBMS_SODA ??
qsodaxArrayFetchDocuments                                    (qsoda)xArrayFetchDocuments                                  query system DBMS_SODA ??
qsodaxDefineDocumentFields                                   (qsoda)xDefineDocumentFields                                 query system DBMS_SODA ??
qsodaxGetPrefetchDocument                                    (qsoda)xGetPrefetchDocument                                  query system DBMS_SODA ??
slmeset_s                                                    (sl)meset_s                                                  operating system (dependent) library ??

It seems that the most eye-catching difference between Oracle database versions 19.5 and 19.4 is three underscore parameters spare parameters being changed to specifically named underscore parameters, two of them have a bug number in them which is not publicly visible.

In v$sysstat/v$sesstat, a group of statistics are renamed from ‘OS %’ to ‘Server %’. All these statistics are about networking. One changed parameter directly points to networking (tcpinfo). These statistics are added in version 19.

One DBA view was added: DBA_DV_COMMON_OPERATION_STATUS. This view is specific and owned by to DVSYS.

NAME
--------------------------------------------------
_fourth_spare_parameter
_one-hundred-and-fifty-third_spare_parameter
_one-hundred-and-forty-sixth_spare_parameter

parameters unique in version 19.5 versus 19.4

NAME
--------------------------------------------------
_bug29302220_tcpinfo_statistics_save_atexit
_bug29903454_ksws_enable_alb
_session_modp_list

parameter values changed isdefault between 19.4 versus 19.5

parameter values unique to 19.4 versus 19.5

parameter values unique to 19.5 versus 19.4

waitevents unique in version 19.4 versus 19.5

waitevents unique in version 19.5 versus 19.4

waitevents changed parameter description between 19.4 versus 19.5

x$ tables unique to 19.4 versus 19.5

x$ tables unique to 19.5 versus 19.4

x$ tables columns unique to 19.4 versus 19.5

x$ tables columns unique to 19.5 versus 19.4

v$ tables unique to 19.4 versus 19.5

v$ tables unique to 19.5 versus 19.4

v$ tables columns unique to 19.4 versus 19.5

v$ tables columns unique to 19.5 versus 19.4

gv$ tables unique to 19.4 versus 19.5

gv$ tables unique to 19.5 versus 19.4

gv$ tables columns unique to 19.4 versus 19.5

gv$ tables columns unique to 19.5 versus 19.4

sysstat statistics unique to 19.4 versus 19.5

NAME
----------------------------------------------------------------------------------------------------
OS Advertised Receive Window
OS Advertised Send Window
OS Data Segments In
OS Data Segments Out
OS Lost Packets
OS Path Maximum Transmission Unit(MTU)
OS Send Congestion Window
OS Time (usec) Busy Sending Data
OS Time (usec) Busy Sending Data under Congestion
OS Time (usec) Last Ack Received
OS Time (usec) Last Ack Sent
OS Time (usec) Last Data Received
OS Time (usec) Last Data Sent
OS Time (usec) Limited by Receive Window
OS Time (usec) Limited by Send Buffer
OS Time (usec) Round Trip Time
OS Time (usec) Round Trip Time Variance
OS Total Bytes Acked
OS Total Bytes Received
OS Total Number of Retransmitted Packets

sysstat statistics unique to 19.5 versus 19.4

NAME
----------------------------------------------------------------------------------------------------
Server Advertised Receive Window
Server Advertised Send Window
Server Data Segments In
Server Data Segments Out
Server Elapsed Time (msec) Last Ack Received
Server Elapsed Time (msec) Last Ack Sent
Server Elapsed Time (msec) Last Data Received
Server Elapsed Time (msec) Last Data Sent
Server Lost Packets
Server Path Maximum Transmission Unit(MTU)
Server Send Congestion Window
Server Time (usec) Approximate Elapsed Time from the packet dro
Server Time (usec) Busy Sending Data
Server Time (usec) Limited by Receive Window
Server Time (usec) Limited by Send Buffer
Server Time (usec) Round Trip Time
Server Time (usec) Round Trip Time Variance
Server Total Bytes Acked
Server Total Bytes Received
Server Total Number of Retransmitted Packets

sys_time_model statistics unique to 19.4 versus 19.5

sys_time_model statistics unique to 19.5 versus 19.4

dba tables unique to 19.4 versus 19.5

dba tables unique to 19.5 versus 19.4

NAME
----------------------------------------------------------------------------------------------------
DBA_DV_COMMON_OPERATION_STATUS

dba tables columns unique to 19.5 versus 19.4

dba tables columns unique to 19.4 versus 19.5

cdb tables unique to 19.4 versus 19.5

cdb tables unique to 19.5 versus 19.4

cdb tables column unique to 19.4 versus 19.5

cdb tables column unique to 19.5 versus 19.4

And here are the differences in symbols (c functions).
The functions unique to 19.4 versus 19.5 (alias removed) are partly jon thus java related functions, which all have ‘domain’ in the name. Plus some diverse functions.

There is a significant amount number of functions added. The first ones starting with ‘aux_’ and ‘decode_krb5’ might have something to do with ‘krb5’, alias Kerberos version 5.

There is a ‘dbnest_dir_check’ function in between ‘aux’ and ‘decode’. It seems ‘dbnest’ is an entire layer in the database code at least, however I do not know if it relates to something that is public knowledge.

Among all kinds of singular added functions, what further is noticeable, is a high number of ASM related functions being added, of which a lot of them refer to ‘oss’ (oracle storage server?).

Additionally, a bunch of ‘ksp_modp’ functions are added which might have something to do with the ‘_session_modp_list’ parameter, and a bunch of parameters starting with ‘ksws_alb’ which might have something to do with the ‘_bug29903454_ksws_enable_alb’ parameter.

Further a group of functions starting with ‘soda’, which might be related to DBMS_SODA: a package implementing Simple Oracle Document Access (SODA), for managing document collections using PL/SQL.

code symbol names unique in version 19.4 versus 19.5

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
drexogidx                                                    (drex)ogidx                                                  text index query ??
joncl_at_domains                                             (jon)cl_at_domains                                           java ??
joncl_domains                                                (jon)cl_domains                                              java ??
joncl_set_domains                                            (jon)cl_set_domains                                          java ??
joncl_snor_domains                                           (jon)cl_snor_domains                                         java ??
joniul_at_domains                                            (jon)iul_at_domains                                          java ??
joniul_domains                                               (jon)iul_domains                                             java ??
joniul_set_domains                                           (jon)iul_set_domains                                         java ??
joniul_snor_domains                                          (jon)iul_snor_domains                                        java ??
jonndl_at_domains                                            (jon)ndl_at_domains                                          java ??
jonndl_domains                                               (jon)ndl_domains                                             java ??
jonndl_set_domains                                           (jon)ndl_set_domains                                         java ??
jonndl_snor_domains                                          (jon)ndl_snor_domains                                        java ??
jonoce_at_domains                                            (jon)oce_at_domains                                          java ??
jonoce_domains                                               (jon)oce_domains                                             java ??
jonoce_set_domains                                           (jon)oce_set_domains                                         java ??
jonoce_snor_domains                                          (jon)oce_snor_domains                                        java ??
jonosd_at_domains                                            (jon)osd_at_domains                                          java ??
jonosd_domains                                               (jon)osd_domains                                             java ??
jonosd_set_domains                                           (jon)osd_set_domains                                         java ??
jonosd_snor_domains                                          (jon)osd_snor_domains                                        java ??
jonvcl_at_domains                                            (jon)vcl_at_domains                                          java ??
jonvcl_domains                                               (jon)vcl_domains                                             java ??
jonvcl_set_domains                                           (jon)vcl_set_domains                                         java ??
jonvcl_snor_domains                                          (jon)vcl_snor_domains                                        java ??
jsv_try_scan_3choice                                         (js)v_try_scan_3choice                                       job scheduing ??
kcbtse_raise_tbs_datafile_enc_mismatch_error                 (kcbtse)_raise_tbs_datafile_enc_mismatch_error               kernel cache buffers tablespace encryption ??
kffgAttrMaxPxCb                                              (kffg)AttrMaxPxCb                                            kernel automatic storage management KFF file group ??
kffgVolClientCheck                                           (kffg)VolClientCheck                                         kernel automatic storage management KFF file group ??
kgiflini                                                     (kgi)flini                                                   kernel generic instantiation manager ??
kgiflpop                                                     (kgi)flpop                                                   kernel generic instantiation manager ??
kjgcr_GetTopCPU                                              (kj)gcr_GetTopCPU                                            kernel lock management ??
kjgcr_RunSyncTask                                            (kj)gcr_RunSyncTask                                          kernel lock management ??
kjgcr_SlaveReqGetSlot                                        (kj)gcr_SlaveReqGetSlot                                      kernel lock management ??
kkdlpGetObjn                                                 (kkdl)pGetObjn                                               kernel compile dictionary lookup ??
kkjcjchk                                                     (kkj)cjchk                                                   kernel compile job queue ??
kkoarFreeCtx                                                 (kkoar)FreeCtx                                               kernel compile optimizer automatic (sql) reoptimisation ??
kkoarFreeStats                                               (kkoar)FreeStats                                             kernel compile optimizer automatic (sql) reoptimisation ??
kkqojeFroNoRid                                               (kkqoj)eFroNoRid                                             kernel compile query  outer join ??
kkquIsCorrTransInList                                        (kkqu)IsCorrTransInList                                      kernel compile query  subquery unnesting ??
kslsesftcb_int                                               (ksl)sesftcb_int                                             kernel service (VOS) latching and post-wait ??
ktcsnaInitMinActScnInEnv                                     (ktc)snaInitMinActScnInEnv                                   kernel transaction control component ??
nsmplxhash                                                   (ns)mplxhash                                                 network session ??
pdy7F89_Create_Frame_Descriptor                              (pdy)7F89_Create_Frame_Descriptor                            PLSQL code execution codegen MCode gen ??
pdzdM60_Make_RBST                                            (pdz)dM60_Make_RBST                                          PLSQL code execution codegen infrastructure ??
pdzdM66_Insert_At_Root                                       (pdz)dM66_Insert_At_Root                                     PLSQL code execution codegen infrastructure ??
pdzdM67_Create                                               (pdz)dM67_Create                                             PLSQL code execution codegen infrastructure ??
qerocnStart                                                  (qeroc)nStart                                                query execute rowsource object collection iterator ??
qesmmCUpdProfile                                             (qesmm)CUpdProfile                                           query execute services memory manager (pga) ??
qosdMonNonCGCol                                              (qosd)MonNonCGCol                                            query optimizer statistics sql plan directive ??
qsodasqlGetInsertOneSQL                                      (qsoda)sqlGetInsertOneSQL                                    query system DBMS_SODA ??

code symbol names unique in version 19.5 versus 19.4

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
aux_loadptr_external_principal_identifier_ptr                aux_loadptr_external_principal_identifier_ptr                ??
aux_loadptr_pkinit_krb5_principal_name                       aux_loadptr_pkinit_krb5_principal_name                       ??
aux_loadptr_ptr_kdf_alg_id                                   aux_loadptr_ptr_kdf_alg_id                                   ??
aux_loadptr_ptr_seqof_external_principal_identifier          aux_loadptr_ptr_seqof_external_principal_identifier          ??
aux_loadptr_ptr_supported_kdfs                               aux_loadptr_ptr_supported_kdfs                               ??
aux_loadptr_subject_pk_info_ptr                              aux_loadptr_subject_pk_info_ptr                              ??
aux_present_opt_ptr_kdf_alg_id                               aux_present_opt_ptr_kdf_alg_id                               ??
aux_present_opt_ptr_seqof_external_principal_identifier      aux_present_opt_ptr_seqof_external_principal_identifier      ??
aux_present_opt_ptr_supported_kdfs                           aux_present_opt_ptr_supported_kdfs                           ??
aux_present_opt_subject_pk_info_ptr                          aux_present_opt_subject_pk_info_ptr                          ??
aux_storeptr_external_principal_identifier_ptr               aux_storeptr_external_principal_identifier_ptr               ??
aux_storeptr_pkinit_krb5_principal_name                      aux_storeptr_pkinit_krb5_principal_name                      ??
aux_storeptr_ptr_kdf_alg_id                                  aux_storeptr_ptr_kdf_alg_id                                  ??
aux_storeptr_ptr_seqof_external_principal_identifier         aux_storeptr_ptr_seqof_external_principal_identifier         ??
aux_storeptr_ptr_supported_kdfs                              aux_storeptr_ptr_supported_kdfs                              ??
aux_storeptr_subject_pk_info_ptr                             aux_storeptr_subject_pk_info_ptr                             ??
dbnest_dir_check                                             dbnest_dir_check                                             ??
decode_krb5_auth_pack                                        decode_krb5_auth_pack                                        ??
decode_krb5_auth_pack_draft9                                 decode_krb5_auth_pack_draft9                                 ??
decode_krb5_kdc_dh_key_info                                  decode_krb5_kdc_dh_key_info                                  ??
decode_krb5_pa_pk_as_rep                                     decode_krb5_pa_pk_as_rep                                     ??
decode_krb5_pa_pk_as_req                                     decode_krb5_pa_pk_as_req                                     ??
decode_krb5_pa_pk_as_req_draft9                              decode_krb5_pa_pk_as_req_draft9                              ??
decode_krb5_principal_name                                   decode_krb5_principal_name                                   ??
decode_krb5_reply_key_pack                                   decode_krb5_reply_key_pack                                   ??
decode_krb5_reply_key_pack_draft9                            decode_krb5_reply_key_pack_draft9                            ??
decode_krb5_td_dh_parameters                                 decode_krb5_td_dh_parameters                                 ??
decode_krb5_td_trusted_certifiers                            decode_krb5_td_trusted_certifiers                            ??
encode_krb5_auth_pack                                        encode_krb5_auth_pack                                        ??
encode_krb5_auth_pack_draft9                                 encode_krb5_auth_pack_draft9                                 ??
encode_krb5_kdc_dh_key_info                                  encode_krb5_kdc_dh_key_info                                  ??
encode_krb5_pa_pk_as_rep                                     encode_krb5_pa_pk_as_rep                                     ??
encode_krb5_pa_pk_as_rep_draft9                              encode_krb5_pa_pk_as_rep_draft9                              ??
encode_krb5_pa_pk_as_req                                     encode_krb5_pa_pk_as_req                                     ??
encode_krb5_pa_pk_as_req_draft9                              encode_krb5_pa_pk_as_req_draft9                              ??
encode_krb5_reply_key_pack                                   encode_krb5_reply_key_pack                                   ??
encode_krb5_reply_key_pack_draft9                            encode_krb5_reply_key_pack_draft9                            ??
encode_krb5_td_dh_parameters                                 encode_krb5_td_dh_parameters                                 ??
encode_krb5_td_trusted_certifiers                            encode_krb5_td_trusted_certifiers                            ??
kafcpy_one_row                                               (kaf)cpy_one_row                                             kernel access fetch ??
kcbflc_reap_internal                                         (kcb)flc_reap_internal                                       kernel cache buffers ??
kcbo_incr_doc                                                (kcbo)_incr_doc                                              kernel cache buffers object queue ??
kcbtse_raise_tbs_datafile_enc_mismatch                       (kcbtse)_raise_tbs_datafile_enc_mismatch                     kernel cache buffers tablespace encryption ??
kcfis_clear_lob                                              (kcfis)_clear_lob                                            kernel cache file management intelligent storage ??
kcslcu8                                                      (kcs)lcu8                                                    kernel cache scn management ??
kdilm_row_diskcompress_policy_type                           (kdil)m_row_diskcompress_policy_type                         kernel data index load ??
kdiss_optimize                                               (kdiss)_optimize                                             kernel data index split skip scan ??
kdlfForcedType                                               (kdl)fForcedType                                             kernel data lob ??
kdmrIMCULoadAvg                                              (kdmr)IMCULoadAvg                                            kernel data in-memory data layer background ??
kewraxc_attribute_xcb                                        (kewr)axc_attribute_xcb                                      kernel event AWR repository ??
kewxgenstbcb                                                 (kewx)genstbcb                                               kernel event AWR sysaux? ??
kewxgsc_generic_space_cbf                                    (kewx)gsc_generic_space_cbf                                  kernel event AWR sysaux? ??
kfIsZdlra                                                    (kf)IsZdlra                                                  kernel automatic storage management ??
kfdDskComputeHashForRow                                      (kfd)DskComputeHashForRow                                    kernel automatic storage management disk ??
kfdDskTableCbInternal                                        (kfd)DskTableCbInternal                                      kernel automatic storage management disk ??
kffgAddChildBlk                                              (kffg)AddChildBlk                                            kernel automatic storage management KFF file group ??
kffgAddDefault                                               (kffg)AddDefault                                             kernel automatic storage management KFF file group ??
kffgAttrMaxParCb                                             (kffg)AttrMaxParCb                                           kernel automatic storage management KFF file group ??
kffgAttrParSchmOvrdCb                                        (kffg)AttrParSchmOvrdCb                                      kernel automatic storage management KFF file group ??
kffgAuthenticate                                             (kffg)Authenticate                                           kernel automatic storage management KFF file group ??
kffgCalDeltaRedChg                                           (kffg)CalDeltaRedChg                                         kernel automatic storage management KFF file group ??
kffgCalRealQuota                                             (kffg)CalRealQuota                                           kernel automatic storage management KFF file group ??
kffgCalTargetRedun                                           (kffg)CalTargetRedun                                         kernel automatic storage management KFF file group ??
kffgCmp                                                      (kffg)Cmp                                                    kernel automatic storage management KFF file group ??
kffgFillFg                                                   (kffg)FillFg                                                 kernel automatic storage management KFF file group ??
kffgFindFree                                                 (kffg)FindFree                                               kernel automatic storage management KFF file group ??
kffgGetParOvrd                                               (kffg)GetParOvrd                                             kernel automatic storage management KFF file group ??
kffgIsParitySupported                                        (kffg)IsParitySupported                                      kernel automatic storage management KFF file group ??
kffgIsRedunAvailable                                         (kffg)IsRedunAvailable                                       kernel automatic storage management KFF file group ??
kffgIsUnsplit                                                (kffg)IsUnsplit                                              kernel automatic storage management KFF file group ??
kffgMakeName                                                 (kffg)MakeName                                               kernel automatic storage management KFF file group ??
kffgParseParOvrd                                             (kffg)ParseParOvrd                                           kernel automatic storage management KFF file group ??
kffgVerifyRedunChg                                           (kffg)VerifyRedunChg                                         kernel automatic storage management KFF file group ??
kffgcSyncDBLazy                                              (kffg)cSyncDBLazy                                            kernel automatic storage management KFF file group ??
kffgcSyncEnd                                                 (kffg)cSyncEnd                                               kernel automatic storage management KFF file group ??
kffgftMap                                                    (kffg)ftMap                                                  kernel automatic storage management KFF file group ??
kffgptAddDefault                                             (kffg)ptAddDefault                                           kernel automatic storage management KFF file group ??
kffgptFixRedun                                               (kffg)ptFixRedun                                             kernel automatic storage management KFF file group ??
kffgptGetStrVal                                              (kffg)ptGetStrVal                                            kernel automatic storage management KFF file group ??
kffgptLookup                                                 (kffg)ptLookup                                               kernel automatic storage management KFF file group ??
kffgptLookupDump                                             (kffg)ptLookupDump                                           kernel automatic storage management KFF file group ??
kffgptParseNum                                               (kffg)ptParseNum                                             kernel automatic storage management KFF file group ??
kfgComputeGrpProp                                            (kfg)ComputeGrpProp                                          kernel automatic storage management diskgroups ??
kfgGrpTableCbInternal                                        (kfg)GrpTableCbInternal                                      kernel automatic storage management diskgroups ??
kfiQuantize2                                                 (kf)iQuantize2                                               kernel automatic storage management ??
kfioComputeStripeIndex                                       (kfio)ComputeStripeIndex                                     kernel automatic storage management translation I/O layer ??
kfioGetXSetSize                                              (kfio)GetXSetSize                                            kernel automatic storage management translation I/O layer ??
kfioLoadParity                                               (kfio)LoadParity                                             kernel automatic storage management translation I/O layer ??
kfioParityWriteBegin                                         (kfio)ParityWriteBegin                                       kernel automatic storage management translation I/O layer ??
kfk_alloc_kfkio_freelist                                     (kfk)_alloc_kfkio_freelist                                   kernel automatic storage management KFK ??
kfk_asm_ioerror                                              (kfk)_asm_ioerror                                            kernel automatic storage management KFK ??
kfk_bld_sbmtptr_arry                                         (kfk)_bld_sbmtptr_arry                                       kernel automatic storage management KFK ??
kfk_bld_waitptr_arry                                         (kfk)_bld_waitptr_arry                                       kernel automatic storage management KFK ??
kfk_block_asmlib                                             (kfk)_block_asmlib                                           kernel automatic storage management KFK ??
kfk_block_oss                                                (kfk)_block_oss                                              kernel automatic storage management KFK ??
kfk_block_ufs                                                (kfk)_block_ufs                                              kernel automatic storage management KFK ??
kfk_build_osdstruct                                          (kfk)_build_osdstruct                                        kernel automatic storage management KFK ??
kfk_chk_xor_valid                                            (kfk)_chk_xor_valid                                          kernel automatic storage management KFK ??
kfk_do_abn_xor_chk                                           (kfk)_do_abn_xor_chk                                         kernel automatic storage management KFK ??
kfk_error_emulate                                            (kfk)_error_emulate                                          kernel automatic storage management KFK ??
kfk_fill_ksfdrq                                              (kfk)_fill_ksfdrq                                            kernel automatic storage management KFK ??
kfk_find_io_subsys                                           (kfk)_find_io_subsys                                         kernel automatic storage management KFK ??
kfk_get_asmioc_oper                                          (kfk)_get_asmioc_oper                                        kernel automatic storage management KFK ??
kfk_get_free_kfkrq                                           (kfk)_get_free_kfkrq                                         kernel automatic storage management KFK ??
kfk_get_unreaped_ios                                         (kfk)_get_unreaped_ios                                       kernel automatic storage management KFK ??
kfk_hard_abn_set                                             (kfk)_hard_abn_set                                           kernel automatic storage management KFK ??
kfk_hard_getoffs                                             (kfk)_hard_getoffs                                           kernel automatic storage management KFK ??
kfk_iodone_merge_slave                                       (kfk)_iodone_merge_slave                                     kernel automatic storage management KFK ??
kfk_lib_abn_check                                            (kfk)_lib_abn_check                                          kernel automatic storage management KFK ??
kfk_lib_xor_check                                            (kfk)_lib_xor_check                                          kernel automatic storage management KFK ??
kfk_oss_cell_timedout                                        (kfk)_oss_cell_timedout                                      kernel automatic storage management KFK ??
kfk_oss_dump_orq                                             (kfk)_oss_dump_orq                                           kernel automatic storage management KFK ??
kfk_oss_get_orq                                              (kfk)_oss_get_orq                                            kernel automatic storage management KFK ??
kfk_oss_offload_fill_orq                                     (kfk)_oss_offload_fill_orq                                   kernel automatic storage management KFK ??
kfk_oss_offload_submit                                       (kfk)_oss_offload_submit                                     kernel automatic storage management KFK ??
kfk_oss_post_wait                                            (kfk)_oss_post_wait                                          kernel automatic storage management KFK ??
kfk_oss_put_orq                                              (kfk)_oss_put_orq                                            kernel automatic storage management KFK ??
kfk_parse_disk_path                                          (kfk)_parse_disk_path                                        kernel automatic storage management KFK ??
kfk_proc_submitted_lib_reqs                                  (kfk)_proc_submitted_lib_reqs                                kernel automatic storage management KFK ??
kfk_process_an_ioq                                           (kfk)_process_an_ioq                                         kernel automatic storage management KFK ??
kfk_put_free_kfkios                                          (kfk)_put_free_kfkios                                        kernel automatic storage management KFK ??
kfk_reap_io_from_kfks                                        (kfk)_reap_io_from_kfks                                      kernel automatic storage management KFK ??
kfk_reap_lib_async_io                                        (kfk)_reap_lib_async_io                                      kernel automatic storage management KFK ??
kfk_reap_oss_async_io                                        (kfk)_reap_oss_async_io                                      kernel automatic storage management KFK ??
kfk_reap_ufs_async_io                                        (kfk)_reap_ufs_async_io                                      kernel automatic storage management KFK ??
kfk_rebuild_kfdrqs                                           (kfk)_rebuild_kfdrqs                                         kernel automatic storage management KFK ??
kfk_retry_submit_oss_io                                      (kfk)_retry_submit_oss_io                                    kernel automatic storage management KFK ??
kfk_submit_lib_io                                            (kfk)_submit_lib_io                                          kernel automatic storage management KFK ??
kfk_submit_oss_io                                            (kfk)_submit_oss_io                                          kernel automatic storage management KFK ??
kfk_submit_oss_io_no_wtevt                                   (kfk)_submit_oss_io_no_wtevt                                 kernel automatic storage management KFK ??
kfk_submit_oss_io_wtevt                                      (kfk)_submit_oss_io_wtevt                                    kernel automatic storage management KFK ??
kfk_submit_ufs_io                                            (kfk)_submit_ufs_io                                          kernel automatic storage management KFK ??
kfk_transit_fetchIO                                          (kfk)_transit_fetchIO                                        kernel automatic storage management KFK ??
kfk_transit_waitIO                                           (kfk)_transit_waitIO                                         kernel automatic storage management KFK ??
kfk_update_iodone_stats                                      (kfk)_update_iodone_stats                                    kernel automatic storage management KFK ??
kfk_zero_reap_array                                          (kfk)_zero_reap_array                                        kernel automatic storage management KFK ??
kfmdIsDGMountedCluster                                       (kfmd)IsDGMountedCluster                                     kernel automatic storage management node monitor interface implementation layer
                                                                                                                          for diskgroup registration ??

kfmsGetGrpMemCnt                                             (kfms)GetGrpMemCnt                                           kernel automatic storage management node monitor interface implementation layers
                                                                                                                           support function interface ??

kgfdDiscoverSubmit                                           (kgfd)DiscoverSubmit                                         kernel generic ASM io subsystem driver ??
kgh_size_sanity_check                                        (kgh)_size_sanity_check                                      kernel generic heap manager ??
kjac_allow_tg                                                (kjac)_allow_tg                                              kernel lock management application continuity  ??
kjccspbat                                                    (kjc)cspbat                                                  kernel lock management communication ??
kkocfbFindOldActual                                          (kkocfb)FindOldActual                                        kernel compile optimizer cardinality feedback ??
kkopinlpInt                                                  (kko)pinlpInt                                                kernel compile optimizer ??
kkscsGetPQSharingParameters                                  (kkscs)GetPQSharingParameters                                kernel compile shared objects (cursor) child cursor ??
kkscsGetSessionLimitDOP                                      (kkscs)GetSessionLimitDOP                                    kernel compile shared objects (cursor) child cursor ??
kokegAddOneGarbageItem                                       (koke)gAddOneGarbageItem                                     kernel objects kernel side OBJECTS images ??
kokegRepinLob                                                (koke)gRepinLob                                              kernel objects kernel side OBJECTS images ??
kpochalts                                                    (kpo)chalts                                                  kernel programmatic interface oracle ??
kpossGetEncNonTemplateOverflow                               (kpo)ssGetEncNonTemplateOverflow                             kernel programmatic interface oracle ??
kpossSetEncNonTemplateOverflow                               (kpo)ssSetEncNonTemplateOverflow                             kernel programmatic interface oracle ??
kpossTemplateMatch_                                          (kpo)ssTemplateMatch_                                        kernel programmatic interface oracle ??
kpossTemplateSet                                             (kpo)ssTemplateSet                                           kernel programmatic interface oracle ??
kpueONSTraceWrite                                            (kpu)eONSTraceWrite                                          kernel programmatic interface user ??
kpueONSTrcCb                                                 (kpu)eONSTrcCb                                               kernel programmatic interface user ??
kpussTemplateSend                                            (kpu)ssTemplateSend                                          kernel programmatic interface user ??
kpuxcOpenCurCntDecr                                          (kpu)xcOpenCurCntDecr                                        kernel programmatic interface user ??
kpuxcOpenCurCntIncr                                          (kpu)xcOpenCurCntIncr                                        kernel programmatic interface user ??
kpuxcRefCurDfnMarkOpen                                       (kpu)xcRefCurDfnMarkOpen                                     kernel programmatic interface user ??
kpuxcRefCurOutBndMarkOpen                                    (kpu)xcRefCurOutBndMarkOpen                                  kernel programmatic interface user ??
kpuxcSessionSignatureRecv                                    (kpu)xcSessionSignatureRecv                                  kernel programmatic interface user ??
kpuxcSessionTemplateSend                                     (kpu)xcSessionTemplateSend                                   kernel programmatic interface user ??
kpuxcSessionTemplatesFree                                    (kpu)xcSessionTemplatesFree                                  kernel programmatic interface user ??
kpuxcTACService                                              (kpu)xcTACService                                            kernel programmatic interface user ??
kpuxsoCurPrefetchFree                                        (kpu)xsoCurPrefetchFree                                      kernel programmatic interface user ??
kpuxsoOperInit                                               (kpu)xsoOperInit                                             kernel programmatic interface user ??
krbbCountThreads                                             (krbb)CountThreads                                           kernel redo backup/restore creation of a backuppiece ??
ksdhng_wdat_rem_dup                                          (ksdhng)_wdat_rem_dup                                        kernel service (VOS) debug internal errors hang analyze ??
ksfdafGetXsetSize                                            (ksfdaf)GetXsetSize                                          kernel service (VOS) functions disk IO asm files ??
ksfdafParityWrite                                            (ksfdaf)ParityWrite                                          kernel service (VOS) functions disk IO asm files ??
ksfdafParityWriteAbort                                       (ksfdaf)ParityWriteAbort                                     kernel service (VOS) functions disk IO asm files ??
ksfdafParityWriteBegin                                       (ksfdaf)ParityWriteBegin                                     kernel service (VOS) functions disk IO asm files ??
ksfdafParityWriteEnd                                         (ksfdaf)ParityWriteEnd                                       kernel service (VOS) functions disk IO asm files ??
kslwt_event_stats                                            (kslwt)_event_stats                                          kernel service (VOS) latching and post-wait wait interface ??
kslwt_push_stats                                             (kslwt)_push_stats                                           kernel service (VOS) latching and post-wait wait interface ??
ksmsq_spendingi                                              (ksmsq)_spendingi                                            kernel service (VOS) memory sga heap message queue services ??
ksp_init_modp_send                                           (ksp)_init_modp_send                                         kernel service (VOS) parameter ??
ksp_modp_get_enckeyvals                                      (ksp)_modp_get_enckeyvals                                    kernel service (VOS) parameter ??
ksp_modp_get_keyvals                                         (ksp)_modp_get_keyvals                                       kernel service (VOS) parameter ??
ksp_modp_set_enckeyvals                                      (ksp)_modp_set_enckeyvals                                    kernel service (VOS) parameter ??
ksp_modp_set_keyvals                                         (ksp)_modp_set_keyvals                                       kernel service (VOS) parameter ??
ksp_modp_update_sign                                         (ksp)_modp_update_sign                                       kernel service (VOS) parameter ??
kspdecbuf                                                    (ksp)decbuf                                                  kernel service (VOS) parameter ??
kspencbuf                                                    (ksp)encbuf                                                  kernel service (VOS) parameter ??
ksucln_expired_cleanup_timer                                 (ksucln)_expired_cleanup_timer                               kernel service (VOS) user cleanup detached process ??
kswrListDelete                                               (ksw)rListDelete                                             kernel service (VOS) workgroup ??
ksws_alb_alloc_pga_memory                                    (ksws)_alb_alloc_pga_memory                                  kernel service (VOS) workgroup services ??
ksws_alb_block_service                                       (ksws)_alb_block_service                                     kernel service (VOS) workgroup services ??
ksws_alb_check_sessions_to_mark                              (ksws)_alb_check_sessions_to_mark                            kernel service (VOS) workgroup services ??
ksws_alb_check_threshold                                     (ksws)_alb_check_threshold                                   kernel service (VOS) workgroup services ??
ksws_alb_clear_mark_drain_session                            (ksws)_alb_clear_mark_drain_session                          kernel service (VOS) workgroup services ??
ksws_alb_dump_service                                        (ksws)_alb_dump_service                                      kernel service (VOS) workgroup services ??
ksws_alb_dump_session                                        (ksws)_alb_dump_session                                      kernel service (VOS) workgroup services ??
ksws_alb_filter_session                                      (ksws)_alb_filter_session                                    kernel service (VOS) workgroup services ??
ksws_alb_mark_target_sessions                                (ksws)_alb_mark_target_sessions                              kernel service (VOS) workgroup services ??
ksws_alb_process_directives                                  (ksws)_alb_process_directives                                kernel service (VOS) workgroup services ??
ksws_alb_run_gv_sql                                          (ksws)_alb_run_gv_sql                                        kernel service (VOS) workgroup services ??
ksws_alb_safe_session                                        (ksws)_alb_safe_session                                      kernel service (VOS) workgroup services ??
ksws_alb_safe_to_block_service                               (ksws)_alb_safe_to_block_service                             kernel service (VOS) workgroup services ??
ksws_alb_selection_bias_apply                                (ksws)_alb_selection_bias_apply                              kernel service (VOS) workgroup services ??
ksws_alb_set_mark_drain_session                              (ksws)_alb_set_mark_drain_session                            kernel service (VOS) workgroup services ??
ksws_alb_unblock_fix_delta_services                          (ksws)_alb_unblock_fix_delta_services                        kernel service (VOS) workgroup services ??
ksws_alb_unblock_service                                     (ksws)_alb_unblock_service                                   kernel service (VOS) workgroup services ??
ksws_alb_update_service_session_metrics                      (ksws)_alb_update_service_session_metrics                    kernel service (VOS) workgroup services ??
ksws_alb_update_threshold                                    (ksws)_alb_update_threshold                                  kernel service (VOS) workgroup services ??
ksws_alb_valid_session                                       (ksws)_alb_valid_session                                     kernel service (VOS) workgroup services ??
kwraChkOptdef                                                (k)wraChkOptdef                                              kernel ??
kwrachkopnTree                                               (k)wrachkopnTree                                             kernel ??
kwraplschkLogdef                                             (k)wraplschkLogdef                                           kernel ??
kxccXRkxck                                                   (kxcc)XRkxck                                                 kernel execution constraints catch all ??
kzctxgIsGlobal                                               (kzctxg)IsGlobal                                             kernel security Code-Based Access Control context routines global_context ??
kzrtGetAndCheckTriggerCache                                  (kzrt)GetAndCheckTriggerCache                                kernel security virtual private database privileges ??
kzrtTriggerPresent                                           (kzrt)TriggerPresent                                         kernel security virtual private database privileges ??
kzvdveCommOpBypass                                           (kzvd)veCommOpBypass                                         kernel security data vault ??
kzvdveIsAppCommonUserId                                      (kzvd)veIsAppCommonUserId                                    kernel security data vault ??
kzvradmAllowCommOp                                           (kz)vradmAllowCommOp                                         kernel security ??
kzvradmAllowCommOpCB                                         (kz)vradmAllowCommOpCB                                       kernel security ??
kzvradmAllowCommOp_int                                       (kz)vradmAllowCommOp_int                                     kernel security ??
kzvutilGetACOStatus                                          (kz)vutilGetACOStatus                                        kernel security ??
kzvutilGetACOStatus0                                         (kz)vutilGetACOStatus0                                       kernel security ??
psdvcn                                                       (psd)vcn                                                     PLSQL infrastructure programmatic interfaces to/from  ??
qesrcTol_New                                                 (qesrc)Tol_New                                               query execute services result cache ??
qsodasqlGetContentDefine                                     (qsoda)sqlGetContentDefine                                   query system DBMS_SODA ??
qsodasqlGetCreatedOnDefine                                   (qsoda)sqlGetCreatedOnDefine                                 query system DBMS_SODA ??
qsodasqlGetInsertSQL                                         (qsoda)sqlGetInsertSQL                                       query system DBMS_SODA ??
qsodasqlGetKeyDefine                                         (qsoda)sqlGetKeyDefine                                       query system DBMS_SODA ??
qsodasqlGetLastModDefine                                     (qsoda)sqlGetLastModDefine                                   query system DBMS_SODA ??
qsodasqlGetMediaDefine                                       (qsoda)sqlGetMediaDefine                                     query system DBMS_SODA ??
qsodasqlGetVersionDefine                                     (qsoda)sqlGetVersionDefine                                   query system DBMS_SODA ??
qsodasqlPrefetchPrepare                                      (qsoda)sqlPrefetchPrepare                                    query system DBMS_SODA ??
qsodasqlSelectDocFieldsSQL                                   (qsoda)sqlSelectDocFieldsSQL                                 query system DBMS_SODA ??
qsodaxArrayFetchDocuments                                    (qsoda)xArrayFetchDocuments                                  query system DBMS_SODA ??
qsodaxDefineDocumentFields                                   (qsoda)xDefineDocumentFields                                 query system DBMS_SODA ??
qsodaxGetPrefetchDocument                                    (qsoda)xGetPrefetchDocument                                  query system DBMS_SODA ??
sskgds_dw_is_context_valid                                   (sskg)ds_dw_is_context_valid                                 operating system dependent system kernel generic ??
sskgds_dw_is_pccontext_valid                                 (sskg)ds_dw_is_pccontext_valid                               operating system dependent system kernel generic ??

In the article oracle memory troubleshooting using analysis on heap dumps I introduced heap_analyze.awk.

The reason the tool exists is because I am using it myself. Therefore, I ran into additional things that I wanted the tool to do. I added some stuff, which is that significant, that I decided to make another blogpost to introduce the new features.

1. Percentages
In order to get an idea of the relative size of the summarised topic, I added a percentage. For example:

top 5 allocation by total size per alloc reason per heap
==================================================================================================
heap             alloc_reason            #chunks       total_size   %
--------------------------------------------------------------------------------------------------
sga heap(1,0)    perm                         19        182947456  72
sga heap(1,0)                                 43         14488960   5
sga heap(1,0)    SO private sga               18         14284168   5
sga heap(1,0)    KGLHD                      5787          4318400   1
sga heap(1,0)    KSFD SGA I/O b                1          4190416   1

2. Enhanced perm (permanent) memory descriptions
It seems that for PGA heap dumps, sometimes there is a description for memory area’s that are perm (permanent memory, memory allocated for the lifetime of the process). This is how that’s visible in the dump:

PERMANENT CHUNKS:
  Chunk     7fcbcad6c020 sz=    20632    perm      "perm           "  alo=20632
            7fcbcad6c020 sz=    20632    cpmlst    "callback hsn   "

I must say I don’t know what ‘cpmlst’ means, so if anyone knows or has a good guess, please let me know. However, the two addresses and sizes are an exact match, so I now change the alloc_reason for the cpmlst text.
This is helpful because there is a quite some memory allocated as perm. Sadly, this is not done for SGA dumps.

3. Excel mode
If you want to store and compare different dumps, one way of doing that is pasting the output in Microsoft excel. Once you set the ‘Text To Columns’ option to space as a separator, it will put the information in its own cells. But there are a few problems with that, one of them is that the heap names and alloc_reasons can have spaces in them, so that the placement of the figures can vary. I created excel mode for that.
If you set excel mode (set the variable excel_mode to 1 on line 5):

#!/bin/awk -f
BEGIN {
  printf "Heap dump analyzer v0.2 by Frits Hoogland\n";
  group_sum_nr=5;
  excel_mode=1;
}

In this mode, the horizontal lines (with ‘-‘ and ‘=’) are omitted in the output, and all spaces are changed to underscores, so a table stays consistent when pasted in excel:

heap             alloc_reason            #chunks       total_size   %
sga_heap(1,0)    perm___________              19        182947456  72
sga_heap(1,0)    _______________              43         14488960   5
sga_heap(1,0)    SO_private_sga_              18         14284168   5
sga_heap(1,0)    KGLHD__________            5787          4318400   1
sga_heap(1,0)    KSFD_SGA_I/O_b_               1          4190416   1

4. New table which shows memory by type
Another way of looking at memory in a heap is by grouping it by type. This allows you to very quickly see if a certain type of chunk is dominating a heap:

heap                        type    #chunks         min_size         max_size       total_size   %
--------------------------------------------------------------------------------------------------
top call heap               free          7            16408            65512           390064  99
top call heap           recreate          2              992              992             1984   0
top call heap               perm          2              120              904             1024   0

This is an overview of the top call heap of a session that is not active, and therefore most of it should be empty, which is true for this dump.

Once again, get the awk script here: https://gitlab.com/FritsHoogland/oracle_memory_analyze/blob/master/heap_analyze.awk

This blogpost is about analysing Oracle heap dumps. It is an extension to earlier work, Tanel Poder’s heap dump analyzer. So hat tip to Tanel, he’s done the hard work, I merely increased the presentation options. The heap analyser script that I wrote analyses Oracle heapdumps from the trace file that the dump was written to by the Oracle database. Because the heap dump representation is the same between PGA and SGA memory, it can work on both. The reason for this is that memory management is done by the same memory manager, and is commonly called ‘kgh’ (kernel generic heap) managed memory.

Please mind that for PGA analysis, not all memory is managed by the kgh memory manager. For example memory used for networking (sqlnet) is allocated totally outside of the kgh memory manager.

Let’s take the output of a PGA heap dump at level 29 (PGA, UGA, CGA, top call heaps, call heaps, session heap; executed via ‘alter session set events ‘immediate trace name heapdump level 29”):

$ ./heap_analyze.awk /u01/app/oracle/diag/rdbms/tt/tt/trace/tt_ora_6515.trc
Heap dump analyzer v0.1 by Frits Hoogland
heap size totals (all sizes in bytes)
==============================================================================================
heap name                       total size
----------------------------------------------------------------------------------------------
top call heap                       393096
callheap                             36200
pga heap                           1016816
top uga heap                        262048
session heap                        261744
------------------------------------------
total                              1969904

top 5 allocation by total size per alloc reason per heap
==============================================================================================
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
top call heap                                  5           354456
top call heap    callheap                     11            37616
top call heap    perm                          2             1024
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
callheap                                      10            35920
callheap         ctxtcds: kksQui               1              160
callheap         perm                          1              120
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
pga heap         perm                         44           290256
pga heap         Fixed UGA heap                1           226232
pga heap         diag pga                     21           214048
pga heap                                       7            62616
pga heap         kfkio bucket                  1            40984
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
top uga heap     session heap                  4           261928
top uga heap     perm                          1              120
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
session heap     perm                          3            63608
session heap                                   8            63528
session heap     koh-kghu sessi                5            28288
session heap     kxsFrame4kPage                6            24912
session heap     kxsFrame8kPage                3            24744

top 5 allocations by total size per alloc reason, type, chunk size per heap
==============================================================================================
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
top call heap                                  1       free           131048           131048
top call heap                                  2       free            65512           131024
top call heap                                  1       free            63496            63496
top call heap                                  1       free            28888            28888
top call heap    callheap                      6   freeable             4224            25344
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
callheap                                       6       free             4184            25104
callheap                                       2       free             4192             8384
callheap                                       1       free             1784             1784
callheap                                       1       free              648              648
callheap         ctxtcds: kksQui               1   freeable              160              160
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
pga heap         Fixed UGA heap                1   recreate           226232           226232
pga heap         perm                          1       perm            71720            71720
pga heap         diag pga                      1   freeable            55048            55048
pga heap         diag pga                      2   freeable            27456            54912
pga heap                                       1       free            47296            47296
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
session heap     perm                          1       perm            43584            43584
session heap                                   1       free            36544            36544
session heap     kxsFrame4kPage                6   freeable             4152            24912
session heap     kxsFrame8kPage                3   freeable             8248            24744
session heap                                   1       free            20352            20352
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
top uga heap     session heap                  3   freeable            65512           196536
top uga heap     session heap                  1   recreate            65392            65392
top uga heap     perm                          1       perm              120              120

top 5 allocations by total size per heap, alloc reason, type, chunk size
==============================================================================================
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
pga heap         Fixed UGA heap                1   recreate           226232           226232
top uga heap     session heap                  3   freeable            65512           196536
top call heap                                  1       free           131048           131048
top call heap                                  2       free            65512           131024
pga heap         perm                          1       perm            71720            71720

I figured that the first thing you want to see, are the heaps that are in the dump, and their sizes. That’s what is visible in rows 2-12.
I summed the heap sizes, which might make sense, or it might not.
In this case, with a heap dump that includes PGA, UGA and CGA plus session and call heaps, it means there are heaps in the dump that are part of another heap that is in the dump. So the total size here is bogus. This means that you need to have an understanding of what is actually dumped.

The next section, top 5 allocations by total size per alloc reason per dump, shows a per-heap summary by the allocation reason in the dump. If there’s no alloc reason, it’s free memory. Because this is a heap dump of a session that has done nothing (I just started sqlplus and ran the dump of its own PGA memory), you see that a lot of memory chunks are free memory. If you look closely to the allocation reasons, you can see that the ‘top call heap’ has a memory section that is called ‘callheap’ which is slightly larger than the ‘callheap’ section in the heap totals, and the ‘top uga heap’ section has a memory section called ‘session heap’ that is slightly larger than the ‘session heap’ section in the heap totals. In this case, it means that you can actually see the subheads in the parent heap allocation totals. The subheap size must be slightly larger in the parent heap because of headers in the memory allocations which are needed memory management. Please mind that this is based on my knowledge of how process memory is created, the only way to be absolutely sure is that a heap is part of another heap is to look at the memory addresses. This output only shows the heap names, not the addresses. The purpose of this section is to have an understanding of where memory is allocated to in a heap.

The following section, top 5 allocations by total size per alloc reason, type, chunk size per heap shows a per-heap summary by reason, type and chunksize, so you can investigate if specific types and sizes of chunks are causing issues or weird behaviour.

The last section is the same as the previous section, but doesn’t do it per heap. This is identical to what Tanel’s heapdump_analyzer shows.

You can find the heap_analyze.awk script here: https://gitlab.com/FritsHoogland/oracle_memory_analyze/blob/master/heap_analyze.awk

This is the output of a dump of the SGA of my small test database (oradebug dump heapdump 2):

$ ./heap_analyze.awk /u01/app/oracle/diag/rdbms/tt/tt/trace/tt_ora_9461.trc
Heap dump analyzer v0.1 by Frits Hoogland
heap size totals (all sizes in bytes)
==============================================================================================
heap name                       total size
----------------------------------------------------------------------------------------------
sga heap(1,0)                    285210904
sga heap(1,3)                     83885560
------------------------------------------
total                            369096464

top 5 allocation by total size per alloc reason per heap
==============================================================================================
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
sga heap(1,0)    perm                         22        188068904
sga heap(1,0)                                141         17975320
sga heap(1,0)    SO private sga               17         14268008
sga heap(1,0)    KQR PO                     8271          7568912
sga heap(1,0)    KGLHD                      9381          7023960
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
sga heap(1,3)                                 51         14120288
sga heap(1,3)    SQLA^6d9b8a7e               337          1380352
sga heap(1,3)    SQLA^31cc505b               167           684032
sga heap(1,3)    SQLA^aab93e92               162           663552
sga heap(1,3)    PLDIA^191e0a8d              155           634880

top 5 allocations by total size per alloc reason, type, chunk size per heap
==============================================================================================
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
sga heap(1,0)    perm                          1       perm         15937496         15937496
sga heap(1,0)    perm                          1       perm         15931312         15931312
sga heap(1,0)    perm                          1       perm         15811464         15811464
sga heap(1,0)    perm                          1       perm         15741952         15741952
sga heap(1,0)    perm                          1       perm         15723584         15723584
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
sga heap(1,3)                                  1       free          9883208          9883208
sga heap(1,3)                                  4     R-free           839480          3357920
sga heap(1,3)    SQLA^6d9b8a7e               336  freeableU             4096          1376256
sga heap(1,3)                                  1     R-free           839360           839360
sga heap(1,3)    SQLA^31cc505b               166  freeableU             4096           679936

top 5 allocations by total size per heap, alloc reason, type, chunk size
==============================================================================================
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
sga heap(1,0)    perm                          1       perm         15937496         15937496
sga heap(1,0)    perm                          1       perm         15931312         15931312
sga heap(1,0)    perm                          1       perm         15811464         15811464
sga heap(1,0)    perm                          1       perm         15741952         15741952
sga heap(1,0)    perm                          1       perm         15723584         15723584

It’s interesting to see only subpool 1 sub-sub pool 1 and 3 are used. Subpool 1,1 contains a lot of permanent allocations and a lot of allocations that might have a more permanent nature, like KQR (dictionary cache) allocations, Subpool 1,3 seems to have allocations that are deemed more transient in nature, like SQLA (sql area) allocations. This might be wildly different in databases that are actually heavily used, this is an idle lab database.

Please mind it’s important to understand that dumping the shared pool requires obtaining the respective latches, so doing that on a live production system might lead to (severe) issues. Only do this if you know what you are doing. For the PGA there can only be one using process by definition, but be careful there too, if you request a PGA dump you are interacting with memory that is deemed private by the process that is using that.

If you require more rows to be shown than the 5 that are shown, set the ‘group_sum_nr’ variable to the amount you need on row 4 of the script.

One of the continuing issues that I see is accountability (who did what on what server) on Linux or other operating systems that are using ssh.

The traditional solution for this problem is personal accounts, which provides logon and logoff registration.
Personal accounts often do have the least amount of rights which means typical administration tasks can not be done with them, in order to actually do something you need to sudo to an application account or root, which is logged by sudo, on enterprise linuxes to /var/log/secure. So some important facts are logged (logon/logoff/switching users), but a lot of important things are not logged at all, or maybe can be indirectly derived by combining data of operating system audit together with application specific logging.

With cloud, everything gotten a lot more dynamic, and it’s really common for me to see client cloud environments where there aren’t personal accounts at all, everybody just uses the common cloud user (ec2-user, opc, etc.). Needless to say, this is an error and an issue if you want to build a professional setup. Maybe it’s me and I am unlucky, but I have a sneaking suspicion this is actually quite common.

First things first: if you are serious about security and want to actually track what is happening, you MUST have central logging. Otherwise, if a user can switch to root, it can stop the logging and overwrite or change the evidence. Of course with central logging when a user can switch to root, the logging can still be stopped, but then at least you should have proof who did that, because the logging on another machine can not be changed. If not, you might as well not take any action at all, because you can’t be sure what you see is true.

Personal accounts: I think in almost any situation that is even a tiny bit dynamic, creating and removing users locally on each server is not a sustainable practice; even if you can get it up to date, you need to continuously maintain it. Some cloud and some on-premises situations have LDAP. By centralising the user administration, user management becomes manageable. However, the amount of audit information you get from it is limited (the aforementioned logon, logoff and switch user administration). It allows you to grant access to an individual server per user. One downside is that you need to add operating system packages to provide the LDAP authentication functionality, and obviously you explicitly need to configure each server to authenticate users with an LDAP server.

There is another way, which is way less intrusive, which is using native secure shell daemon functionality (so no extra installs): that is authenticate CA (certificate authority) signed user keys. To use this, you need an additional key pair, which serves as the CA keys, put the CA public key on each server and point the secure shell daemon to this key being the CA key. This serves as an additional authentication, so all existing secure shell daemon functionality, like username/password authentication and public key authentication using the authorized_keys file is still valid. This authentication mechanism is also self contained: there is no communication needed to an external authentication service (like an LDAP server); it uses a key that is placed on the server. The downside is that it’s an all-or-nothing setting, which means that you get access to all hosts which have a certain CA key set, you can’t disable a certain host for a certain user.

The authentication of with using CA key based authentication is based on the user’s signed key instead of named account. This is a huge difference from using LDAP/named accounts, where the authentication is done based on the actual account name. If you let users sign their key with the CA key theirselves, you need to keep track of the identity of who signed the key, because that declares the actual identity, not the account that logged on to a server.

This functionality is provided by native ssh key management software. If you want to use authentication of users based on CA signed keys, you need to manually securely store and protect CA private key, and keep an impeccable registration of who signed which key, because that declares the actual user identity. For the user there isn’t a lot of difference, its private key is still the ultimate declaration of its true identity (as you will see, this mechanism requires a user to specify its public and private keys).

Luckily, there is a tool that provides the key signing functionality together with auditing (the registration), and secure authentication of a user: Hashicorp vault.

It’s my advise to let users use certificates for authentication to vault rather than passwords, additional to their ssh key pair. Probably you can write an entire blogpost about it, but passwords are cumbersome.

1. User side initial setup.
A user needs two key pairs, one for authenticating to vault using a pem encoded pair:

$ openssl req -newkey rsa:2048 -new -nodes -x509 -days 365 -keyout fritshoogland_key.pem -out fritshoogland_pub.pem

This will ask some questions, like country, state, locality, etc.
It’s important that a user fills out its true name with ‘common name’ and password with ‘password’.
As an administrator you can check the contents of the public key with ‘openssl x509 -in PUBKEY.pem -text’.

The other one is the regular ssh keypair:

$ ss-keygen -t rsa

It will ask for a password, and save the files in ~/.ssh, ‘id_rsa’ for the private key, and ‘id_rsa.pub’ for the public key.

It is important to realise that the ultimate identity is dependent on the private keys of both key pairs, which should be kept securely by the user and never handed over anyone else. This setup and any normal usage ever only needs the public keys to be provided publicly, which is the function of a public key, hence the name.

2. vault side initial setup.
There are two mechanisms that are going to be used inside vault, which is an authentication mechanism based on certificates, and a secrets service for ssh.
The certificate authentication facility needs to be setup using a vault root/admin account. The first thing is to setup the certificate authentication method:

$ vault auth enable cert

It only needs to be done once for a given path, and the authentication mechanism can optionally be mounted at a different path by adding ‘-path=’ if you want to use separate the certificates from others, because this allows the same authentication mechanism to be used more than once.

The next thing that needs to be setup is the ssh secrets service (this too can be mounted at a different path by adding ‘-path’):

$ vault secrets enable ssh

Now we can extract the CA public key from this setup, which is used as the CA public key on the servers to validate the user keys signed with it:

$ vault write ssh/config/ca > trusted-user-ca-key.pem

The next thing we need is a signing role that will be used to sign a user key with the CA. This allows you to specify any properties you want the user CA signed key to hold, like ttl (time to live) and allowed_users:

echo "{
  \"allow_user_certificates\": true,
  \"allowed_users\": \"vagrant\",
  \"default_extensions\": [
    {
      \"permit-pty\": \"\"
    }
  ],
  \"key_type\": \"ca\",
  \"default_user\": \"vagrant\",
  \"ttl\": \"24h\"
}" | vault write ssh-client/roles/sign-role -

There are a couple of limitations set here:
Line 3: this specifies that only a user named ‘vagrant’ is allowed (which is the default user in my lab, alike ec2-user, opc, etc.).
Line 6: permit-pty is needed to let the session get access to the terminal.
Line 11: the TTL, time to live, of the generated signed key is 24 hours. This means every day a new signed key needs to be obtained. Because a user should be able to do this itself, this doesn’t generate an administrative burden. This does allow you to disable access for any user using this signed keys authentication method in a day.

The last thing in this step is to create a policy that allows access to the sign role. This policy must be granted to a user (certificate) so it can actually perform the CA key signing. In order to deny access, you can simply remove this policy from a certificate, which then disables the ability to perform the CA key signing.

echo "path \"ssh/sign/sign-role\" {
  capabilities = [\"create\", \"update\"]
} " | vault policy write policy_ssh-client -

3. ssh daemon setup to allow CA signed key authentication
The changes to the servers that must allow CA signed authentication are really modest.

First of all, the CA key that is obtained in step 2 must be transferred to the server. Currently I am doing this in the /etc/ssh directory, where some other keys are stored too, together with the other ssh settings.

The second thing that needs to be done, is the ssh daemon configuration file, /etc/ssh/sshd_config must be changed to include the setting TrustedUserCAKeys, which must be set to trusted-user-ca-key.pem file:

TrustedUserCAKeys trusted-user-ca-key.pem

After this change, the ssh daemon must be restarted to pick up the configuration change (systemctl restart ssh). This should not interrupt current sessions.

4. Enable a user by uploading its key into vault and assign the ssh policy
The next step is to have a user that must be provided access upload it’s certificate public key in vault and bound to the policy . This of course must be done by the vault administrator. This task does two things at the same time: a) upload the key (certificate=) and b) attach the policy policy_ssh-client (policies=):

$ vault write auth/cert/certs/frits.hoogland display_name="Frits Hoogland" certificate=@fritshoogland_pub.pem policies="policy_ssh-client"

Please mind only the public key is needed.
Also mind that at this point, no access is possible for the user at this point, vault has been configured to allow the user to be authenticated by the key pair for which the public key fritshoogland_pub.pem is uploaded, but no user public key is signed yet.

5. Let the user sign its key with the CA key using vault
In order to let a user sign its key, it can use its public and private certificate pair for authentication, and let vault sign the public key:

$ vault write -field=signed_key ssh/sign/sign-role public_key=@$HOME/.ssh/id_rsa.pub -client-cert=fritshoogland_pub.pem -client-key=fritshoogland_key.pem > ~/.ssh/signed-cert.pub

Now a signed key has been created, it can be used to log on to a server that has the CA key that signed this key set. To do this, simply specify ‘-i’ (for identity) a second time (after the private key):

$ ssh -i ~/.ssh/id_rsa -i ~/.ssh/signed-cert.pub vagrant@192.168.66.51
Last login: Sun Oct 13 13:09:03 2019 from 192.168.66.50

As you can see, the amount of work for a user needing access using this mechanism is really modest, it’s a single command to obtain/refresh the signed key. With signed keys that last 24 hours, you need to obtain a new signed key every day.

The audit log of vault will tell which certificate authenticated and shows the public ssh key that is signed, and the response, which contains the signed key and the serial of the signed key (use the ‘jq’ executable to format the json audit log; cat /var/log/vault/audit.log | jq):

{
  "time": "2019-10-13T13:05:36.524628543Z",
  "type": "response",
  "auth": {
    "client_token": "s.yr6lGhRxwK8ySc4cgEC6HBIi",
    "accessor": "w4Q5bVyabJ9c3hfgeG1gdyuS",
    "display_name": "cert-Test User",
    "policies": [
      "default",
      "policy_ssh-client"
    ],
    "token_policies": [
      "default",
      "policy_ssh-client"
    ],
    "metadata": {
      "authority_key_id": "f5:02:c8:54:6b:bd:36:66:1f:55:d2:4d:60:a8:0c:d0:19:32:e0:bb",
      "cert_name": "test_user",
      "common_name": "testuser",
      "serial_number": "12954687727334453172",
      "subject_key_id": "f5:02:c8:54:6b:bd:36:66:1f:55:d2:4d:60:a8:0c:d0:19:32:e0:bb"
    },
    "entity_id": "64cd1dd1-f94e-6370-8f8d-bc9ae68babf3",
    "token_type": "service"
  },
  "request": {
    "id": "c28e810a-8a52-1760-b346-1c4bd44e3800",
    "operation": "update",
    "client_token": "s.yr6lGhRxwK8ySc4cgEC6HBIi",
    "client_token_accessor": "w4Q5bVyabJ9c3hfgeG1gdyuS",
    "namespace": {
      "id": "root"
    },
    "path": "ssh-client/sign/sign-role",
    "data": {
      "-client-cert": "test_pub.pem",
      "-client-key": "test_key.pem",
      "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCue3QBya7lZLt2JxOwMTPQQF0NrV/ahCNXr/NX0iFkC6PDtSQZ00YN34JXviR8IL4NHuvW/MMXGJFMHk9Y6sXgY6coIkhan/DVhJlt+fSzUEdUXAWygR4Nsq8Bmk8a3YEU5lBjOfdHGHLM42lG3FmZpTdNDLMGaLkAvFjsYqklsT4mEfxjkBHeY3fbt5zKoPkNnLS3m/O4oFO0uwT6Qo8CjlN6lFibiFNpUC2t+2b0knfrFZn0Jc5u+4JdJoFMvh8xGckCL85r2hFSS85ENHEwLq3kKMb2V7AOW06RaneFC5LDp93q31aUWi8nc2xtKMTQDzf/qUcpgB+xhKW0Bejl test@localhost.localdomain\n"
    },
    "remote_address": "127.0.0.1"
  },
  "response": {
    "data": {
      "serial_number": "f11b7664d9c1ed4b",
      "signed_key": "ssh-rsa-cert-v01@openssh.com AAAAHHNzaC1yc2EtY2VydC12MDFAb3BlbnNzaC5jb20AAAAgmqm82e96fHybofd18OK9X42OSX9Y8sBReFxNtU0gSdoAAAADAQABAAABAQCue3QBya7lZLt2JxOwMTPQQF0NrV/ahCNXr/NX0iFkC6PDtSQZ00YN34JXviR8IL4NHuvW/MMXGJFMHk9Y6sXgY6coIkhan/DVhJlt+fSzUEdUXAWygR4Nsq8Bmk8a3YEU5lBjOfdHGHLM42lG3FmZpTdNDLMGaLkAvFjsYqklsT4mEfxjkBHeY3fbt5zKoPkNnLS3m/O4oFO0uwT6Qo8CjlN6lFibiFNpUC2t+2b0knfrFZn0Jc5u+4JdJoFMvh8xGckCL85r2hFSS85ENHEwLq3kKMb2V7AOW06RaneFC5LDp93q31aUWi8nc2xtKMTQDzf/qUcpgB+xhKW0Bejl8Rt2ZNnB7UsAAAABAAAAVXZhdWx0LWNlcnQtVGVzdCBVc2VyLWFlYTkwNGFiOTE3YjNlZGM1MzJjMDBhNDU5NDc1NDFmYTE5NmIwNjA5ZjJkZDdmNmY1MDI4YWJhZTBmODgwMWQAAAALAAAAB3ZhZ3JhbnQAAAAAXaMhAgAAAABdpHKgAAAAAAAAABIAAAAKcGVybWl0LXB0eQAAAAAAAAAAAAACFwAAAAdzc2gtcnNhAAAAAwEAAQAAAgEArPomMYoG/HajnbzfLVdFOjGP64lXS1+wdnG97C0glHHyvP7E8kcK8Iqt7PbCTY7hbpajF2Z/PTqAgp2DNtdvEftD4HKxUF7Qpa40fToBcj0SVcA/Ht4qfJ1dH8fOdnOCOL9/wUgUmOQlprwYbvvPkjX9Rg4kYkoxkBrT1ZLg5+0QTmly/44ZVphrsqyXk3vOydcnyK8MTd5IVZ0hLNTNx/cDBeCnLwBkg2cs1us5b6uRqbUchqjNP61eHyPCEJykhsFpRSFCdqVHU9gkynj00/6JWuNRBtP0DFiJPJ8oUI9BLUBeBQn5jEgfhh8obnZ2ih/M7LOHF1cYggAVSgtG+XQ+jOzYr39pvLbABYncXCATSW2M62F6bnFFMcCixx5vBwvhAXMiOJpENjmfmmaCpa17t4SmSs284taNmPa5Upq17zyy/QBofeCrz35qBuGfAlO7G9jGP+/tkTOv2lbMw+BJGRobaH/1uypkX7NpuG8rEDdht5xm9pr1Xwdb1iD632nKEkLDtOjrH2X/PR9k9EWPbBhF3HtAPzv/esKYVa7Cm6heLEVPRn5ZsBXJA2+4j3Kq2cwDVp5DFpEttAf4trMx3S4AP9rHwS02Y5zWdRT16HdYAwjfpY0+m0kAdNdKwStdxTTG4wgzjYN8dZSOZP2UIe1LA5bFm1HbBieCjysAAAIPAAAAB3NzaC1yc2EAAAIAqrEuTPFL9ULr/46Qx7kPCY292yxgqilLqlcqEmJ8fwFiukRk/w4wk+qgnLLAc72aPD4tUqjpw0xB3QaCmdh0YK6TAjQ+RC8hXYKz6OHZMlbFBSXRwa7poKWSs4YVu+M+WtQ9oibYpHrZcEj5N3mO+XHP/mbrEa/jZi9yvqwI4bJ4OY1ktH6bihLd7q79pejWq5c1+1ppswSyO2tcyJShGYb8V/UPmIRgqo2OvMjnrgTtF7VnPZHh+H/kSlB+6PiTgdQQDVSf72cUi9hGcgXCas71bAFeamq/fvoeB2dKfl7ZrhjGKE5Mx90G8gVGulz8a2kbMOgP3bjNvKlc6DfKiJuHbpfxyNn/9P/cvYFYONMFxup3H/3/rDVs4a4M+Qp9nDmalGe+4muwdlLdt6Y/dkG3WAbJvKvVPpXjFca234Y2gSAv3lJVqURHbaxkE1fus3gCmNtjRNcHA/rGQW/vnEvVjXLfRBHRyAdT+TY38iewG1tk7iEx6EKx77PgFtgMO01pQLYe94VfG2ynuBOlUfIDms/gm6jwVfo/PUR1hP/Q5vTMCNBt1RwgEwa0EWI7LhNApVE66FXDAJ6a4aUvulXc8KdWODytJzaHMhM5mpn88xNFeH7SK0aeEs4C4Fu4XQVrobm8eE0Xz9K7faRXCpdNtrtRvh8joU0H+GfVsHE=\n"
    }
  }
}

The important part here to link information in /var/log/secure to signed certificate is the serial.
In this case, the serial is f11b7664d9c1ed4b. You need to convert that hexadecimal number to decimal:

echo "ibase=16; F11B7664D9C1ED4B" | bc
17373610163033992523

(please mind I needed to install bc on Centos7, and the hexadecimal number must be specified in uppercase)

Now if we move over to the server where this key was used, we can simply search for ‘17373610163033992523’ in the /var/log/messages file:

# grep 17373610163033992523 /var/log/secure
Oct 13 13:09:03 localhost sshd[4959]: Accepted publickey for vagrant from 192.168.66.50 port 50936 ssh2: RSA-CERT ID vault-cert-Test User-aea904ab917b3edc532c00a45947541fa196b0609f2dd7f6f5028abae0f8801d (serial 17373610163033992523) CA RSA SHA256:Jcv3wpnbWWRlHDCRNqm6jfhB9qKnvCByBRIR4wr7CLI
Oct 13 13:09:09 localhost sshd[4986]: Accepted publickey for vagrant from 192.168.66.50 port 50938 ssh2: RSA-CERT ID vault-cert-Test User-aea904ab917b3edc532c00a45947541fa196b0609f2dd7f6f5028abae0f8801d (serial 17373610163033992523) CA RSA SHA256:Jcv3wpnbWWRlHDCRNqm6jfhB9qKnvCByBRIR4wr7CLI

The second session (at 13:09:09) used sshd process 4986. We can use the linux auditing facility to display all use of that process id:

# ausearch -p 4959 -i
type=USER_AUTH msg=audit(10/13/2019 13:09:09.152:905) : pid=4986 uid=root auid=unset ses=unset subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=key algo=ssh-rsa-cert-v01@openssh.com size=2048 fp=none rport=50938 acct=vagrant exe=/usr/sbin/sshd hostname=? addr=192.168.66.50 terminal=? res=success'
----
type=USER_ACCT msg=audit(10/13/2019 13:09:09.154:906) : pid=4986 uid=root auid=unset ses=unset subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct=vagrant exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=ssh res=success'
----
type=CRYPTO_KEY_USER msg=audit(10/13/2019 13:09:09.160:907) : pid=4986 uid=root auid=unset ses=unset subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=both spid=4987 suid=sshd rport=50938 laddr=192.168.66.51 lport=22  exe=/usr/sbin/sshd hostname=? addr=192.168.66.50 terminal=? res=success'
----
type=USER_AUTH msg=audit(10/13/2019 13:09:09.160:908) : pid=4986 uid=root auid=unset ses=unset subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=success acct=vagrant exe=/usr/sbin/sshd hostname=? addr=192.168.66.50 terminal=ssh res=success'
----
type=CRED_ACQ msg=audit(10/13/2019 13:09:09.160:909) : pid=4986 uid=root auid=unset ses=unset subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct=vagrant exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=ssh res=success'
----
type=LOGIN msg=audit(10/13/2019 13:09:09.160:910) : pid=4986 uid=root subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=unset auid=vagrant tty=(none) old-ses=4294967295 ses=29 res=yes
----
type=USER_ROLE_CHANGE msg=audit(10/13/2019 13:09:09.276:911) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='pam: default-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 selected-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=ssh res=success'
----
type=USER_START msg=audit(10/13/2019 13:09:09.296:912) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_selinux,pam_namespace,pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_lastlog acct=vagrant exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=ssh res=success'
----
type=USER_LOGIN msg=audit(10/13/2019 13:09:09.348:917) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=vagrant exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=/dev/pts/1 res=success'

What you see here is at first the process of the secure shell daemon working with the authentication of the process, before it’s actually providing shell access. This is noticeable by the session-id, ses, which unset. Once it has authenticated, it is given a true audit session number, which is 29. Now we can look at everything this session id, including changing user, using this session id:

# ausearch --session 29 -i
----
type=LOGIN msg=audit(10/13/2019 13:09:09.160:910) : pid=4986 uid=root subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=unset auid=vagrant tty=(none) old-ses=4294967295 ses=29 res=yes
----
type=USER_ROLE_CHANGE msg=audit(10/13/2019 13:09:09.276:911) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='pam: default-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 selected-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=ssh res=success'
----
type=USER_START msg=audit(10/13/2019 13:09:09.296:912) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_selinux,pam_namespace,pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_lastlog acct=vagrant exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=ssh res=success'
----
type=CRYPTO_KEY_USER msg=audit(10/13/2019 13:09:09.296:913) : pid=4989 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:22:dd:a8:71:eb:4d:44:5f:61:6a:4e:eb:55:9b:b5:f1:3c:bb:44:d2:3f:56:9d:a5:f8:3a:74:69:e4:02:4b:01 direction=? spid=4989 suid=root  exe=/usr/sbin/sshd hostname=? addr=? terminal=? res=success'
----
type=CRYPTO_KEY_USER msg=audit(10/13/2019 13:09:09.296:914) : pid=4989 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:77:fc:eb:26:0c:37:76:9d:b6:89:98:ca:5a:25:ad:d2:b9:c0:0b:01:4f:fb:e1:0d:a8:b8:45:41:56:68:ee:49 direction=? spid=4989 suid=root  exe=/usr/sbin/sshd hostname=? addr=? terminal=? res=success'
----
type=CRYPTO_KEY_USER msg=audit(10/13/2019 13:09:09.296:915) : pid=4989 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:9e:58:bc:ae:c9:68:4c:91:cc:a9:65:0a:a2:cd:e5:a8:62:40:14:22:79:80:52:da:0f:cd:78:87:f1:6c:d6:7f direction=? spid=4989 suid=root  exe=/usr/sbin/sshd hostname=? addr=? terminal=? res=success'
----
type=CRED_ACQ msg=audit(10/13/2019 13:09:09.296:916) : pid=4989 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct=vagrant exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=ssh res=success'
----
type=USER_LOGIN msg=audit(10/13/2019 13:09:09.348:917) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=vagrant exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=/dev/pts/1 res=success'
----
type=USER_START msg=audit(10/13/2019 13:09:09.348:918) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=vagrant exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=/dev/pts/1 res=success'
----
type=CRYPTO_KEY_USER msg=audit(10/13/2019 13:09:09.355:919) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:9e:58:bc:ae:c9:68:4c:91:cc:a9:65:0a:a2:cd:e5:a8:62:40:14:22:79:80:52:da:0f:cd:78:87:f1:6c:d6:7f direction=? spid=4990 suid=vagrant  exe=/usr/sbin/sshd hostname=? addr=? terminal=? res=success'
----
type=USER_END msg=audit(10/13/2019 13:58:38.973:922) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=vagrant exe=/usr/sbin/sshd hostname=? addr=? terminal=/dev/pts/1 res=success'
----
type=USER_LOGOUT msg=audit(10/13/2019 13:58:38.973:923) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=vagrant exe=/usr/sbin/sshd hostname=? addr=? terminal=/dev/pts/1 res=success'
----
type=CRYPTO_KEY_USER msg=audit(10/13/2019 13:58:38.982:924) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:9e:58:bc:ae:c9:68:4c:91:cc:a9:65:0a:a2:cd:e5:a8:62:40:14:22:79:80:52:da:0f:cd:78:87:f1:6c:d6:7f direction=? spid=4989 suid=vagrant  exe=/usr/sbin/sshd hostname=? addr=? terminal=? res=success'
----
type=CRYPTO_KEY_USER msg=audit(10/13/2019 13:58:38.983:925) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=both spid=4989 suid=vagrant rport=50938 laddr=192.168.66.51 lport=22  exe=/usr/sbin/sshd hostname=? addr=192.168.66.50 terminal=? res=success'
----
type=USER_END msg=audit(10/13/2019 13:58:38.992:926) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_selinux,pam_loginuid,pam_selinux,pam_namespace,pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_lastlog acct=vagrant exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=ssh res=success'
----
type=CRED_DISP msg=audit(10/13/2019 13:58:38.992:927) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct=vagrant exe=/usr/sbin/sshd hostname=192.168.66.50 addr=192.168.66.50 terminal=ssh res=success'
----
type=CRYPTO_KEY_USER msg=audit(10/13/2019 13:58:38.992:928) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:22:dd:a8:71:eb:4d:44:5f:61:6a:4e:eb:55:9b:b5:f1:3c:bb:44:d2:3f:56:9d:a5:f8:3a:74:69:e4:02:4b:01 direction=? spid=4986 suid=root  exe=/usr/sbin/sshd hostname=? addr=? terminal=? res=success'
----
type=CRYPTO_KEY_USER msg=audit(10/13/2019 13:58:38.992:929) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:77:fc:eb:26:0c:37:76:9d:b6:89:98:ca:5a:25:ad:d2:b9:c0:0b:01:4f:fb:e1:0d:a8:b8:45:41:56:68:ee:49 direction=? spid=4986 suid=root  exe=/usr/sbin/sshd hostname=? addr=? terminal=? res=success'
----
type=CRYPTO_KEY_USER msg=audit(10/13/2019 13:58:38.992:930) : pid=4986 uid=root auid=vagrant ses=29 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:9e:58:bc:ae:c9:68:4c:91:cc:a9:65:0a:a2:cd:e5:a8:62:40:14:22:79:80:52:da:0f:cd:78:87:f1:6c:d6:7f direction=? spid=4986 suid=root  exe=/usr/sbin/sshd hostname=? addr=? terminal=? res=success'

This shows the information that the linux audit facility logged for this session.

Conclusion
This is a solution for accountability in dynamic environments, where traditional solutions like LDAP would lesser easy fit in.
It works by using native secure shell daemon functionality, which is using an additional certificate that is used as “authority”, hence the name “certificate authority” (CA), which is set in the secure shell daemon as CA public certificate.
The essence is that an additional key is produced using the CA key, which is called a signed key. This key is linked with the CA public key, and therefore is authenticated by it. A single person should be the only owner of this signed key, because the signed key is what is what determines the identity.

Vault acts as a server where the signing and the administration of the signing is audited and automated so users can self-service generating CA signed keys, the only thing an administrator has to do is upload a certificate and grant the ssh policy, and remove the policy from a certificate of a person that should not get access anymore.

Because authentication and ssh functionality can be mounted multiple times in vault, you can create multiple groups of certificates which can use multiple CAs (using multiple mounts of the ssh functionality).

There is more functionality that can be set, allowed, disallowed etc. with CA signed keys.

%d bloggers like this: