The Oracle database log writer is the process that fundamentally influences database change performance. Under normal circumstances the log writer must persist the changes made to the blocks before the actual change is committed. Therefore, it’s vitally important to understand what the log writer is exactly doing. This is widely known by the Oracle database community.

The traditional method for looking at log writer performance is looking at the wait event ‘log file parallel write’ and the CPU time, and comparing that to the ‘log file sync’ alias “commit wait time”. If ‘log file parallel write’ and ‘log file sync’ roughly match, a commit is waiting on the log writer IO latency, if it isn’t then it’s unclear, and things get vague.

Now we get to the gist of this blogpost: since Oracle 12, there are additional statistics in V$SESSTAT and v$SYSSTAT that describe where the log writer spending it’s time on. However, and sadly, these are not documented by Oracle. The statistics are:

– redo write gather time
– redo write schedule time
– redo write finish time
– redo write time (usec)
– redo write total time

Probably ‘redo write time (usec)’ doesn’t belong to these statistics because the timing is as good as the same as ‘redo write finish time’. Also the statistic ‘redo write time (usec)’ is updated by ‘kcrfw_postprocess_write’ right after the main writing function (kcrfw_do_write), whilst the other redo write timing statistics are updated/made visible by ksu_dispatch_tac outside of the actual writing functions, at the approximate rate of every 3 seconds.

All these statistics contain cumulative timing figure in microseconds (usec).

All these timings start at the start of the function kcrfw_redo_write_driver. This is the main function that handles all the redo writing, including investigating the fixed area structure (kcrfs_) and the shared pool structures (made visible using X$KCRFSTRAND) to see if there are changes.

1. redo write gather time.
This times reading the structures that control allocation of the redo structures, validating foreground processes are done writing the change vectors and applying these to the blocks, updating the LWN SCN. Part of this also happens when there is nothing to write, but then the time is not added to this statistic, only if it found something to write.

2. redo write schedule time.
This includes the time of 1, but now it found something to write, entered the writing function (kcrfw_do_write), and updated the ksfd IO related structures for writing.

3. redo write finish time.
This includes the time of 1 and 2, but this additionally times all the IO related functions for writing the changes it found in the public redo strands, as indicated by the shared pool and fixed area structures.

4. redo write time (usec).
This probably is not part of this group of statistics, but it works in exactly the same way: it’s a cumulative figure, it counts time since the start of the kcrfw_redo_write_driver and includes a tiny bit more code, it times until it has left the kcrfw_do_write function and enters the kcrfw_postprocess_write function.

5. redo write total time.
This includes the time of 1, 2 and 3, but additionally times the kcrfw_post function, which performs the signalling of the foreground processes that have committed and wait for a semop() signal.

This means that the Oracle database since Oracle 12 provides the timing information to see where the log writer is spending it’s time on. This can prevent a lot of guesswork.

This post is about a change in how the time is measured for the event ‘log file parallel write’. This is important for the performance tuning of any change activity in an Oracle database, because with the default commit settings, a foreground session that commits changes waits in the wait event ‘log file sync’, which is a wait on logwriter activity, for which the wait event ‘log file parallel write’ always has been the indicator of the time spend on IO.

Log file sync
First things first: a foreground session normally waits on the wait event ‘log file sync’ when it commits waiting for its change vectors to be written to the online redologfile(s) by the logwriter. It is wrong to always assume a ‘log file sync’ will be present. If, somehow, the logwriter manages to increase the ON DISK SCN to or beyond the foreground session’s commit SCN, there will be no ‘log file sync’ wait event.

I found the absence of the ‘log file sync’ wait event by artificially slowing the committing session down. However, with the improvements in parallelism in the Oracle code, and even more with the improvements on the hardware layer for lowering write latency (especially with persistent memory), this might be a situation that is actually visible in the near future. This is true for at least version 11.2.0.4 and higher (probably earlier too, but these are versions that I verified).

Log file parallel write
However, this post is about the event that in a lot of cases is the culprit for the ‘log file sync’ time: it’s about the ‘log file parallel write’ wait event, which takes place in the logwriter (when the logwriter itself writes, it could delegate that to the logwriter worker processes, LGnn), which normally would be measured by Oracle database tuners to validate logwriter IO latency in case of foreground sessions showing high ‘log file sync’ waits indicating waiting on the logwriter.

I was looking at the logwriter code path using the ‘debug trace’ tool from intel pin tools and used Tanel Poder’s snapper using begin and end snapshots on the logwriter process on Linux, non ASM, Oracle 19.5. I put the logwriter into single logwriter mode (“_use_single_log_writer”=true) and performed redo, which is bound to be picked up by the logwriter.

There I noticed the statistics indicated a write to have been taking place, whilst there was no ‘log file parallel write’ wait event. Weird… So I dug deeper and looked at the functions taking place. These are the functions that I found: https://gitlab.com/snippets/1926207

The important bits are:
– In the function ksfdaio() the IO request is submitted:
ksfdaio>ksfdgo>ksfd_skgfqio>skgfqio>skgfr_lio_listio64>io_submit
– And in the function ksfdblock() the IO request result is read (“reaped”):
ksfdblock>ksfdpoll>ksfdwtio>skgfrwat>skgfrliopo>io_getevents

But no wait event is called *at all!*

This is a clear change from earlier versions, where the logwriter write would always yield a ‘log file parallel write’ wait event. But when did this change happen? To answer that question, I installed version 12.1.0.2 (no patches) to look at the code path and found: https://gitlab.com/snippets/1926211

The important bits are:
– The wait interface is invoked: kslwtbctx()
– In the function ksfdgo() the IO request is submitted:
ksfd_sbio>ksfdgo>ksfd_skgfqio>skgfqio>skgfr_lio_listio64>io_submit
– And in the function ksfdwtio() to IO request result is read (“reaped”):
ksfd_sbio>ksfdwtio>skgfrwat>skgfospo>skgfrliopo>io_getevents
– And the wait interface is ended: kslwtectx()

Ah! So a clear change! So when did this change happen? I installed 12.2.0.1 and found the same ‘waiting’ codepath, then installed 18.3, and found the waiting codepath. So the change probably happened in Oracle 19! So I installed 19.3, and found the waiting codepath once again!! So was it a change in PSU 19.4 or 19.5? I installed 19.5, and found the waiting codepath too.

O.M.G. Why am I seeing different behaviour between two databases on the exact same version? Well, I set some undocumented parameters, so I replicated these in my newly installed database…and it didn’t change.

What else is different? Well…one is a multi-tenant database, and the other isn’t…but surely that doesn’t change the IO codepath?? Of course this is easy to check, so I dropped the non-CDB database and created a CDB one, and…it now showed the same “non waiting” codepath.

So, apparently, using the oracle database in multi-tenant mode changes the IO codepath behaviour. I checked the codepath going down in versions, and it turns out this change appeared in 12.2.0.1.0, so essentially in Oracle 12.2 (Oracle 18 and 19 are actually Oracle 12.2.0.2 and Oracle 12.2.0.3). To be clear on this: I didn’t find this change in 12.1.0.2.

Does this mean the wait events ‘log file parallel write’ does not show up at all? No, if the call io_getevents after io_submit does not return all submitted IOs, it must wait for it, and at that point invoke the wait interface and then go into a blocking io_getevents call, so the actual waiting is timed. This is shown in this snippet: https://gitlab.com/snippets/1926212

This shows ksfdblock>ksfdpoll>ksfdwtio>skgfrwat>skgfrliopo>io_getevents, which returns 0 (what means that it didn’t found any IOs in the completion queue), which then returns up to the ksfdpoll function, invokes the wait interface, kslwtbctx, and then dives to io_getevents again, but now in blocking mode to wait for the IO.

I also replayed the tests on ASM, which shows the exact same (non-)waiting behaviour.

Conclusion
For Oracle versions 12.1.0.2 up to 19.5, the wait event ‘log file parallel write’ includes both the submission of the IO request (io_submit call) and the waiting for the submitted IOs to finish (io_getevents call). This means the waiting time of the event in this case is the total latency of the IO request, including operating system processing.

Starting from version 12.2, only if the multi-tenant option is used, this is changed, and the wait event ‘log file parallel write’ only is shown if the submitted IOs are not available after they are submitted, and thus the logwriter process has to wait for them. If the wait is shown, the time in the event is the actual waiting time for the IOs to finish from the perspective of the logwriter process, not the IO latency time.

Of course if you have multiple log groups on disks with different latencies, this wait event waits for all IOs to finish, which means it times the slowest IO of all submitted IOs.

This blogpost is about Oracle database and grid infrastructure software homes, which patches should be applied to which homes, and what it then looks like. This is fully documented by MyOracleSupport notes, but you will see that with version 18 and up this is unclear.

I keep a script-set that automatically installs and patches the Oracle database software and creates a database. This script-set is called vagrant-builder, and it can install any version with any PSU applied between 11.2.0.2 up to 19.5, which is the latest PSU of the latest version, with a few exceptions: for 11.2.0.2 and 11.2.0.3 I only created an install for the base version and the latest PSU for the database, and version 12.1.0.1 is left out entirely.

I recently reviewed my installs and verified everything is carried out correctly. First a simple overview of what I think should be applied on the database and grid infrastructure install:

Version  Grid               Database
-------- ------             ----------
11.2.0.2 -                  DB PSU
11.2.0.3 -                  DB PSU+OJVM
11.2.0.4 GI PSU+JDBC patch  DB PSU+OJVM
12.1.0.2 GI PSU+JDBC patch  DB PSU+OJVM
12.2.0.1 GI PSU             DB PSU+OJVM
18       GI PSU             DB PSU+OJVM
19       GI PSU             DB PSU+OJVM

(‘-‘ means not investigated)
My idea of what should be applied is based on MOS note 1929745.1: Oracle recommended patches.

Grid patches, JDBC patch
The JDBC patch for grid infrastructure 11.2.0.4 and 12.1.0.2 is a patch that updates java classes. Therefore the patch is a generic one, the java classes do not contain operating system dependent machine code. The table in the MOS note also show differences for the JDBC patch between = januari 2014 = july 2016 and higher. No surprise there. It’s especially easy because all versions in premier support as of the date of this blogpost do not need the JDBC patch.

Grid patches, GI PSU
For the GI PSU, there are a lot of patches that contain the GI PSU, because outside of the GI patch itself, there are also combination patches that for example contain both the GI and the DB PSU. I like to keep it as simple as I can. Therefore, I stick to MOS note 2118136.2: Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), bundle patches, Patchsets and Base releases, and look at the following:
– versions 11.2.0.4/12.1.0.2: Oracle Database PSU, SPU(CPU), Bundle Patches (Versions 12.1 & lower), version (11.2.0.4/12.1.0.2), GI PSU column.
– versions 12.2.0.1 and up: Oracle Database Updates, version (12.2.0.1,18.0.0.0,19.0.0.0), GI Update column.

Database patches, DB PSU
Here too there are multiple patches that can be used to apply the DB PSU, and I stick with MOS note 2118136.2: Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), bundle patches, Patchsets and Base releases, and look at the following:
– versions 11.2.0.4/12.1.0.2: Oracle Database PSU, SPU(CPU), Bundle Patches (Versions 12.1 & lower), version (11.2.0.4/12.1.0.2), PSU column.
– versions 12.2.0.1 and up: Oracle Database Updates, version (12.2.0.1,18.0.0.0,19.0.0.0), DB Update column.

Database patches, OJVM
There are multiple MOS documents talking about the database JavaVM patch, and there are multiple patches, but here I stick with MOS note 2118136.2: Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), bundle patches, Patchsets and Base releases once again, and look at the following:
– OJVM Update/PSU/Bundle Patches, 11.2.0.4/12.1.0.2/12.2.0.1/18.0.0.0/19.0.0.0, OJVM Update.

Okay. So we got the table above that is based on MOS note 1929745.1, and we got all the patches organised in MOS note 2118136.2. So that’s nice and simple, right?

Well, not entirely…

As I said, I was checking up on the latest PSU installs. This is the ‘opatch lspatches’ overview of 12.1.0.2 and 12.2.0.1:

-- 12.1.0.2
GI 190716
---------
23727148;
29509318;OCW PATCH SET UPDATE 12.1.0.2.190716 (29509318)
29494060;Database Patch Set Update : 12.1.0.2.190716 (29494060)
29423125;ACFS PATCH SET UPDATE 12.1.0.2.190716 (29423125)
26983807;WLM Patch Set Update: 12.1.0.2.180116 (26983807)
DB 190716
---------
29774383;Database PSU 12.1.0.2.190716, Oracle JavaVM Component (JUL2019)
29494060;Database Patch Set Update : 12.1.0.2.190716 (29494060)
-- 12.2.0.1
GI 191015
---------
30138470;Database Oct 2019 Release Update : 12.2.0.1.191015 (30138470)
30122828;ACFS OCT 2019 RELEASE UPDATE 12.2.0.1.0 (30122828)
30122814;OCW OCT 2019 RELEASE UPDATE 12.2.0.1.191015 (30122814)
30093408;TOMCAT RELEASE UPDATE 12.2.0.1.0(ID:190208.0920) (30093408)
26839277;DBWLM RELEASE UPDATE 12.2.0.1.0(ID:170913) (26839277
DB 191015
---------
30133625;OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)
30138470;Database Oct 2019 Release Update : 12.2.0.1.191015 (30138470)

For the database (DB), we see two patches, which is exactly what is expected.
– The database patch, which is called release update with one and patch set update with the other.
– The OJVM update which also named differently between the two versions, but very much recognisable as such.
I think it’s bad that the names vary, but this is totally expected.

For the grid infrastructure (GI), we see 5 patches in both situations, but these are not the same patches between the homes!
– The unnamed patch in the 12.1.0.2 home is the JDBC patch, which should only be applied to 12.1.0.2, not to higher versions.
– The OCW patch (oracle clusterware).
– The database patch.
– The ACFS patch. The version indication in the patch name changed.
– The WLM/DBWLM patch. Sadly the name changed, and the name with 12.2.0.1 is actually quite useless, I can’t tell the actual version, I have to look up the patch number.
– Starting from version 12.2.0.1, there is a tomcat installation in the grid home, as this patch indicates. The name here is not helpful because it doesn’t indicate the actual version, like with the DBWLM patch.

So, outside of in my opinion bad naming, and a weird inclusion of a competing product of Oracle (Tomcat versus Weblogic), this still follows the rules of logic.

Now let’s look at the same output for version 18 and 19:

--18
GI 18.8
30116128;ACFS RELEASE UPDATE 18.8.0.0.0 (30116128)
30113775;OCW RELEASE UPDATE 18.8.0.0.0 (30113775)
30112122;Database Release Update : 18.8.0.0.191015 (30112122)
30093398;TOMCAT RELEASE UPDATE 18.0.0.0.0 (30093398)
28655963;DBWLM RELEASE UPDATE 18.4.0.0.0 (28655963)
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
DB 18.8
30133603;OJVM RELEASE UPDATE: 18.8.0.0.191015 (30133603)
30112122;Database Release Update : 18.8.0.0.191015 (30112122)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
--19
GI 19.5
30125133;Database Release Update : 19.5.0.0.191015 (30125133)
30122167;ACFS RELEASE UPDATE 19.5.0.0.0 (30122167)
30122149;OCW RELEASE UPDATE 19.5.0.0.0 (30122149)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)
DB 19.5
30128191;OJVM RELEASE UPDATE: 19.5.0.0.191015 (30128191)
30125133;Database Release Update : 19.5.0.0.191015 (30125133)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

I must say that the naming in general looks more consistent, that is a good thing!

For the database (DB) we see THREE patches (I suspected 2):
– The database release update patch, which nicely reports its version.
– The OVM patch, which also reports its version.
– This is weird. One of the grid infrastructure patches, the “OCW” patch, is applied to the database home. In fact, this is applied to the base release. Because it’s not a patch that is documented to be needed to be applied, this will sit at this version and never be updated. However, this unexpected patch is consistently applied to the base release for both version 18 and 19.

For the grid home (GI) we see an inconsistent number of patches (!) between 18.8 and 19.5. This is partly as expected, but I found an weird patch applied too. Let’s go over the patches:
– The ACFS patch is totally expected and appropriately named.
– The OCW patch is expected and appropriately named too.
– The Database Release Update patch is expected and appropriately named.
– The TOMCAT patch is expected. I don’t understand why it can’t have the RU numbering, but at least it’s consistent between 18 and 19.
– The DBWLM patch is only applied to the 18 home. As far as I understand, this is how it’s supposed to be, DBWLM is not regularly updated like the other ones above, so it’s okay to have an older version of it with the other patches, and if there isn’t a patch to apply, it can simply not be there, like with version 19 in this case.
– The OJVM patch puzzles me. I don’t know what to think of it. Also, it’s only applied to the base release of version 18, not to the base release of version 19. This, very much like the OCW patch, will never be updated. But I just don’t understand, this patches the java virtual machine in the database, which for GI is the ASM instance, for which, as far as I know, the java virtual machine isn’t used.

Conclusion.
The naming of the patches as visible with “opatch lspatches” has certainly improved with version 18 and higher. Still it would be helpful if the grid infrastructure tomcat patch would follow the same naming of the other patches.

I am in doubts about the inclusion of two patches in the base releases of versions 18 and 19:
The OJVM patch inclusion in the base release of grid infrastructure of version 18 only.
The OCW patch inclusion in the base release of the database of version 18 and 19.

Addendum.
After debating this on twitter and with my colleagues, I found that my OCW assumptions were incorrect. The grid infrastructure patch versions 12.2 and up reasonably clearly describes that the OCW sub-patch that is part of the GI PSU/RU patch should be applied to the database home too (if cluster ware is used for that home). So that means that if you got another version of the database home than the grid infrastructure home and it is used with grid infrastructure, you should download the grid infrastructure PSU/RU patch and apply the OCW sub patch to the oracle database home, because the OCW patch is not in the database PSU/RU patch.
Because the database home patch itself is also in the grid infrastructure PSU/RU patch, I see no reason to bother downloading the database patch, and now only download and use the grid infrastructure PSU/RU patch, because that contains the database home patch as well as the OCW patch.

Addendum 2.
The OCW patch being installed into the grid infrastructure home and additionally in the database home turns out to be a change with PSU 11.2.0.4.5. Before that, the OCW patch did exist with the grid infrastructure home patch, but couldn’t be applied to the database home. This change was documented in the patch readme, but wasn’t really heavily marketed by Oracle.

Addendum 3.
Despite OCW being documented as being additionally applied to the database home for 11.2.0.4(.5) and higher, in my tests it was not possible to apply the 11.2.0.4 OCW patch to a database home for any 11.2.0.4 PSU; opatch fails with a dependency problem. It does succeed starting from 12.1.0.2 (12.1.0.1 not tested) and higher.

I have to say that when looking in the 11.2.0.4 OCW patch metadata, it says ‘rac’, so the OCW patch might succeed when the home is explicitly installed for a cluster database. I tested with grid infrastructure installed for a single machine (“siha”).

This blogpost is about an inconsistency I found in the X$ tables X$KQFTA and X$KQFCO. This is very specific. If you don’t care about that, you can skip this post.

The Oracle database’s “dynamic performance views” are views that are prefixed with “GV$” and “V$”. The “G” with “GV$” stands for “global” and gets you the results from its “V$” equivalent for all instances. In most cases, but that’s not the purpose of this blog. In most cases, the “V$” version simply is the “GV$” view with inst_id (instance id, used in RAC to specify the instance id) set to the current instance. I’ll refer to both simply as “V$” for simplicity.

The “V$” views are mostly build on top of “X$” tables. The “X$” tables are Oracle internal tables, and not officially supported, therefore you should use the “V$” views.

One of the things that certain “X$” tables expose is information about the “X$” tables and their columns theirselves. These “X$” tables are visible in “X$KQFTA” (kernel query fixed tables tables) for the “X$” tables, and “X$KQFCO” (kernel query fixed tables columns) for the columns of the “X$” tables. I learned about these tables a long time ago from Kyle Hailey, Anjo Kolk and James Morle regarding direct memory attachment for wait events, the reason being the ability to find the memory address for certain “X$” tables to fetch the data these hold directly. We are talking about the 1990s here. In fact, Kyle later put the history of how he learned about it in the “Oracle insights: tales of the oaktable” book.

I gather information about every PSU of every Oracle database version on linux once it comes out. This is how I am able to produce the posts about the difference between a current and a previous PSU. One of the things that I gather is information about X$ tables and their columns. In order to obtain the columns, I use X$KQFTA and X$KQFCO and join these in the way that is publicly known:

select t.kqftanam, c.kqfconam
from x$kqfta t, x$kqfco c
where t.inst_id = 1
and c.inst_id = 1
and t.indx = c.kqfcotab;

And this worked well…

…until I produced a difference report of versions 12.1.0.2.191015 and 12.1.0.2.190716, where the report reported the following difference:

x$ tables columns unique to 12.1.0.2.190716 versus 12.1.0.2.191015

NAME                                                                                                 COLUMN_NAME
---------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------------
X$KQFTVRTTST0                                                                                        ADDR
X$KQFTVRTTST0                                                                                        CHGROWID
X$KQFTVRTTST0                                                                                        CHGROWOP
X$KQFTVRTTST0                                                                                        COL0_0
X$KQFTVRTTST0                                                                                        COL0_1
X$KQFTVRTTST0                                                                                        COL0_2
X$KQFTVRTTST0                                                                                        COL0_3
X$KQFTVRTTST0                                                                                        COL0_4
...and so on...

This could happen, these are columns that seem to have vanished with 12.1.0.2.191015. However, I decided to check it, although I haven’t got access to the 12.1.0.2.191015 PSU, because it’s a PSU version only available for paid extended support.

I looked at the columns of X$KQFTVRTST0 in my current database, which happens to be an ancient 11.2.0.2 database. However, much to my surprise, this gave me radically different columns:

SYS@o112 AS SYSDBA> desc X$KQFTVRTTST0
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 COLA                                                                                                                       VARCHAR2(64)
 COLB                                                                                                                       NUMBER

Okay, something weird is going on here.

First let’s look in which versions X$KQFTVRTTST0 is present:

ORAVER@o112 > @x_table_in_versions.sql

DATABASE_VERSION     NAME
-------------------- --------------------------------------------------------------------------------------------------------------------------------
11.2.0.1.0           X$KQFTVRTTST0
11.2.0.2.0           X$KQFTVRTTST0
11.2.0.2.12          X$KQFTVRTTST0
11.2.0.3.0           X$KQFTVRTTST0
11.2.0.3.15          X$KQFTVRTTST0
11.2.0.4.0           X$KQFTVRTTST0
11.2.0.4.1           X$KQFTVRTTST0
11.2.0.4.160119      X$KQFTVRTTST0
11.2.0.4.160419      X$KQFTVRTTST0
11.2.0.4.160719      X$KQFTVRTTST0
11.2.0.4.161018      X$KQFTVRTTST0
11.2.0.4.170418      X$KQFTVRTTST0
11.2.0.4.170718      X$KQFTVRTTST0
11.2.0.4.170814      X$KQFTVRTTST0
11.2.0.4.180116      X$KQFTVRTTST0
11.2.0.4.180417      X$KQFTVRTTST0
11.2.0.4.180717      X$KQFTVRTTST0
11.2.0.4.190115      X$KQFTVRTTST0
11.2.0.4.191015      X$KQFTVRTTST0
11.2.0.4.2           X$KQFTVRTTST0
11.2.0.4.3           X$KQFTVRTTST0
11.2.0.4.4           X$KQFTVRTTST0
11.2.0.4.5           X$KQFTVRTTST0
11.2.0.4.6           X$KQFTVRTTST0
11.2.0.4.7           X$KQFTVRTTST0
11.2.0.4.8           X$KQFTVRTTST0
12.1.0.2.0           X$KQFTVRTTST0
12.1.0.2.1           X$KQFTVRTTST0
12.1.0.2.160119      X$KQFTVRTTST0
12.1.0.2.160419      X$KQFTVRTTST0
12.1.0.2.160719      X$KQFTVRTTST0
12.1.0.2.161018      X$KQFTVRTTST0
12.1.0.2.170117      X$KQFTVRTTST0
12.1.0.2.170418      X$KQFTVRTTST0
12.1.0.2.170718      X$KQFTVRTTST0
12.1.0.2.170814      X$KQFTVRTTST0
12.1.0.2.171017      X$KQFTVRTTST0
12.1.0.2.180116      X$KQFTVRTTST0
12.1.0.2.180417      X$KQFTVRTTST0
12.1.0.2.181016      X$KQFTVRTTST0
12.1.0.2.190115      X$KQFTVRTTST0
12.1.0.2.190416      X$KQFTVRTTST0
12.1.0.2.190716      X$KQFTVRTTST0
12.1.0.2.191015      X$KQFTVRTTST0
12.1.0.2.3           X$KQFTVRTTST0
12.1.0.2.4           X$KQFTVRTTST0
12.1.0.2.5           X$KQFTVRTTST0
12.2.0.1.0           X$KQFTVRTTST0
12.2.0.1.170620      X$KQFTVRTTST0
12.2.0.1.170718      X$KQFTVRTTST0
12.2.0.1.170814      X$KQFTVRTTST0
12.2.0.1.171017      X$KQFTVRTTST0
12.2.0.1.180116      X$KQFTVRTTST0
12.2.0.1.180417      X$KQFTVRTTST0
12.2.0.1.180717      X$KQFTVRTTST0
12.2.0.1.181016      X$KQFTVRTTST0
12.2.0.1.190115      X$KQFTVRTTST0
12.2.0.1.190416      X$KQFTVRTTST0
12.2.0.1.190716      X$KQFTVRTTST0
12.2.0.1.191015      X$KQFTVRTTST0
18.3                 X$KQFTVRTTST0
18.4                 X$KQFTVRTTST0
18.5                 X$KQFTVRTTST0
18.6                 X$KQFTVRTTST0
18.7                 X$KQFTVRTTST0
18.8                 X$KQFTVRTTST0
19.2                 X$KQFTVRTTST0
19.3                 X$KQFTVRTTST0
19.4                 X$KQFTVRTTST0
19.5                 X$KQFTVRTTST0

So, X$KQFVRTTST0 appeared with 11.2.0.1. Now let’s take a look at how the columns look like in X$KQFVRTTST0 looks in version 11.2.0.1:

ORAVER@o112 > @x_table_cols_in_versions.sql

DATABASE_VERSION	       TABLE_NAME		      COLUMN_NAME
------------------------------ ------------------------------ ----------------------------------------
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_0
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_0
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_0
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_0
11.2.0.1.0		       X$KQFTVRTTST0		      CHGROWOP
11.2.0.1.0		       X$KQFTVRTTST0		      CHGROWID
11.2.0.1.0		       X$KQFTVRTTST0		      COL_OPERATION
11.2.0.1.0		       X$KQFTVRTTST0		      INST_ID
11.2.0.1.0		       X$KQFTVRTTST0		      INDX
11.2.0.1.0		       X$KQFTVRTTST0		      ADDR

These are the columns which were shown in the difference report that said they were removed. Now let’s look at the columns in version 19.5:

ORAVER@o112 > @x_table_cols_in_versions.sql

DATABASE_VERSION               TABLE_NAME                     COLUMN_NAME
------------------------------ ------------------------------ ----------------------------------------
19.5                           X$KQFTVRTTST0                  COL3_9
19.5                           X$KQFTVRTTST0                  COL3_8
19.5                           X$KQFTVRTTST0                  COL3_7
19.5                           X$KQFTVRTTST0                  COL3_6
19.5                           X$KQFTVRTTST0                  COL3_5
19.5                           X$KQFTVRTTST0                  COL3_4
19.5                           X$KQFTVRTTST0                  COL3_3
19.5                           X$KQFTVRTTST0                  COL3_2
19.5                           X$KQFTVRTTST0                  COL3_1
19.5                           X$KQFTVRTTST0                  COL3_0
19.5                           X$KQFTVRTTST0                  COL2_9
19.5                           X$KQFTVRTTST0                  COL2_8
19.5                           X$KQFTVRTTST0                  COL2_7
19.5                           X$KQFTVRTTST0                  COL2_6
19.5                           X$KQFTVRTTST0                  COL2_5
19.5                           X$KQFTVRTTST0                  COL2_4
19.5                           X$KQFTVRTTST0                  COL2_3
19.5                           X$KQFTVRTTST0                  COL2_2
19.5                           X$KQFTVRTTST0                  COL2_1
19.5                           X$KQFTVRTTST0                  COL2_0
19.5                           X$KQFTVRTTST0                  COL1_9
19.5                           X$KQFTVRTTST0                  COL1_8
19.5                           X$KQFTVRTTST0                  COL1_7
19.5                           X$KQFTVRTTST0                  COL1_6
19.5                           X$KQFTVRTTST0                  COL1_5
19.5                           X$KQFTVRTTST0                  COL1_4
19.5                           X$KQFTVRTTST0                  COL1_3
19.5                           X$KQFTVRTTST0                  COL1_2
19.5                           X$KQFTVRTTST0                  COL1_1
19.5                           X$KQFTVRTTST0                  COL1_0
19.5                           X$KQFTVRTTST0                  COL0_9
19.5                           X$KQFTVRTTST0                  COL0_8
19.5                           X$KQFTVRTTST0                  COL0_7
19.5                           X$KQFTVRTTST0                  COL0_6
19.5                           X$KQFTVRTTST0                  COL0_5
19.5                           X$KQFTVRTTST0                  COL0_4
19.5                           X$KQFTVRTTST0                  COL0_3
19.5                           X$KQFTVRTTST0                  COL0_2
19.5                           X$KQFTVRTTST0                  COL0_1
19.5                           X$KQFTVRTTST0                  COL0_0
19.5                           X$KQFTVRTTST0                  CHGROWOP
19.5                           X$KQFTVRTTST0                  CHGROWID
19.5                           X$KQFTVRTTST0                  COL_OPERATION
19.5                           X$KQFTVRTTST0                  CON_ID
19.5                           X$KQFTVRTTST0                  INST_ID
19.5                           X$KQFTVRTTST0                  INDX
19.5                           X$KQFTVRTTST0                  ADDR

These are exactly the same! So the columns for X$KQFTVRTTST0 that I obtained from the database are the same for all versions in my data, but when I look in an actual database using ‘describe’ I see ADDR, INDX, INST_ID and COLA and COLB. So my data seems to be wrong!

My first reaction was to change the columns for X$KQFTVRTST0 for every version to correct this. But then I realised that I had to investigate this deeper to understand the actual problem, there might be more X$ tables involved.

The first thing to do, is to see if I can reproduce the issue in my database. This can be quite easily be done by describing X$KQFTVRTTST0 to see the columns, and then query these using the above query that demonstrates the join between X$KQFTA and X$KQFCO by adding a filter on ‘X$KQFTVRTST0’:

SYS@o112 AS SYSDBA> desc X$KQFTVRTTST0
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 COLA                                                                                                                       VARCHAR2(64)
 COLB                                                                                                                       NUMBER
SYS@o112 AS SYSDBA> l
  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and kqftanam = 'X$KQFTVRTTST0'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$KQFTVRTTST0                  ADDR
X$KQFTVRTTST0                  INDX
X$KQFTVRTTST0                  INST_ID
X$KQFTVRTTST0                  COL_OPERATION
X$KQFTVRTTST0                  CHGROWID
X$KQFTVRTTST0                  CHGROWOP
X$KQFTVRTTST0                  COL0_0
X$KQFTVRTTST0                  COL0_1
X$KQFTVRTTST0                  COL0_2
X$KQFTVRTTST0                  COL0_3
X$KQFTVRTTST0                  COL0_4
X$KQFTVRTTST0                  COL0_5
X$KQFTVRTTST0                  COL0_6
X$KQFTVRTTST0                  COL0_7
X$KQFTVRTTST0                  COL0_8
X$KQFTVRTTST0                  COL0_9
X$KQFTVRTTST0                  COL1_0
X$KQFTVRTTST0                  COL1_1
X$KQFTVRTTST0                  COL1_2
X$KQFTVRTTST0                  COL1_3
X$KQFTVRTTST0                  COL1_4
X$KQFTVRTTST0                  COL1_5
X$KQFTVRTTST0                  COL1_6
X$KQFTVRTTST0                  COL1_7
X$KQFTVRTTST0                  COL1_8
X$KQFTVRTTST0                  COL1_9
X$KQFTVRTTST0                  COL2_0
X$KQFTVRTTST0                  COL2_1
X$KQFTVRTTST0                  COL2_2
X$KQFTVRTTST0                  COL2_3
X$KQFTVRTTST0                  COL2_4
X$KQFTVRTTST0                  COL2_5
X$KQFTVRTTST0                  COL2_6
X$KQFTVRTTST0                  COL2_7
X$KQFTVRTTST0                  COL2_8
X$KQFTVRTTST0                  COL2_9
X$KQFTVRTTST0                  COL3_0
X$KQFTVRTTST0                  COL3_1
X$KQFTVRTTST0                  COL3_2
X$KQFTVRTTST0                  COL3_3
X$KQFTVRTTST0                  COL3_4
X$KQFTVRTTST0                  COL3_5
X$KQFTVRTTST0                  COL3_6
X$KQFTVRTTST0                  COL3_7
X$KQFTVRTTST0                  COL3_8
X$KQFTVRTTST0                  COL3_9

Bingo! Describe tells me about COLA and COLB, which is the actual situation, and querying X$KQFTA and X$KQFCO gives me different, wrong columns.

First let’s investigate if we can find the correct columns in X$KQFCO, and see what the difference is between X$KQFTA.INDX and X$KQFCO.KQFCOTAB:

SYS@o112 AS SYSDBA> select indx from x$kqfta where kqftanam = 'X$KQFTVRTTST0';

      INDX
----------
       841

SYS@o112 AS SYSDBA> select kqfcotab from x$kqfco where kqfconam = 'COLA';

  KQFCOTAB
----------
       842

Aha! So the join is off by one! So, that must mean that if I decrease KQFCOTAB by one, I do get the correct output:

SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab-1
  6* and kqftanam = 'X$KQFTVRTTST0'
  7  /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$KQFTVRTTST0                  ADDR
X$KQFTVRTTST0                  INDX
X$KQFTVRTTST0                  INST_ID
X$KQFTVRTTST0                  COLA
X$KQFTVRTTST0                  COLB

Bingo! The next thing to do is see if this is only a problem for X$KQFTVRTTST0, or a bigger problem. Let’s take the next table based on X$KQFTA.INDX, and see if this shows the same problem:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 842;

KQFTANAM
------------------------------
X$DIAG_ADR_CONTROL
SYS@o112 AS SYSDBA> desc X$DIAG_ADR_CONTROL
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 ADR_PATH_IDX                                                                                                               VARCHAR2(445)
 ADR_HOME                                                                                                                   VARCHAR2(445)
 ADRID                                                                                                                      NUMBER
 SHORTP_POLICY                                                                                                              NUMBER
 LONGP_POLICY                                                                                                               NUMBER
 LAST_MOD_TIME                                                                                                              TIMESTAMP(9) WITH TIME ZONE
 LAST_AUTOPRG_TIME                                                                                                          TIMESTAMP(9) WITH TIME ZONE
 LAST_MANUPRG_TIME                                                                                                          TIMESTAMP(9) WITH TIME ZONE
 ADRDIR_VERSION                                                                                                             NUMBER
 ADRSCHM_VERSION                                                                                                            NUMBER
 ADRSCHMV_SUMMARY                                                                                                           NUMBER
 ADRALERT_VERSION                                                                                                           NUMBER
 CREATE_TIME                                                                                                                TIMESTAMP(9) WITH TIME ZONE
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1   select t.kqftanam, c.kqfconam
  2   from x$kqfta t, x$kqfco c
  3   where t.inst_id = 1
  4   and c.inst_id = 1
  5   and t.indx = c.kqfcotab
  6*  and kqftanam = 'X$DIAG_ADR_CONTROL'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_ADR_CONTROL             ADDR
X$DIAG_ADR_CONTROL             INDX
X$DIAG_ADR_CONTROL             INST_ID
X$DIAG_ADR_CONTROL             COLA
X$DIAG_ADR_CONTROL             COLB

Clearly the columns are wrong, these are the columns of X$KQFTVRTTST0! Let’s subtract X$KQFCO.KQFCOTAB by one:

SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1   select t.kqftanam, c.kqfconam
  2   from x$kqfta t, x$kqfco c
  3   where t.inst_id = 1
  4   and c.inst_id = 1
  5   and t.indx = c.kqfcotab-1
  6*  and kqftanam = 'X$DIAG_ADR_CONTROL'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_ADR_CONTROL             ADDR
X$DIAG_ADR_CONTROL             INDX
X$DIAG_ADR_CONTROL             INST_ID
X$DIAG_ADR_CONTROL             ADR_PATH_IDX
X$DIAG_ADR_CONTROL             ADR_HOME
X$DIAG_ADR_CONTROL             ADRID
X$DIAG_ADR_CONTROL             SHORTP_POLICY
X$DIAG_ADR_CONTROL             LONGP_POLICY
X$DIAG_ADR_CONTROL             LAST_MOD_TIME
X$DIAG_ADR_CONTROL             LAST_AUTOPRG_TIME
X$DIAG_ADR_CONTROL             LAST_MANUPRG_TIME
X$DIAG_ADR_CONTROL             ADRDIR_VERSION
X$DIAG_ADR_CONTROL             ADRSCHM_VERSION
X$DIAG_ADR_CONTROL             ADRSCHMV_SUMMARY
X$DIAG_ADR_CONTROL             ADRALERT_VERSION
X$DIAG_ADR_CONTROL             CREATE_TIME

And this is correct again!

Now let’s verify the highest X$KQFTA.INDX number in the version of this database:

SYS@o112 AS SYSDBA> select max(indx) from x$kqfta;

 MAX(INDX)
----------
       927

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 927;

KQFTANAM
------------------------------
X$DIAG_VTEST_EXISTS

SYS@o112 AS SYSDBA> desc X$DIAG_VTEST_EXISTS
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 ADR_PATH_IDX                                                                                                               VARCHAR2(445)
 ADR_HOME                                                                                                                   VARCHAR2(445)
 PROBLEM_ID                                                                                                                 NUMBER
 PROBLEM_KEY                                                                                                                VARCHAR2(552)
 FIRST_INCIDENT                                                                                                             NUMBER
 FIRSTINC_TIME                                                                                                              TIMESTAMP(9) WITH TIME ZONE
 LAST_INCIDENT                                                                                                              NUMBER
 LASTINC_TIME                                                                                                               TIMESTAMP(9) WITH TIME ZONE
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and kqftanam = 'X$DIAG_VTEST_EXISTS'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_VTEST_EXISTS            ADDR
X$DIAG_VTEST_EXISTS            INDX
X$DIAG_VTEST_EXISTS            INST_ID
X$DIAG_VTEST_EXISTS            ADR_PATH_IDX
X$DIAG_VTEST_EXISTS            ADR_HOME
X$DIAG_VTEST_EXISTS            INCIDENT_ID
X$DIAG_VTEST_EXISTS            PROBLEM_ID
X$DIAG_VTEST_EXISTS            CREATE_TIME
X$DIAG_VTEST_EXISTS            CLOSE_TIME
X$DIAG_VTEST_EXISTS            STATUS
X$DIAG_VTEST_EXISTS            FLAGS
X$DIAG_VTEST_EXISTS            FLOOD_CONTROLLED
X$DIAG_VTEST_EXISTS            ERROR_FACILITY
X$DIAG_VTEST_EXISTS            ERROR_NUMBER
X$DIAG_VTEST_EXISTS            ERROR_ARG1
X$DIAG_VTEST_EXISTS            ERROR_ARG2
X$DIAG_VTEST_EXISTS            ERROR_ARG3
X$DIAG_VTEST_EXISTS            ERROR_ARG4
X$DIAG_VTEST_EXISTS            ERROR_ARG5
X$DIAG_VTEST_EXISTS            ERROR_ARG6
X$DIAG_VTEST_EXISTS            ERROR_ARG7
X$DIAG_VTEST_EXISTS            ERROR_ARG8
X$DIAG_VTEST_EXISTS            SIGNALLING_COMPONENT
X$DIAG_VTEST_EXISTS            SIGNALLING_SUBCOMPONENT
X$DIAG_VTEST_EXISTS            SUSPECT_COMPONENT
X$DIAG_VTEST_EXISTS            SUSPECT_SUBCOMPONENT
X$DIAG_VTEST_EXISTS            ECID
X$DIAG_VTEST_EXISTS            IMPACT
X$DIAG_VTEST_EXISTS            IMPACT_STR
X$DIAG_VTEST_EXISTS            STATUS_STR
X$DIAG_VTEST_EXISTS            ERROR_ARG9
X$DIAG_VTEST_EXISTS            ERROR_ARG10
X$DIAG_VTEST_EXISTS            ERROR_ARG11
X$DIAG_VTEST_EXISTS            ERROR_ARG12

So that’s wrong, let’s try subtracting by one:

SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab-1
  6* and kqftanam = 'X$DIAG_VTEST_EXISTS'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_VTEST_EXISTS            ADDR
X$DIAG_VTEST_EXISTS            INDX
X$DIAG_VTEST_EXISTS            INST_ID
X$DIAG_VTEST_EXISTS            ADR_PATH_IDX
X$DIAG_VTEST_EXISTS            ADR_HOME
X$DIAG_VTEST_EXISTS            PROBLEM_ID
X$DIAG_VTEST_EXISTS            PROBLEM_KEY
X$DIAG_VTEST_EXISTS            FIRST_INCIDENT
X$DIAG_VTEST_EXISTS            FIRSTINC_TIME
X$DIAG_VTEST_EXISTS            LAST_INCIDENT
X$DIAG_VTEST_EXISTS            LASTINC_TIME

And correct again!

So at least starting from X$KQFTA.INDX = 841 up to the highest INDX number, 927, X$KQFCO.KQFCOTAB is off by one.

Starting from there, I decided to test INDX = 1, which was consistent with KQFCOTAB. Then I decided to go down in INDX number starting from 841 being the number of X$KQFTVRTTST0. In my database number 840 is X$KTCNQROW, which was off by one, then I found something weird:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 839;

KQFTANAM
------------------------------
X$SKGXP_MISC

SYS@o112 AS SYSDBA> desc X$SKGXP_MISC
ERROR:
ORA-00942: table or view does not exist
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1      select t.kqftanam, c.kqfconam
  2      from x$kqfta t, x$kqfco c
  3      where t.inst_id = 1
  4      and c.inst_id = 1
  5      and t.indx = c.kqfcotab
  6*    and kqftanam = 'X$SKGXP_MISC'
SYS@o112 AS SYSDBA> /

no rows selected

So the X$ view X$SKGXP_MISC exists in X$KQFTA, but doesn’t exist in reality, nor in X$KQFCO. Please mind this is a separate, different issue from the X$KQFCO.KQFCOTAB inconsistency this blogpost is about. My current thinking is the issue of existing in X$KQFTA and not queryable and (rightfully because of that), not available in X$KQFCO, might be either just a screwup/bug, or the X$ tables being reserved in the X$ table array, but not actually created, maybe because of setting during compilation. Just a hunch, I don’t know.

The same issue of being visible in X$KQFTA and not existing is true for:
838, X$SKGXP_CONNECTION
837, X$SKGXP_PORT
836, X$KSXP_STATS

INDX number 835 is X$KSXPTESTTBL. This table has the X$KQFCO.KQFCOTAB off by one issue:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 835;

KQFTANAM
------------------------------
X$KSXPTESTTBL
SYS@o112 AS SYSDBA> desc X$KSXPTESTTBL
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 KSXPTESTF1                                                                                                                 NUMBER
 KSXPTESTF2                                                                                                                 VARCHAR2(10)
 KSXPTESTF3                                                                                                                 NUMBER
 KSXPTESTF4                                                                                                                 NUMBER
 KSXPTESTF5                                                                                                                 RAW(8)
 KSXPTESTF6                                                                                                                 DATE
 KSXPTESTF7                                                                                                                 TIMESTAMP(3)
 KSXPTESTF8                                                                                                                 TIMESTAMP(3) WITH TIME ZONE
 KSXPTESTF9                                                                                                                 VARCHAR2(10)
 KSXPTESTF10                                                                                                                RAW(2)
 KSXPTESTF11                                                                                                                RAW(10)
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and t.kqftanam = 'X$KSXPTESTTBL'
SYS@o112 AS SYSDBA> /

no rows selected
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab-1
  6* and t.kqftanam = 'X$KSXPTESTTBL'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$KSXPTESTTBL                  ADDR
X$KSXPTESTTBL                  INDX
X$KSXPTESTTBL                  INST_ID
X$KSXPTESTTBL                  KSXPTESTF1
X$KSXPTESTTBL                  KSXPTESTF2
X$KSXPTESTTBL                  KSXPTESTF3
X$KSXPTESTTBL                  KSXPTESTF4
X$KSXPTESTTBL                  KSXPTESTF5
X$KSXPTESTTBL                  KSXPTESTF6
X$KSXPTESTTBL                  KSXPTESTF7
X$KSXPTESTTBL                  KSXPTESTF8
X$KSXPTESTTBL                  KSXPTESTF9
X$KSXPTESTTBL                  KSXPTESTF10
X$KSXPTESTTBL                  KSXPTESTF11

After X$KSXPTESTTBL, INDX number 834 is X$CELL_NAME:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 834;

KQFTANAM
------------------------------
X$CELL_NAME
SYS@o112 AS SYSDBA> desc X$CELL_NAME
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 CELLPATH_CELLROW                                                                                                           VARCHAR2(400)
 CELLHASHVAL_CELLROW                                                                                                        NUMBER
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and t.kqftanam = 'X$CELL_NAME'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$CELL_NAME                    ADDR
X$CELL_NAME                    INDX
X$CELL_NAME                    INST_ID
X$CELL_NAME                    CELLPATH_CELLROW
X$CELL_NAME                    CELLHASHVAL_CELLROW

And with X$CELL_NAME the X$KQFTA.INDX=X$KQFCO.KQFCOTAB link is restored.

So, this means that starting with X$KSXPTESTTBL the X$KQFCO.KQFCOTAB number is off by one. In fact, I tested version 18 (18.4), and found that the value X$KQFTA.INDX is 1217, so much higher, but still with X$KSXPTESTTBL the inconsistency of X$KQFCO.KQFCOTAB being one too high to correctly join with X$KQFTA.INDX.

Conclusion.
This means that based on my current investigations, it seems that starting from Oracle database version 11.2.0.1 up to version 19 and current PSU, starting with the X$KQFTA.INDX value of the table X$KSXPTESTTBL in X$KQFTA, the X$KQFCO.KQFCOTAB value needs to be subtracted by one to get the correct fields from X$KQFCO.KQFCOTAB to join to X$KQFTA.INDX.

A lot of X$ tables are not impacted, the majority of the tables have an X$KQFTA.INDX value lower than X$KSXPTESTTBL, but there are a reasonable amount of tables that are impacted, a lot of them related to ADR (starting with X$DIAG).

The reason for showing an additional oddity with X$SKGXP_MISC, X$SKGXP_CONNECTION, X$SKGXP_PORT and X$KSXP_STATS is that the X$ table where the join problem starts is X$KSXPTESTTBL. All these tables have something to do with cross instance IPC communication (skgxp/ksxp), and a lot of them have the weird property that they don’t exist whilst are visible in X$KQFTA, this might have something to do with X$KQFCO.KQFCOTAB being off by one. Of course this is something I can’t check, because it requires source code access, but I thought it would be good to mention.

It’s interesting to see that the join of X$KQFTA.INDX to X$KQFCO.KQFCOTAB is present in Oracle’s data dictionary with the view V$INDEXED_FIXED_COLUMN:

SQL> select view_definition from v$fixed_view_definition where view_name = 'GV$INDEXED_FIXED_COLUMN';

VIEW_DEFINITION
--------------------------------------------------------------------------------
select c.inst_id,kqftanam, kqfcoidx, kqfconam, kqfcoipo, c.con_id  from x$kqfco
c, x$kqfta t where t.indx = c.kqfcotab and kqfcoidx != 0

Which by now should be clear is not correct for X$ tables with an X$KQFTA.INDX number equal or higher than X$KSXPTESTTBL.

The amount of changes between version 18.7 and 18.8 is truly minimal. There’s one spare parameter renamed to an underscore parameter, which seems to be a back port of functionality created version 19:

DATABASE_VERSION                                                                                     NAME
---------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------
18.8                                                                                                 _ldap_password_oneway_auth
19.2                                                                                                 _ldap_password_oneway_auth
19.3                                                                                                 _ldap_password_oneway_auth
19.4                                                                                                 _ldap_password_oneway_auth
19.5                                                                                                 _ldap_password_oneway_auth

And it seems the DBA view DBA_REGISTRY_BACKPORTS is corrupted with 18.8, there only is a synonym with that name, which throws an ORA 1775: looping chain of synonyms when used. This is the reason it shows as gone in 18.8 (unique to version 18.7).

parameters unique in version 18.7 versus 18.8

NAME
--------------------------------------------------
_one-hundred-and-forty-sixth_spare_parameter

parameters unique in version 18.8 versus 18.7

NAME
--------------------------------------------------
_ldap_password_oneway_auth

parameter values changed isdefault between 18.7 versus 18.8

parameter values unique to 18.7 versus 18.8

parameter values unique to 18.8 versus 18.7

waitevents unique in version 18.7 versus 18.8

waitevents unique in version 18.8 versus 18.7

waitevents changed parameter description between 18.7 versus 18.8

x$ tables unique to 18.7 versus 18.8

x$ tables unique to 18.8 versus 18.7

x$ tables columns unique to 18.7 versus 18.8

x$ tables columns unique to 18.8 versus 18.7

v$ tables unique to 18.7 versus 18.8

v$ tables unique to 18.8 versus 18.7

v$ tables columns unique to 18.7 versus 18.8

v$ tables columns unique to 18.8 versus 18.7

gv$ tables unique to 18.7 versus 18.8

gv$ tables unique to 18.8 versus 18.7

gv$ tables columns unique to 18.7 versus 18.8

gv$ tables columns unique to 18.8 versus 18.7

sysstat statistics unique to 18.7 versus 18.8

sysstat statistics unique to 18.8 versus 18.7

sys_time_model statistics unique to 18.7 versus 18.8

sys_time_model statistics unique to 18.8 versus 18.7

dba tables unique to 18.7 versus 18.8

NAME
----------------------------------------------------------------------------------------------------
DBA_REGISTRY_BACKPORTS

dba tables unique to 18.8 versus 18.7

dba tables columns unique to 18.8 versus 18.7

dba tables columns unique to 18.7 versus 18.8

cdb tables unique to 18.7 versus 18.8

NAME
----------------------------------------------------------------------------------------------------
CDB_REGISTRY_BACKPORTS

cdb tables unique to 18.8 versus 18.7

cdb tables column unique to 18.7 versus 18.8

cdb tables column unique to 18.8 versus 18.7

And here are the differences in symbols (c functions).
The amount of functions added and removed is not huge this time (remember last time littecms was added).
The same java/domain functions as seen with version 19 have been removed, and the same PLSQL codegen layer functions, plus a singe DBMS_SODA function. There are some other functions that do not align with 19 removed.

The same batch of DBMS_SODA related functions are added, plus a lot of functions different than version 19.

code symbol names unique in version 18.7 versus 18.8

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
joncl_at_domains                                             (jon)cl_at_domains                                           java ??
joncl_domains                                                (jon)cl_domains                                              java ??
joncl_set_domains                                            (jon)cl_set_domains                                          java ??
joncl_snor_domains                                           (jon)cl_snor_domains                                         java ??
joniul_at_domains                                            (jon)iul_at_domains                                          java ??
joniul_domains                                               (jon)iul_domains                                             java ??
joniul_set_domains                                           (jon)iul_set_domains                                         java ??
joniul_snor_domains                                          (jon)iul_snor_domains                                        java ??
jonndl_at_domains                                            (jon)ndl_at_domains                                          java ??
jonndl_domains                                               (jon)ndl_domains                                             java ??
jonndl_set_domains                                           (jon)ndl_set_domains                                         java ??
jonndl_snor_domains                                          (jon)ndl_snor_domains                                        java ??
jonoce_at_domains                                            (jon)oce_at_domains                                          java ??
jonoce_domains                                               (jon)oce_domains                                             java ??
jonoce_set_domains                                           (jon)oce_set_domains                                         java ??
jonoce_snor_domains                                          (jon)oce_snor_domains                                        java ??
jonosd_at_domains                                            (jon)osd_at_domains                                          java ??
jonosd_domains                                               (jon)osd_domains                                             java ??
jonosd_set_domains                                           (jon)osd_set_domains                                         java ??
jonosd_snor_domains                                          (jon)osd_snor_domains                                        java ??
jonvcl_at_domains                                            (jon)vcl_at_domains                                          java ??
jonvcl_domains                                               (jon)vcl_domains                                             java ??
jonvcl_set_domains                                           (jon)vcl_set_domains                                         java ??
jonvcl_snor_domains                                          (jon)vcl_snor_domains                                        java ??
kestsInitTCB                                                 (kest)sInitTCB                                               kernel event SQL manageability tokenize (?) ??
kglsim_get_bucket_info                                       (kglsim)_get_bucket_info                                     kernel generic library cache management simulator (library cache advise)  ??
kgskltyp                                                     (kgsk)ltyp                                                   kernel generic service resource manager ??
krvxdsr                                                      (krvx)dsr                                                    kernel redo recovery extract ??
pdzdM60_Make_RBST                                            (pdz)dM60_Make_RBST                                          PLSQL code execution codegen infrastructure ??
pdzdM66_Insert_At_Root                                       (pdz)dM66_Insert_At_Root                                     PLSQL code execution codegen infrastructure ??
qcsoPreNameres_ProcOpt                                       (qcso)PreNameres_ProcOpt                                     query compile semantic analysis (parser) OBJECTS ??
qsodasqlGetInsertOneSQL                                      (qsoda)sqlGetInsertOneSQL                                    query system DBMS_SODA ??

code symbol names unique in version 18.8 versus 18.7

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
audIsSenCol                                                  (aud)IsSenCol                                                audit ??
audOptTab                                                    (aud)OptTab                                                  audit ??
audstwlk                                                     (aud)stwlk                                                   audit ??
kcfis_clear_lob                                              (kcfis)_clear_lob                                            kernel cache file management intelligent storage ??
kclcsb                                                       (kcl)csb                                                     kernel cache lock manager/buffer cache ??
kds_update_turbo_scan_pivot_statistics                       (kds)_update_turbo_scan_pivot_statistics                     kernel data seek/scan ??
kesdtcbResetDiagEvent                                        (kes)dtcbResetDiagEvent                                      kernel event SQL manageability ??
kfdDskComputeHashForRow                                      (kfd)DskComputeHashForRow                                    kernel automatic storage management disk ??
kfdp_chkSufOlnFGAvl                                          (kfdp)_chkSufOlnFGAvl                                        kernel automatic storage management disk PST ??
kfioSrMsgBuf_ack                                             (kfio)SrMsgBuf_ack                                           kernel automatic storage management translation I/O layer ??
kfkcrRefresh                                                 (kfk)crRefresh                                               kernel automatic storage management KFK ??
kfnFreeKfnpnmMem                                             (kfn)FreeKfnpnmMem                                           kernel automatic storage management networking subsystem ??
kgfvTermSkvol                                                (kgf)vTermSkvol                                              kernel generic ASM ??
kjznhcdn                                                     (kjzn)hcdn                                                   kernel lock management DIAG node layer ??
kjznhcds                                                     (kjzn)hcds                                                   kernel lock management DIAG node layer ??
kkopinlpInt                                                  (kko)pinlpInt                                                kernel compile optimizer ??
kkquIsCorrTransInList                                        (kkqu)IsCorrTransInList                                      kernel compile query  subquery unnesting ??
knlpmaxfmsgsize                                              (knl)pmaxfmsgsize                                            kernel replication streams replication ??
kpuxsoCurPrefetchFree                                        (kpu)xsoCurPrefetchFree                                      kernel programmatic interface user ??
kpuxsoDocContentSet                                          (kpu)xsoDocContentSet                                        kernel programmatic interface user ??
kpuxsoOperInit                                               (kpu)xsoOperInit                                             kernel programmatic interface user ??
krb_devtype_chk                                              (krb)_devtype_chk                                            kernel redo backup/restore ??
krvxgtf                                                      (krvx)gtf                                                    kernel redo recovery extract ??
krvxrsr_SkipRedo                                             (krvx)rsr_SkipRedo                                           kernel redo recovery extract ??
kzan_set_ddl_lock_timeout                                    (kza)n_set_ddl_lock_timeout                                  kernel security audit  ??
kzckmWalletInArchive                                         (kzck)mWalletInArchive                                       kernel security Code-Based Access Control encryption support ??
kzckmrOpenRootOnPlug                                         (kzck)mrOpenRootOnPlug                                       kernel security Code-Based Access Control encryption support ??
kzckmrXmlVldTDEKeys                                          (kzck)mrXmlVldTDEKeys                                        kernel security Code-Based Access Control encryption support ??
kzckmrXmlVldTDEKeysFile                                      (kzck)mrXmlVldTDEKeysFile                                    kernel security Code-Based Access Control encryption support ??
kzckmrXmlVldTDEMetaHSM                                       (kzck)mrXmlVldTDEMetaHSM                                     kernel security Code-Based Access Control encryption support ??
pfrcrt                                                       (pfr)crt                                                     PLSQL Code Execution runtime ??
qsodasqlGetContentDefine                                     (qsoda)sqlGetContentDefine                                   query system DBMS_SODA ??
qsodasqlGetCreatedOnDefine                                   (qsoda)sqlGetCreatedOnDefine                                 query system DBMS_SODA ??
qsodasqlGetInsertSQL                                         (qsoda)sqlGetInsertSQL                                       query system DBMS_SODA ??
qsodasqlGetKeyDefine                                         (qsoda)sqlGetKeyDefine                                       query system DBMS_SODA ??
qsodasqlGetLastModDefine                                     (qsoda)sqlGetLastModDefine                                   query system DBMS_SODA ??
qsodasqlGetMediaDefine                                       (qsoda)sqlGetMediaDefine                                     query system DBMS_SODA ??
qsodasqlGetVersionDefine                                     (qsoda)sqlGetVersionDefine                                   query system DBMS_SODA ??
qsodasqlPrefetchPrepare                                      (qsoda)sqlPrefetchPrepare                                    query system DBMS_SODA ??
qsodasqlSelectDocFieldsSQL                                   (qsoda)sqlSelectDocFieldsSQL                                 query system DBMS_SODA ??
qsodaxArrayFetchDocuments                                    (qsoda)xArrayFetchDocuments                                  query system DBMS_SODA ??
qsodaxDefineDocumentFields                                   (qsoda)xDefineDocumentFields                                 query system DBMS_SODA ??
qsodaxGetPrefetchDocument                                    (qsoda)xGetPrefetchDocument                                  query system DBMS_SODA ??
slmeset_s                                                    (sl)meset_s                                                  operating system (dependent) library ??

It seems that the most eye-catching difference between Oracle database versions 19.5 and 19.4 is three underscore parameters spare parameters being changed to specifically named underscore parameters, two of them have a bug number in them which is not publicly visible.

In v$sysstat/v$sesstat, a group of statistics are renamed from ‘OS %’ to ‘Server %’. All these statistics are about networking. One changed parameter directly points to networking (tcpinfo). These statistics are added in version 19.

One DBA view was added: DBA_DV_COMMON_OPERATION_STATUS. This view is specific and owned by to DVSYS.

NAME
--------------------------------------------------
_fourth_spare_parameter
_one-hundred-and-fifty-third_spare_parameter
_one-hundred-and-forty-sixth_spare_parameter

parameters unique in version 19.5 versus 19.4

NAME
--------------------------------------------------
_bug29302220_tcpinfo_statistics_save_atexit
_bug29903454_ksws_enable_alb
_session_modp_list

parameter values changed isdefault between 19.4 versus 19.5

parameter values unique to 19.4 versus 19.5

parameter values unique to 19.5 versus 19.4

waitevents unique in version 19.4 versus 19.5

waitevents unique in version 19.5 versus 19.4

waitevents changed parameter description between 19.4 versus 19.5

x$ tables unique to 19.4 versus 19.5

x$ tables unique to 19.5 versus 19.4

x$ tables columns unique to 19.4 versus 19.5

x$ tables columns unique to 19.5 versus 19.4

v$ tables unique to 19.4 versus 19.5

v$ tables unique to 19.5 versus 19.4

v$ tables columns unique to 19.4 versus 19.5

v$ tables columns unique to 19.5 versus 19.4

gv$ tables unique to 19.4 versus 19.5

gv$ tables unique to 19.5 versus 19.4

gv$ tables columns unique to 19.4 versus 19.5

gv$ tables columns unique to 19.5 versus 19.4

sysstat statistics unique to 19.4 versus 19.5

NAME
----------------------------------------------------------------------------------------------------
OS Advertised Receive Window
OS Advertised Send Window
OS Data Segments In
OS Data Segments Out
OS Lost Packets
OS Path Maximum Transmission Unit(MTU)
OS Send Congestion Window
OS Time (usec) Busy Sending Data
OS Time (usec) Busy Sending Data under Congestion
OS Time (usec) Last Ack Received
OS Time (usec) Last Ack Sent
OS Time (usec) Last Data Received
OS Time (usec) Last Data Sent
OS Time (usec) Limited by Receive Window
OS Time (usec) Limited by Send Buffer
OS Time (usec) Round Trip Time
OS Time (usec) Round Trip Time Variance
OS Total Bytes Acked
OS Total Bytes Received
OS Total Number of Retransmitted Packets

sysstat statistics unique to 19.5 versus 19.4

NAME
----------------------------------------------------------------------------------------------------
Server Advertised Receive Window
Server Advertised Send Window
Server Data Segments In
Server Data Segments Out
Server Elapsed Time (msec) Last Ack Received
Server Elapsed Time (msec) Last Ack Sent
Server Elapsed Time (msec) Last Data Received
Server Elapsed Time (msec) Last Data Sent
Server Lost Packets
Server Path Maximum Transmission Unit(MTU)
Server Send Congestion Window
Server Time (usec) Approximate Elapsed Time from the packet dro
Server Time (usec) Busy Sending Data
Server Time (usec) Limited by Receive Window
Server Time (usec) Limited by Send Buffer
Server Time (usec) Round Trip Time
Server Time (usec) Round Trip Time Variance
Server Total Bytes Acked
Server Total Bytes Received
Server Total Number of Retransmitted Packets

sys_time_model statistics unique to 19.4 versus 19.5

sys_time_model statistics unique to 19.5 versus 19.4

dba tables unique to 19.4 versus 19.5

dba tables unique to 19.5 versus 19.4

NAME
----------------------------------------------------------------------------------------------------
DBA_DV_COMMON_OPERATION_STATUS

dba tables columns unique to 19.5 versus 19.4

dba tables columns unique to 19.4 versus 19.5

cdb tables unique to 19.4 versus 19.5

cdb tables unique to 19.5 versus 19.4

cdb tables column unique to 19.4 versus 19.5

cdb tables column unique to 19.5 versus 19.4

And here are the differences in symbols (c functions).
The functions unique to 19.4 versus 19.5 (alias removed) are partly jon thus java related functions, which all have ‘domain’ in the name. Plus some diverse functions.

There is a significant amount number of functions added. The first ones starting with ‘aux_’ and ‘decode_krb5’ might have something to do with ‘krb5’, alias Kerberos version 5.

There is a ‘dbnest_dir_check’ function in between ‘aux’ and ‘decode’. It seems ‘dbnest’ is an entire layer in the database code at least, however I do not know if it relates to something that is public knowledge.

Among all kinds of singular added functions, what further is noticeable, is a high number of ASM related functions being added, of which a lot of them refer to ‘oss’ (oracle storage server?).

Additionally, a bunch of ‘ksp_modp’ functions are added which might have something to do with the ‘_session_modp_list’ parameter, and a bunch of parameters starting with ‘ksws_alb’ which might have something to do with the ‘_bug29903454_ksws_enable_alb’ parameter.

Further a group of functions starting with ‘soda’, which might be related to DBMS_SODA: a package implementing Simple Oracle Document Access (SODA), for managing document collections using PL/SQL.

code symbol names unique in version 19.4 versus 19.5

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
drexogidx                                                    (drex)ogidx                                                  text index query ??
joncl_at_domains                                             (jon)cl_at_domains                                           java ??
joncl_domains                                                (jon)cl_domains                                              java ??
joncl_set_domains                                            (jon)cl_set_domains                                          java ??
joncl_snor_domains                                           (jon)cl_snor_domains                                         java ??
joniul_at_domains                                            (jon)iul_at_domains                                          java ??
joniul_domains                                               (jon)iul_domains                                             java ??
joniul_set_domains                                           (jon)iul_set_domains                                         java ??
joniul_snor_domains                                          (jon)iul_snor_domains                                        java ??
jonndl_at_domains                                            (jon)ndl_at_domains                                          java ??
jonndl_domains                                               (jon)ndl_domains                                             java ??
jonndl_set_domains                                           (jon)ndl_set_domains                                         java ??
jonndl_snor_domains                                          (jon)ndl_snor_domains                                        java ??
jonoce_at_domains                                            (jon)oce_at_domains                                          java ??
jonoce_domains                                               (jon)oce_domains                                             java ??
jonoce_set_domains                                           (jon)oce_set_domains                                         java ??
jonoce_snor_domains                                          (jon)oce_snor_domains                                        java ??
jonosd_at_domains                                            (jon)osd_at_domains                                          java ??
jonosd_domains                                               (jon)osd_domains                                             java ??
jonosd_set_domains                                           (jon)osd_set_domains                                         java ??
jonosd_snor_domains                                          (jon)osd_snor_domains                                        java ??
jonvcl_at_domains                                            (jon)vcl_at_domains                                          java ??
jonvcl_domains                                               (jon)vcl_domains                                             java ??
jonvcl_set_domains                                           (jon)vcl_set_domains                                         java ??
jonvcl_snor_domains                                          (jon)vcl_snor_domains                                        java ??
jsv_try_scan_3choice                                         (js)v_try_scan_3choice                                       job scheduing ??
kcbtse_raise_tbs_datafile_enc_mismatch_error                 (kcbtse)_raise_tbs_datafile_enc_mismatch_error               kernel cache buffers tablespace encryption ??
kffgAttrMaxPxCb                                              (kffg)AttrMaxPxCb                                            kernel automatic storage management KFF file group ??
kffgVolClientCheck                                           (kffg)VolClientCheck                                         kernel automatic storage management KFF file group ??
kgiflini                                                     (kgi)flini                                                   kernel generic instantiation manager ??
kgiflpop                                                     (kgi)flpop                                                   kernel generic instantiation manager ??
kjgcr_GetTopCPU                                              (kj)gcr_GetTopCPU                                            kernel lock management ??
kjgcr_RunSyncTask                                            (kj)gcr_RunSyncTask                                          kernel lock management ??
kjgcr_SlaveReqGetSlot                                        (kj)gcr_SlaveReqGetSlot                                      kernel lock management ??
kkdlpGetObjn                                                 (kkdl)pGetObjn                                               kernel compile dictionary lookup ??
kkjcjchk                                                     (kkj)cjchk                                                   kernel compile job queue ??
kkoarFreeCtx                                                 (kkoar)FreeCtx                                               kernel compile optimizer automatic (sql) reoptimisation ??
kkoarFreeStats                                               (kkoar)FreeStats                                             kernel compile optimizer automatic (sql) reoptimisation ??
kkqojeFroNoRid                                               (kkqoj)eFroNoRid                                             kernel compile query  outer join ??
kkquIsCorrTransInList                                        (kkqu)IsCorrTransInList                                      kernel compile query  subquery unnesting ??
kslsesftcb_int                                               (ksl)sesftcb_int                                             kernel service (VOS) latching and post-wait ??
ktcsnaInitMinActScnInEnv                                     (ktc)snaInitMinActScnInEnv                                   kernel transaction control component ??
nsmplxhash                                                   (ns)mplxhash                                                 network session ??
pdy7F89_Create_Frame_Descriptor                              (pdy)7F89_Create_Frame_Descriptor                            PLSQL code execution codegen MCode gen ??
pdzdM60_Make_RBST                                            (pdz)dM60_Make_RBST                                          PLSQL code execution codegen infrastructure ??
pdzdM66_Insert_At_Root                                       (pdz)dM66_Insert_At_Root                                     PLSQL code execution codegen infrastructure ??
pdzdM67_Create                                               (pdz)dM67_Create                                             PLSQL code execution codegen infrastructure ??
qerocnStart                                                  (qeroc)nStart                                                query execute rowsource object collection iterator ??
qesmmCUpdProfile                                             (qesmm)CUpdProfile                                           query execute services memory manager (pga) ??
qosdMonNonCGCol                                              (qosd)MonNonCGCol                                            query optimizer statistics sql plan directive ??
qsodasqlGetInsertOneSQL                                      (qsoda)sqlGetInsertOneSQL                                    query system DBMS_SODA ??

code symbol names unique in version 19.5 versus 19.4

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
aux_loadptr_external_principal_identifier_ptr                aux_loadptr_external_principal_identifier_ptr                ??
aux_loadptr_pkinit_krb5_principal_name                       aux_loadptr_pkinit_krb5_principal_name                       ??
aux_loadptr_ptr_kdf_alg_id                                   aux_loadptr_ptr_kdf_alg_id                                   ??
aux_loadptr_ptr_seqof_external_principal_identifier          aux_loadptr_ptr_seqof_external_principal_identifier          ??
aux_loadptr_ptr_supported_kdfs                               aux_loadptr_ptr_supported_kdfs                               ??
aux_loadptr_subject_pk_info_ptr                              aux_loadptr_subject_pk_info_ptr                              ??
aux_present_opt_ptr_kdf_alg_id                               aux_present_opt_ptr_kdf_alg_id                               ??
aux_present_opt_ptr_seqof_external_principal_identifier      aux_present_opt_ptr_seqof_external_principal_identifier      ??
aux_present_opt_ptr_supported_kdfs                           aux_present_opt_ptr_supported_kdfs                           ??
aux_present_opt_subject_pk_info_ptr                          aux_present_opt_subject_pk_info_ptr                          ??
aux_storeptr_external_principal_identifier_ptr               aux_storeptr_external_principal_identifier_ptr               ??
aux_storeptr_pkinit_krb5_principal_name                      aux_storeptr_pkinit_krb5_principal_name                      ??
aux_storeptr_ptr_kdf_alg_id                                  aux_storeptr_ptr_kdf_alg_id                                  ??
aux_storeptr_ptr_seqof_external_principal_identifier         aux_storeptr_ptr_seqof_external_principal_identifier         ??
aux_storeptr_ptr_supported_kdfs                              aux_storeptr_ptr_supported_kdfs                              ??
aux_storeptr_subject_pk_info_ptr                             aux_storeptr_subject_pk_info_ptr                             ??
dbnest_dir_check                                             dbnest_dir_check                                             ??
decode_krb5_auth_pack                                        decode_krb5_auth_pack                                        ??
decode_krb5_auth_pack_draft9                                 decode_krb5_auth_pack_draft9                                 ??
decode_krb5_kdc_dh_key_info                                  decode_krb5_kdc_dh_key_info                                  ??
decode_krb5_pa_pk_as_rep                                     decode_krb5_pa_pk_as_rep                                     ??
decode_krb5_pa_pk_as_req                                     decode_krb5_pa_pk_as_req                                     ??
decode_krb5_pa_pk_as_req_draft9                              decode_krb5_pa_pk_as_req_draft9                              ??
decode_krb5_principal_name                                   decode_krb5_principal_name                                   ??
decode_krb5_reply_key_pack                                   decode_krb5_reply_key_pack                                   ??
decode_krb5_reply_key_pack_draft9                            decode_krb5_reply_key_pack_draft9                            ??
decode_krb5_td_dh_parameters                                 decode_krb5_td_dh_parameters                                 ??
decode_krb5_td_trusted_certifiers                            decode_krb5_td_trusted_certifiers                            ??
encode_krb5_auth_pack                                        encode_krb5_auth_pack                                        ??
encode_krb5_auth_pack_draft9                                 encode_krb5_auth_pack_draft9                                 ??
encode_krb5_kdc_dh_key_info                                  encode_krb5_kdc_dh_key_info                                  ??
encode_krb5_pa_pk_as_rep                                     encode_krb5_pa_pk_as_rep                                     ??
encode_krb5_pa_pk_as_rep_draft9                              encode_krb5_pa_pk_as_rep_draft9                              ??
encode_krb5_pa_pk_as_req                                     encode_krb5_pa_pk_as_req                                     ??
encode_krb5_pa_pk_as_req_draft9                              encode_krb5_pa_pk_as_req_draft9                              ??
encode_krb5_reply_key_pack                                   encode_krb5_reply_key_pack                                   ??
encode_krb5_reply_key_pack_draft9                            encode_krb5_reply_key_pack_draft9                            ??
encode_krb5_td_dh_parameters                                 encode_krb5_td_dh_parameters                                 ??
encode_krb5_td_trusted_certifiers                            encode_krb5_td_trusted_certifiers                            ??
kafcpy_one_row                                               (kaf)cpy_one_row                                             kernel access fetch ??
kcbflc_reap_internal                                         (kcb)flc_reap_internal                                       kernel cache buffers ??
kcbo_incr_doc                                                (kcbo)_incr_doc                                              kernel cache buffers object queue ??
kcbtse_raise_tbs_datafile_enc_mismatch                       (kcbtse)_raise_tbs_datafile_enc_mismatch                     kernel cache buffers tablespace encryption ??
kcfis_clear_lob                                              (kcfis)_clear_lob                                            kernel cache file management intelligent storage ??
kcslcu8                                                      (kcs)lcu8                                                    kernel cache scn management ??
kdilm_row_diskcompress_policy_type                           (kdil)m_row_diskcompress_policy_type                         kernel data index load ??
kdiss_optimize                                               (kdiss)_optimize                                             kernel data index split skip scan ??
kdlfForcedType                                               (kdl)fForcedType                                             kernel data lob ??
kdmrIMCULoadAvg                                              (kdmr)IMCULoadAvg                                            kernel data in-memory data layer background ??
kewraxc_attribute_xcb                                        (kewr)axc_attribute_xcb                                      kernel event AWR repository ??
kewxgenstbcb                                                 (kewx)genstbcb                                               kernel event AWR sysaux? ??
kewxgsc_generic_space_cbf                                    (kewx)gsc_generic_space_cbf                                  kernel event AWR sysaux? ??
kfIsZdlra                                                    (kf)IsZdlra                                                  kernel automatic storage management ??
kfdDskComputeHashForRow                                      (kfd)DskComputeHashForRow                                    kernel automatic storage management disk ??
kfdDskTableCbInternal                                        (kfd)DskTableCbInternal                                      kernel automatic storage management disk ??
kffgAddChildBlk                                              (kffg)AddChildBlk                                            kernel automatic storage management KFF file group ??
kffgAddDefault                                               (kffg)AddDefault                                             kernel automatic storage management KFF file group ??
kffgAttrMaxParCb                                             (kffg)AttrMaxParCb                                           kernel automatic storage management KFF file group ??
kffgAttrParSchmOvrdCb                                        (kffg)AttrParSchmOvrdCb                                      kernel automatic storage management KFF file group ??
kffgAuthenticate                                             (kffg)Authenticate                                           kernel automatic storage management KFF file group ??
kffgCalDeltaRedChg                                           (kffg)CalDeltaRedChg                                         kernel automatic storage management KFF file group ??
kffgCalRealQuota                                             (kffg)CalRealQuota                                           kernel automatic storage management KFF file group ??
kffgCalTargetRedun                                           (kffg)CalTargetRedun                                         kernel automatic storage management KFF file group ??
kffgCmp                                                      (kffg)Cmp                                                    kernel automatic storage management KFF file group ??
kffgFillFg                                                   (kffg)FillFg                                                 kernel automatic storage management KFF file group ??
kffgFindFree                                                 (kffg)FindFree                                               kernel automatic storage management KFF file group ??
kffgGetParOvrd                                               (kffg)GetParOvrd                                             kernel automatic storage management KFF file group ??
kffgIsParitySupported                                        (kffg)IsParitySupported                                      kernel automatic storage management KFF file group ??
kffgIsRedunAvailable                                         (kffg)IsRedunAvailable                                       kernel automatic storage management KFF file group ??
kffgIsUnsplit                                                (kffg)IsUnsplit                                              kernel automatic storage management KFF file group ??
kffgMakeName                                                 (kffg)MakeName                                               kernel automatic storage management KFF file group ??
kffgParseParOvrd                                             (kffg)ParseParOvrd                                           kernel automatic storage management KFF file group ??
kffgVerifyRedunChg                                           (kffg)VerifyRedunChg                                         kernel automatic storage management KFF file group ??
kffgcSyncDBLazy                                              (kffg)cSyncDBLazy                                            kernel automatic storage management KFF file group ??
kffgcSyncEnd                                                 (kffg)cSyncEnd                                               kernel automatic storage management KFF file group ??
kffgftMap                                                    (kffg)ftMap                                                  kernel automatic storage management KFF file group ??
kffgptAddDefault                                             (kffg)ptAddDefault                                           kernel automatic storage management KFF file group ??
kffgptFixRedun                                               (kffg)ptFixRedun                                             kernel automatic storage management KFF file group ??
kffgptGetStrVal                                              (kffg)ptGetStrVal                                            kernel automatic storage management KFF file group ??
kffgptLookup                                                 (kffg)ptLookup                                               kernel automatic storage management KFF file group ??
kffgptLookupDump                                             (kffg)ptLookupDump                                           kernel automatic storage management KFF file group ??
kffgptParseNum                                               (kffg)ptParseNum                                             kernel automatic storage management KFF file group ??
kfgComputeGrpProp                                            (kfg)ComputeGrpProp                                          kernel automatic storage management diskgroups ??
kfgGrpTableCbInternal                                        (kfg)GrpTableCbInternal                                      kernel automatic storage management diskgroups ??
kfiQuantize2                                                 (kf)iQuantize2                                               kernel automatic storage management ??
kfioComputeStripeIndex                                       (kfio)ComputeStripeIndex                                     kernel automatic storage management translation I/O layer ??
kfioGetXSetSize                                              (kfio)GetXSetSize                                            kernel automatic storage management translation I/O layer ??
kfioLoadParity                                               (kfio)LoadParity                                             kernel automatic storage management translation I/O layer ??
kfioParityWriteBegin                                         (kfio)ParityWriteBegin                                       kernel automatic storage management translation I/O layer ??
kfk_alloc_kfkio_freelist                                     (kfk)_alloc_kfkio_freelist                                   kernel automatic storage management KFK ??
kfk_asm_ioerror                                              (kfk)_asm_ioerror                                            kernel automatic storage management KFK ??
kfk_bld_sbmtptr_arry                                         (kfk)_bld_sbmtptr_arry                                       kernel automatic storage management KFK ??
kfk_bld_waitptr_arry                                         (kfk)_bld_waitptr_arry                                       kernel automatic storage management KFK ??
kfk_block_asmlib                                             (kfk)_block_asmlib                                           kernel automatic storage management KFK ??
kfk_block_oss                                                (kfk)_block_oss                                              kernel automatic storage management KFK ??
kfk_block_ufs                                                (kfk)_block_ufs                                              kernel automatic storage management KFK ??
kfk_build_osdstruct                                          (kfk)_build_osdstruct                                        kernel automatic storage management KFK ??
kfk_chk_xor_valid                                            (kfk)_chk_xor_valid                                          kernel automatic storage management KFK ??
kfk_do_abn_xor_chk                                           (kfk)_do_abn_xor_chk                                         kernel automatic storage management KFK ??
kfk_error_emulate                                            (kfk)_error_emulate                                          kernel automatic storage management KFK ??
kfk_fill_ksfdrq                                              (kfk)_fill_ksfdrq                                            kernel automatic storage management KFK ??
kfk_find_io_subsys                                           (kfk)_find_io_subsys                                         kernel automatic storage management KFK ??
kfk_get_asmioc_oper                                          (kfk)_get_asmioc_oper                                        kernel automatic storage management KFK ??
kfk_get_free_kfkrq                                           (kfk)_get_free_kfkrq                                         kernel automatic storage management KFK ??
kfk_get_unreaped_ios                                         (kfk)_get_unreaped_ios                                       kernel automatic storage management KFK ??
kfk_hard_abn_set                                             (kfk)_hard_abn_set                                           kernel automatic storage management KFK ??
kfk_hard_getoffs                                             (kfk)_hard_getoffs                                           kernel automatic storage management KFK ??
kfk_iodone_merge_slave                                       (kfk)_iodone_merge_slave                                     kernel automatic storage management KFK ??
kfk_lib_abn_check                                            (kfk)_lib_abn_check                                          kernel automatic storage management KFK ??
kfk_lib_xor_check                                            (kfk)_lib_xor_check                                          kernel automatic storage management KFK ??
kfk_oss_cell_timedout                                        (kfk)_oss_cell_timedout                                      kernel automatic storage management KFK ??
kfk_oss_dump_orq                                             (kfk)_oss_dump_orq                                           kernel automatic storage management KFK ??
kfk_oss_get_orq                                              (kfk)_oss_get_orq                                            kernel automatic storage management KFK ??
kfk_oss_offload_fill_orq                                     (kfk)_oss_offload_fill_orq                                   kernel automatic storage management KFK ??
kfk_oss_offload_submit                                       (kfk)_oss_offload_submit                                     kernel automatic storage management KFK ??
kfk_oss_post_wait                                            (kfk)_oss_post_wait                                          kernel automatic storage management KFK ??
kfk_oss_put_orq                                              (kfk)_oss_put_orq                                            kernel automatic storage management KFK ??
kfk_parse_disk_path                                          (kfk)_parse_disk_path                                        kernel automatic storage management KFK ??
kfk_proc_submitted_lib_reqs                                  (kfk)_proc_submitted_lib_reqs                                kernel automatic storage management KFK ??
kfk_process_an_ioq                                           (kfk)_process_an_ioq                                         kernel automatic storage management KFK ??
kfk_put_free_kfkios                                          (kfk)_put_free_kfkios                                        kernel automatic storage management KFK ??
kfk_reap_io_from_kfks                                        (kfk)_reap_io_from_kfks                                      kernel automatic storage management KFK ??
kfk_reap_lib_async_io                                        (kfk)_reap_lib_async_io                                      kernel automatic storage management KFK ??
kfk_reap_oss_async_io                                        (kfk)_reap_oss_async_io                                      kernel automatic storage management KFK ??
kfk_reap_ufs_async_io                                        (kfk)_reap_ufs_async_io                                      kernel automatic storage management KFK ??
kfk_rebuild_kfdrqs                                           (kfk)_rebuild_kfdrqs                                         kernel automatic storage management KFK ??
kfk_retry_submit_oss_io                                      (kfk)_retry_submit_oss_io                                    kernel automatic storage management KFK ??
kfk_submit_lib_io                                            (kfk)_submit_lib_io                                          kernel automatic storage management KFK ??
kfk_submit_oss_io                                            (kfk)_submit_oss_io                                          kernel automatic storage management KFK ??
kfk_submit_oss_io_no_wtevt                                   (kfk)_submit_oss_io_no_wtevt                                 kernel automatic storage management KFK ??
kfk_submit_oss_io_wtevt                                      (kfk)_submit_oss_io_wtevt                                    kernel automatic storage management KFK ??
kfk_submit_ufs_io                                            (kfk)_submit_ufs_io                                          kernel automatic storage management KFK ??
kfk_transit_fetchIO                                          (kfk)_transit_fetchIO                                        kernel automatic storage management KFK ??
kfk_transit_waitIO                                           (kfk)_transit_waitIO                                         kernel automatic storage management KFK ??
kfk_update_iodone_stats                                      (kfk)_update_iodone_stats                                    kernel automatic storage management KFK ??
kfk_zero_reap_array                                          (kfk)_zero_reap_array                                        kernel automatic storage management KFK ??
kfmdIsDGMountedCluster                                       (kfmd)IsDGMountedCluster                                     kernel automatic storage management node monitor interface implementation layer
                                                                                                                          for diskgroup registration ??

kfmsGetGrpMemCnt                                             (kfms)GetGrpMemCnt                                           kernel automatic storage management node monitor interface implementation layers
                                                                                                                           support function interface ??

kgfdDiscoverSubmit                                           (kgfd)DiscoverSubmit                                         kernel generic ASM io subsystem driver ??
kgh_size_sanity_check                                        (kgh)_size_sanity_check                                      kernel generic heap manager ??
kjac_allow_tg                                                (kjac)_allow_tg                                              kernel lock management application continuity  ??
kjccspbat                                                    (kjc)cspbat                                                  kernel lock management communication ??
kkocfbFindOldActual                                          (kkocfb)FindOldActual                                        kernel compile optimizer cardinality feedback ??
kkopinlpInt                                                  (kko)pinlpInt                                                kernel compile optimizer ??
kkscsGetPQSharingParameters                                  (kkscs)GetPQSharingParameters                                kernel compile shared objects (cursor) child cursor ??
kkscsGetSessionLimitDOP                                      (kkscs)GetSessionLimitDOP                                    kernel compile shared objects (cursor) child cursor ??
kokegAddOneGarbageItem                                       (koke)gAddOneGarbageItem                                     kernel objects kernel side OBJECTS images ??
kokegRepinLob                                                (koke)gRepinLob                                              kernel objects kernel side OBJECTS images ??
kpochalts                                                    (kpo)chalts                                                  kernel programmatic interface oracle ??
kpossGetEncNonTemplateOverflow                               (kpo)ssGetEncNonTemplateOverflow                             kernel programmatic interface oracle ??
kpossSetEncNonTemplateOverflow                               (kpo)ssSetEncNonTemplateOverflow                             kernel programmatic interface oracle ??
kpossTemplateMatch_                                          (kpo)ssTemplateMatch_                                        kernel programmatic interface oracle ??
kpossTemplateSet                                             (kpo)ssTemplateSet                                           kernel programmatic interface oracle ??
kpueONSTraceWrite                                            (kpu)eONSTraceWrite                                          kernel programmatic interface user ??
kpueONSTrcCb                                                 (kpu)eONSTrcCb                                               kernel programmatic interface user ??
kpussTemplateSend                                            (kpu)ssTemplateSend                                          kernel programmatic interface user ??
kpuxcOpenCurCntDecr                                          (kpu)xcOpenCurCntDecr                                        kernel programmatic interface user ??
kpuxcOpenCurCntIncr                                          (kpu)xcOpenCurCntIncr                                        kernel programmatic interface user ??
kpuxcRefCurDfnMarkOpen                                       (kpu)xcRefCurDfnMarkOpen                                     kernel programmatic interface user ??
kpuxcRefCurOutBndMarkOpen                                    (kpu)xcRefCurOutBndMarkOpen                                  kernel programmatic interface user ??
kpuxcSessionSignatureRecv                                    (kpu)xcSessionSignatureRecv                                  kernel programmatic interface user ??
kpuxcSessionTemplateSend                                     (kpu)xcSessionTemplateSend                                   kernel programmatic interface user ??
kpuxcSessionTemplatesFree                                    (kpu)xcSessionTemplatesFree                                  kernel programmatic interface user ??
kpuxcTACService                                              (kpu)xcTACService                                            kernel programmatic interface user ??
kpuxsoCurPrefetchFree                                        (kpu)xsoCurPrefetchFree                                      kernel programmatic interface user ??
kpuxsoOperInit                                               (kpu)xsoOperInit                                             kernel programmatic interface user ??
krbbCountThreads                                             (krbb)CountThreads                                           kernel redo backup/restore creation of a backuppiece ??
ksdhng_wdat_rem_dup                                          (ksdhng)_wdat_rem_dup                                        kernel service (VOS) debug internal errors hang analyze ??
ksfdafGetXsetSize                                            (ksfdaf)GetXsetSize                                          kernel service (VOS) functions disk IO asm files ??
ksfdafParityWrite                                            (ksfdaf)ParityWrite                                          kernel service (VOS) functions disk IO asm files ??
ksfdafParityWriteAbort                                       (ksfdaf)ParityWriteAbort                                     kernel service (VOS) functions disk IO asm files ??
ksfdafParityWriteBegin                                       (ksfdaf)ParityWriteBegin                                     kernel service (VOS) functions disk IO asm files ??
ksfdafParityWriteEnd                                         (ksfdaf)ParityWriteEnd                                       kernel service (VOS) functions disk IO asm files ??
kslwt_event_stats                                            (kslwt)_event_stats                                          kernel service (VOS) latching and post-wait wait interface ??
kslwt_push_stats                                             (kslwt)_push_stats                                           kernel service (VOS) latching and post-wait wait interface ??
ksmsq_spendingi                                              (ksmsq)_spendingi                                            kernel service (VOS) memory sga heap message queue services ??
ksp_init_modp_send                                           (ksp)_init_modp_send                                         kernel service (VOS) parameter ??
ksp_modp_get_enckeyvals                                      (ksp)_modp_get_enckeyvals                                    kernel service (VOS) parameter ??
ksp_modp_get_keyvals                                         (ksp)_modp_get_keyvals                                       kernel service (VOS) parameter ??
ksp_modp_set_enckeyvals                                      (ksp)_modp_set_enckeyvals                                    kernel service (VOS) parameter ??
ksp_modp_set_keyvals                                         (ksp)_modp_set_keyvals                                       kernel service (VOS) parameter ??
ksp_modp_update_sign                                         (ksp)_modp_update_sign                                       kernel service (VOS) parameter ??
kspdecbuf                                                    (ksp)decbuf                                                  kernel service (VOS) parameter ??
kspencbuf                                                    (ksp)encbuf                                                  kernel service (VOS) parameter ??
ksucln_expired_cleanup_timer                                 (ksucln)_expired_cleanup_timer                               kernel service (VOS) user cleanup detached process ??
kswrListDelete                                               (ksw)rListDelete                                             kernel service (VOS) workgroup ??
ksws_alb_alloc_pga_memory                                    (ksws)_alb_alloc_pga_memory                                  kernel service (VOS) workgroup services ??
ksws_alb_block_service                                       (ksws)_alb_block_service                                     kernel service (VOS) workgroup services ??
ksws_alb_check_sessions_to_mark                              (ksws)_alb_check_sessions_to_mark                            kernel service (VOS) workgroup services ??
ksws_alb_check_threshold                                     (ksws)_alb_check_threshold                                   kernel service (VOS) workgroup services ??
ksws_alb_clear_mark_drain_session                            (ksws)_alb_clear_mark_drain_session                          kernel service (VOS) workgroup services ??
ksws_alb_dump_service                                        (ksws)_alb_dump_service                                      kernel service (VOS) workgroup services ??
ksws_alb_dump_session                                        (ksws)_alb_dump_session                                      kernel service (VOS) workgroup services ??
ksws_alb_filter_session                                      (ksws)_alb_filter_session                                    kernel service (VOS) workgroup services ??
ksws_alb_mark_target_sessions                                (ksws)_alb_mark_target_sessions                              kernel service (VOS) workgroup services ??
ksws_alb_process_directives                                  (ksws)_alb_process_directives                                kernel service (VOS) workgroup services ??
ksws_alb_run_gv_sql                                          (ksws)_alb_run_gv_sql                                        kernel service (VOS) workgroup services ??
ksws_alb_safe_session                                        (ksws)_alb_safe_session                                      kernel service (VOS) workgroup services ??
ksws_alb_safe_to_block_service                               (ksws)_alb_safe_to_block_service                             kernel service (VOS) workgroup services ??
ksws_alb_selection_bias_apply                                (ksws)_alb_selection_bias_apply                              kernel service (VOS) workgroup services ??
ksws_alb_set_mark_drain_session                              (ksws)_alb_set_mark_drain_session                            kernel service (VOS) workgroup services ??
ksws_alb_unblock_fix_delta_services                          (ksws)_alb_unblock_fix_delta_services                        kernel service (VOS) workgroup services ??
ksws_alb_unblock_service                                     (ksws)_alb_unblock_service                                   kernel service (VOS) workgroup services ??
ksws_alb_update_service_session_metrics                      (ksws)_alb_update_service_session_metrics                    kernel service (VOS) workgroup services ??
ksws_alb_update_threshold                                    (ksws)_alb_update_threshold                                  kernel service (VOS) workgroup services ??
ksws_alb_valid_session                                       (ksws)_alb_valid_session                                     kernel service (VOS) workgroup services ??
kwraChkOptdef                                                (k)wraChkOptdef                                              kernel ??
kwrachkopnTree                                               (k)wrachkopnTree                                             kernel ??
kwraplschkLogdef                                             (k)wraplschkLogdef                                           kernel ??
kxccXRkxck                                                   (kxcc)XRkxck                                                 kernel execution constraints catch all ??
kzctxgIsGlobal                                               (kzctxg)IsGlobal                                             kernel security Code-Based Access Control context routines global_context ??
kzrtGetAndCheckTriggerCache                                  (kzrt)GetAndCheckTriggerCache                                kernel security virtual private database privileges ??
kzrtTriggerPresent                                           (kzrt)TriggerPresent                                         kernel security virtual private database privileges ??
kzvdveCommOpBypass                                           (kzvd)veCommOpBypass                                         kernel security data vault ??
kzvdveIsAppCommonUserId                                      (kzvd)veIsAppCommonUserId                                    kernel security data vault ??
kzvradmAllowCommOp                                           (kz)vradmAllowCommOp                                         kernel security ??
kzvradmAllowCommOpCB                                         (kz)vradmAllowCommOpCB                                       kernel security ??
kzvradmAllowCommOp_int                                       (kz)vradmAllowCommOp_int                                     kernel security ??
kzvutilGetACOStatus                                          (kz)vutilGetACOStatus                                        kernel security ??
kzvutilGetACOStatus0                                         (kz)vutilGetACOStatus0                                       kernel security ??
psdvcn                                                       (psd)vcn                                                     PLSQL infrastructure programmatic interfaces to/from  ??
qesrcTol_New                                                 (qesrc)Tol_New                                               query execute services result cache ??
qsodasqlGetContentDefine                                     (qsoda)sqlGetContentDefine                                   query system DBMS_SODA ??
qsodasqlGetCreatedOnDefine                                   (qsoda)sqlGetCreatedOnDefine                                 query system DBMS_SODA ??
qsodasqlGetInsertSQL                                         (qsoda)sqlGetInsertSQL                                       query system DBMS_SODA ??
qsodasqlGetKeyDefine                                         (qsoda)sqlGetKeyDefine                                       query system DBMS_SODA ??
qsodasqlGetLastModDefine                                     (qsoda)sqlGetLastModDefine                                   query system DBMS_SODA ??
qsodasqlGetMediaDefine                                       (qsoda)sqlGetMediaDefine                                     query system DBMS_SODA ??
qsodasqlGetVersionDefine                                     (qsoda)sqlGetVersionDefine                                   query system DBMS_SODA ??
qsodasqlPrefetchPrepare                                      (qsoda)sqlPrefetchPrepare                                    query system DBMS_SODA ??
qsodasqlSelectDocFieldsSQL                                   (qsoda)sqlSelectDocFieldsSQL                                 query system DBMS_SODA ??
qsodaxArrayFetchDocuments                                    (qsoda)xArrayFetchDocuments                                  query system DBMS_SODA ??
qsodaxDefineDocumentFields                                   (qsoda)xDefineDocumentFields                                 query system DBMS_SODA ??
qsodaxGetPrefetchDocument                                    (qsoda)xGetPrefetchDocument                                  query system DBMS_SODA ??
sskgds_dw_is_context_valid                                   (sskg)ds_dw_is_context_valid                                 operating system dependent system kernel generic ??
sskgds_dw_is_pccontext_valid                                 (sskg)ds_dw_is_pccontext_valid                               operating system dependent system kernel generic ??

In the article oracle memory troubleshooting using analysis on heap dumps I introduced heap_analyze.awk.

The reason the tool exists is because I am using it myself. Therefore, I ran into additional things that I wanted the tool to do. I added some stuff, which is that significant, that I decided to make another blogpost to introduce the new features.

1. Percentages
In order to get an idea of the relative size of the summarised topic, I added a percentage. For example:

top 5 allocation by total size per alloc reason per heap
==================================================================================================
heap             alloc_reason            #chunks       total_size   %
--------------------------------------------------------------------------------------------------
sga heap(1,0)    perm                         19        182947456  72
sga heap(1,0)                                 43         14488960   5
sga heap(1,0)    SO private sga               18         14284168   5
sga heap(1,0)    KGLHD                      5787          4318400   1
sga heap(1,0)    KSFD SGA I/O b                1          4190416   1

2. Enhanced perm (permanent) memory descriptions
It seems that for PGA heap dumps, sometimes there is a description for memory area’s that are perm (permanent memory, memory allocated for the lifetime of the process). This is how that’s visible in the dump:

PERMANENT CHUNKS:
  Chunk     7fcbcad6c020 sz=    20632    perm      "perm           "  alo=20632
            7fcbcad6c020 sz=    20632    cpmlst    "callback hsn   "

I must say I don’t know what ‘cpmlst’ means, so if anyone knows or has a good guess, please let me know. However, the two addresses and sizes are an exact match, so I now change the alloc_reason for the cpmlst text.
This is helpful because there is a quite some memory allocated as perm. Sadly, this is not done for SGA dumps.

3. Excel mode
If you want to store and compare different dumps, one way of doing that is pasting the output in Microsoft excel. Once you set the ‘Text To Columns’ option to space as a separator, it will put the information in its own cells. But there are a few problems with that, one of them is that the heap names and alloc_reasons can have spaces in them, so that the placement of the figures can vary. I created excel mode for that.
If you set excel mode (set the variable excel_mode to 1 on line 5):

#!/bin/awk -f
BEGIN {
  printf "Heap dump analyzer v0.2 by Frits Hoogland\n";
  group_sum_nr=5;
  excel_mode=1;
}

In this mode, the horizontal lines (with ‘-‘ and ‘=’) are omitted in the output, and all spaces are changed to underscores, so a table stays consistent when pasted in excel:

heap             alloc_reason            #chunks       total_size   %
sga_heap(1,0)    perm___________              19        182947456  72
sga_heap(1,0)    _______________              43         14488960   5
sga_heap(1,0)    SO_private_sga_              18         14284168   5
sga_heap(1,0)    KGLHD__________            5787          4318400   1
sga_heap(1,0)    KSFD_SGA_I/O_b_               1          4190416   1

4. New table which shows memory by type
Another way of looking at memory in a heap is by grouping it by type. This allows you to very quickly see if a certain type of chunk is dominating a heap:

heap                        type    #chunks         min_size         max_size       total_size   %
--------------------------------------------------------------------------------------------------
top call heap               free          7            16408            65512           390064  99
top call heap           recreate          2              992              992             1984   0
top call heap               perm          2              120              904             1024   0

This is an overview of the top call heap of a session that is not active, and therefore most of it should be empty, which is true for this dump.

Once again, get the awk script here: https://gitlab.com/FritsHoogland/oracle_memory_analyze/blob/master/heap_analyze.awk

%d bloggers like this: