Archive

Tag Archives: internals

This blogpost is about an inconsistency I found in the X$ tables X$KQFTA and X$KQFCO. This is very specific. If you don’t care about that, you can skip this post.

The Oracle database’s “dynamic performance views” are views that are prefixed with “GV$” and “V$”. The “G” with “GV$” stands for “global” and gets you the results from its “V$” equivalent for all instances. In most cases, but that’s not the purpose of this blog. In most cases, the “V$” version simply is the “GV$” view with inst_id (instance id, used in RAC to specify the instance id) set to the current instance. I’ll refer to both simply as “V$” for simplicity.

The “V$” views are mostly build on top of “X$” tables. The “X$” tables are Oracle internal tables, and not officially supported, therefore you should use the “V$” views.

One of the things that certain “X$” tables expose is information about the “X$” tables and their columns theirselves. These “X$” tables are visible in “X$KQFTA” (kernel query fixed tables tables) for the “X$” tables, and “X$KQFCO” (kernel query fixed tables columns) for the columns of the “X$” tables. I learned about these tables a long time ago from Kyle Hailey, Anjo Kolk and James Morle regarding direct memory attachment for wait events, the reason being the ability to find the memory address for certain “X$” tables to fetch the data these hold directly. We are talking about the 1990s here. In fact, Kyle later put the history of how he learned about it in the “Oracle insights: tales of the oaktable” book.

I gather information about every PSU of every Oracle database version on linux once it comes out. This is how I am able to produce the posts about the difference between a current and a previous PSU. One of the things that I gather is information about X$ tables and their columns. In order to obtain the columns, I use X$KQFTA and X$KQFCO and join these in the way that is publicly known:

select t.kqftanam, c.kqfconam
from x$kqfta t, x$kqfco c
where t.inst_id = 1
and c.inst_id = 1
and t.indx = c.kqfcotab;

And this worked well…

…until I produced a difference report of versions 12.1.0.2.191015 and 12.1.0.2.190716, where the report reported the following difference:

x$ tables columns unique to 12.1.0.2.190716 versus 12.1.0.2.191015

NAME                                                                                                 COLUMN_NAME
---------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------------
X$KQFTVRTTST0                                                                                        ADDR
X$KQFTVRTTST0                                                                                        CHGROWID
X$KQFTVRTTST0                                                                                        CHGROWOP
X$KQFTVRTTST0                                                                                        COL0_0
X$KQFTVRTTST0                                                                                        COL0_1
X$KQFTVRTTST0                                                                                        COL0_2
X$KQFTVRTTST0                                                                                        COL0_3
X$KQFTVRTTST0                                                                                        COL0_4
...and so on...

This could happen, these are columns that seem to have vanished with 12.1.0.2.191015. However, I decided to check it, although I haven’t got access to the 12.1.0.2.191015 PSU, because it’s a PSU version only available for paid extended support.

I looked at the columns of X$KQFTVRTST0 in my current database, which happens to be an ancient 11.2.0.2 database. However, much to my surprise, this gave me radically different columns:

SYS@o112 AS SYSDBA> desc X$KQFTVRTTST0
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 COLA                                                                                                                       VARCHAR2(64)
 COLB                                                                                                                       NUMBER

Okay, something weird is going on here.

First let’s look in which versions X$KQFTVRTTST0 is present:

ORAVER@o112 > @x_table_in_versions.sql

DATABASE_VERSION     NAME
-------------------- --------------------------------------------------------------------------------------------------------------------------------
11.2.0.1.0           X$KQFTVRTTST0
11.2.0.2.0           X$KQFTVRTTST0
11.2.0.2.12          X$KQFTVRTTST0
11.2.0.3.0           X$KQFTVRTTST0
11.2.0.3.15          X$KQFTVRTTST0
11.2.0.4.0           X$KQFTVRTTST0
11.2.0.4.1           X$KQFTVRTTST0
11.2.0.4.160119      X$KQFTVRTTST0
11.2.0.4.160419      X$KQFTVRTTST0
11.2.0.4.160719      X$KQFTVRTTST0
11.2.0.4.161018      X$KQFTVRTTST0
11.2.0.4.170418      X$KQFTVRTTST0
11.2.0.4.170718      X$KQFTVRTTST0
11.2.0.4.170814      X$KQFTVRTTST0
11.2.0.4.180116      X$KQFTVRTTST0
11.2.0.4.180417      X$KQFTVRTTST0
11.2.0.4.180717      X$KQFTVRTTST0
11.2.0.4.190115      X$KQFTVRTTST0
11.2.0.4.191015      X$KQFTVRTTST0
11.2.0.4.2           X$KQFTVRTTST0
11.2.0.4.3           X$KQFTVRTTST0
11.2.0.4.4           X$KQFTVRTTST0
11.2.0.4.5           X$KQFTVRTTST0
11.2.0.4.6           X$KQFTVRTTST0
11.2.0.4.7           X$KQFTVRTTST0
11.2.0.4.8           X$KQFTVRTTST0
12.1.0.2.0           X$KQFTVRTTST0
12.1.0.2.1           X$KQFTVRTTST0
12.1.0.2.160119      X$KQFTVRTTST0
12.1.0.2.160419      X$KQFTVRTTST0
12.1.0.2.160719      X$KQFTVRTTST0
12.1.0.2.161018      X$KQFTVRTTST0
12.1.0.2.170117      X$KQFTVRTTST0
12.1.0.2.170418      X$KQFTVRTTST0
12.1.0.2.170718      X$KQFTVRTTST0
12.1.0.2.170814      X$KQFTVRTTST0
12.1.0.2.171017      X$KQFTVRTTST0
12.1.0.2.180116      X$KQFTVRTTST0
12.1.0.2.180417      X$KQFTVRTTST0
12.1.0.2.181016      X$KQFTVRTTST0
12.1.0.2.190115      X$KQFTVRTTST0
12.1.0.2.190416      X$KQFTVRTTST0
12.1.0.2.190716      X$KQFTVRTTST0
12.1.0.2.191015      X$KQFTVRTTST0
12.1.0.2.3           X$KQFTVRTTST0
12.1.0.2.4           X$KQFTVRTTST0
12.1.0.2.5           X$KQFTVRTTST0
12.2.0.1.0           X$KQFTVRTTST0
12.2.0.1.170620      X$KQFTVRTTST0
12.2.0.1.170718      X$KQFTVRTTST0
12.2.0.1.170814      X$KQFTVRTTST0
12.2.0.1.171017      X$KQFTVRTTST0
12.2.0.1.180116      X$KQFTVRTTST0
12.2.0.1.180417      X$KQFTVRTTST0
12.2.0.1.180717      X$KQFTVRTTST0
12.2.0.1.181016      X$KQFTVRTTST0
12.2.0.1.190115      X$KQFTVRTTST0
12.2.0.1.190416      X$KQFTVRTTST0
12.2.0.1.190716      X$KQFTVRTTST0
12.2.0.1.191015      X$KQFTVRTTST0
18.3                 X$KQFTVRTTST0
18.4                 X$KQFTVRTTST0
18.5                 X$KQFTVRTTST0
18.6                 X$KQFTVRTTST0
18.7                 X$KQFTVRTTST0
18.8                 X$KQFTVRTTST0
19.2                 X$KQFTVRTTST0
19.3                 X$KQFTVRTTST0
19.4                 X$KQFTVRTTST0
19.5                 X$KQFTVRTTST0

So, X$KQFVRTTST0 appeared with 11.2.0.1. Now let’s take a look at how the columns look like in X$KQFVRTTST0 looks in version 11.2.0.1:

ORAVER@o112 > @x_table_cols_in_versions.sql

DATABASE_VERSION	       TABLE_NAME		      COLUMN_NAME
------------------------------ ------------------------------ ----------------------------------------
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL3_0
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL2_0
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL1_0
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_9
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_8
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_7
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_6
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_5
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_4
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_3
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_2
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_1
11.2.0.1.0		       X$KQFTVRTTST0		      COL0_0
11.2.0.1.0		       X$KQFTVRTTST0		      CHGROWOP
11.2.0.1.0		       X$KQFTVRTTST0		      CHGROWID
11.2.0.1.0		       X$KQFTVRTTST0		      COL_OPERATION
11.2.0.1.0		       X$KQFTVRTTST0		      INST_ID
11.2.0.1.0		       X$KQFTVRTTST0		      INDX
11.2.0.1.0		       X$KQFTVRTTST0		      ADDR

These are the columns which were shown in the difference report that said they were removed. Now let’s look at the columns in version 19.5:

ORAVER@o112 > @x_table_cols_in_versions.sql

DATABASE_VERSION               TABLE_NAME                     COLUMN_NAME
------------------------------ ------------------------------ ----------------------------------------
19.5                           X$KQFTVRTTST0                  COL3_9
19.5                           X$KQFTVRTTST0                  COL3_8
19.5                           X$KQFTVRTTST0                  COL3_7
19.5                           X$KQFTVRTTST0                  COL3_6
19.5                           X$KQFTVRTTST0                  COL3_5
19.5                           X$KQFTVRTTST0                  COL3_4
19.5                           X$KQFTVRTTST0                  COL3_3
19.5                           X$KQFTVRTTST0                  COL3_2
19.5                           X$KQFTVRTTST0                  COL3_1
19.5                           X$KQFTVRTTST0                  COL3_0
19.5                           X$KQFTVRTTST0                  COL2_9
19.5                           X$KQFTVRTTST0                  COL2_8
19.5                           X$KQFTVRTTST0                  COL2_7
19.5                           X$KQFTVRTTST0                  COL2_6
19.5                           X$KQFTVRTTST0                  COL2_5
19.5                           X$KQFTVRTTST0                  COL2_4
19.5                           X$KQFTVRTTST0                  COL2_3
19.5                           X$KQFTVRTTST0                  COL2_2
19.5                           X$KQFTVRTTST0                  COL2_1
19.5                           X$KQFTVRTTST0                  COL2_0
19.5                           X$KQFTVRTTST0                  COL1_9
19.5                           X$KQFTVRTTST0                  COL1_8
19.5                           X$KQFTVRTTST0                  COL1_7
19.5                           X$KQFTVRTTST0                  COL1_6
19.5                           X$KQFTVRTTST0                  COL1_5
19.5                           X$KQFTVRTTST0                  COL1_4
19.5                           X$KQFTVRTTST0                  COL1_3
19.5                           X$KQFTVRTTST0                  COL1_2
19.5                           X$KQFTVRTTST0                  COL1_1
19.5                           X$KQFTVRTTST0                  COL1_0
19.5                           X$KQFTVRTTST0                  COL0_9
19.5                           X$KQFTVRTTST0                  COL0_8
19.5                           X$KQFTVRTTST0                  COL0_7
19.5                           X$KQFTVRTTST0                  COL0_6
19.5                           X$KQFTVRTTST0                  COL0_5
19.5                           X$KQFTVRTTST0                  COL0_4
19.5                           X$KQFTVRTTST0                  COL0_3
19.5                           X$KQFTVRTTST0                  COL0_2
19.5                           X$KQFTVRTTST0                  COL0_1
19.5                           X$KQFTVRTTST0                  COL0_0
19.5                           X$KQFTVRTTST0                  CHGROWOP
19.5                           X$KQFTVRTTST0                  CHGROWID
19.5                           X$KQFTVRTTST0                  COL_OPERATION
19.5                           X$KQFTVRTTST0                  CON_ID
19.5                           X$KQFTVRTTST0                  INST_ID
19.5                           X$KQFTVRTTST0                  INDX
19.5                           X$KQFTVRTTST0                  ADDR

These are exactly the same! So the columns for X$KQFTVRTTST0 that I obtained from the database are the same for all versions in my data, but when I look in an actual database using ‘describe’ I see ADDR, INDX, INST_ID and COLA and COLB. So my data seems to be wrong!

My first reaction was to change the columns for X$KQFTVRTST0 for every version to correct this. But then I realised that I had to investigate this deeper to understand the actual problem, there might be more X$ tables involved.

The first thing to do, is to see if I can reproduce the issue in my database. This can be quite easily be done by describing X$KQFTVRTTST0 to see the columns, and then query these using the above query that demonstrates the join between X$KQFTA and X$KQFCO by adding a filter on ‘X$KQFTVRTST0’:

SYS@o112 AS SYSDBA> desc X$KQFTVRTTST0
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 COLA                                                                                                                       VARCHAR2(64)
 COLB                                                                                                                       NUMBER
SYS@o112 AS SYSDBA> l
  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and kqftanam = 'X$KQFTVRTTST0'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$KQFTVRTTST0                  ADDR
X$KQFTVRTTST0                  INDX
X$KQFTVRTTST0                  INST_ID
X$KQFTVRTTST0                  COL_OPERATION
X$KQFTVRTTST0                  CHGROWID
X$KQFTVRTTST0                  CHGROWOP
X$KQFTVRTTST0                  COL0_0
X$KQFTVRTTST0                  COL0_1
X$KQFTVRTTST0                  COL0_2
X$KQFTVRTTST0                  COL0_3
X$KQFTVRTTST0                  COL0_4
X$KQFTVRTTST0                  COL0_5
X$KQFTVRTTST0                  COL0_6
X$KQFTVRTTST0                  COL0_7
X$KQFTVRTTST0                  COL0_8
X$KQFTVRTTST0                  COL0_9
X$KQFTVRTTST0                  COL1_0
X$KQFTVRTTST0                  COL1_1
X$KQFTVRTTST0                  COL1_2
X$KQFTVRTTST0                  COL1_3
X$KQFTVRTTST0                  COL1_4
X$KQFTVRTTST0                  COL1_5
X$KQFTVRTTST0                  COL1_6
X$KQFTVRTTST0                  COL1_7
X$KQFTVRTTST0                  COL1_8
X$KQFTVRTTST0                  COL1_9
X$KQFTVRTTST0                  COL2_0
X$KQFTVRTTST0                  COL2_1
X$KQFTVRTTST0                  COL2_2
X$KQFTVRTTST0                  COL2_3
X$KQFTVRTTST0                  COL2_4
X$KQFTVRTTST0                  COL2_5
X$KQFTVRTTST0                  COL2_6
X$KQFTVRTTST0                  COL2_7
X$KQFTVRTTST0                  COL2_8
X$KQFTVRTTST0                  COL2_9
X$KQFTVRTTST0                  COL3_0
X$KQFTVRTTST0                  COL3_1
X$KQFTVRTTST0                  COL3_2
X$KQFTVRTTST0                  COL3_3
X$KQFTVRTTST0                  COL3_4
X$KQFTVRTTST0                  COL3_5
X$KQFTVRTTST0                  COL3_6
X$KQFTVRTTST0                  COL3_7
X$KQFTVRTTST0                  COL3_8
X$KQFTVRTTST0                  COL3_9

Bingo! Describe tells me about COLA and COLB, which is the actual situation, and querying X$KQFTA and X$KQFCO gives me different, wrong columns.

First let’s investigate if we can find the correct columns in X$KQFCO, and see what the difference is between X$KQFTA.INDX and X$KQFCO.KQFCOTAB:

SYS@o112 AS SYSDBA> select indx from x$kqfta where kqftanam = 'X$KQFTVRTTST0';

      INDX
----------
       841

SYS@o112 AS SYSDBA> select kqfcotab from x$kqfco where kqfconam = 'COLA';

  KQFCOTAB
----------
       842

Aha! So the join is off by one! So, that must mean that if I decrease KQFCOTAB by one, I do get the correct output:

SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab-1
  6* and kqftanam = 'X$KQFTVRTTST0'
  7  /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$KQFTVRTTST0                  ADDR
X$KQFTVRTTST0                  INDX
X$KQFTVRTTST0                  INST_ID
X$KQFTVRTTST0                  COLA
X$KQFTVRTTST0                  COLB

Bingo! The next thing to do is see if this is only a problem for X$KQFTVRTTST0, or a bigger problem. Let’s take the next table based on X$KQFTA.INDX, and see if this shows the same problem:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 842;

KQFTANAM
------------------------------
X$DIAG_ADR_CONTROL
SYS@o112 AS SYSDBA> desc X$DIAG_ADR_CONTROL
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 ADR_PATH_IDX                                                                                                               VARCHAR2(445)
 ADR_HOME                                                                                                                   VARCHAR2(445)
 ADRID                                                                                                                      NUMBER
 SHORTP_POLICY                                                                                                              NUMBER
 LONGP_POLICY                                                                                                               NUMBER
 LAST_MOD_TIME                                                                                                              TIMESTAMP(9) WITH TIME ZONE
 LAST_AUTOPRG_TIME                                                                                                          TIMESTAMP(9) WITH TIME ZONE
 LAST_MANUPRG_TIME                                                                                                          TIMESTAMP(9) WITH TIME ZONE
 ADRDIR_VERSION                                                                                                             NUMBER
 ADRSCHM_VERSION                                                                                                            NUMBER
 ADRSCHMV_SUMMARY                                                                                                           NUMBER
 ADRALERT_VERSION                                                                                                           NUMBER
 CREATE_TIME                                                                                                                TIMESTAMP(9) WITH TIME ZONE
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1   select t.kqftanam, c.kqfconam
  2   from x$kqfta t, x$kqfco c
  3   where t.inst_id = 1
  4   and c.inst_id = 1
  5   and t.indx = c.kqfcotab
  6*  and kqftanam = 'X$DIAG_ADR_CONTROL'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_ADR_CONTROL             ADDR
X$DIAG_ADR_CONTROL             INDX
X$DIAG_ADR_CONTROL             INST_ID
X$DIAG_ADR_CONTROL             COLA
X$DIAG_ADR_CONTROL             COLB

Clearly the columns are wrong, these are the columns of X$KQFTVRTTST0! Let’s subtract X$KQFCO.KQFCOTAB by one:

SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1   select t.kqftanam, c.kqfconam
  2   from x$kqfta t, x$kqfco c
  3   where t.inst_id = 1
  4   and c.inst_id = 1
  5   and t.indx = c.kqfcotab-1
  6*  and kqftanam = 'X$DIAG_ADR_CONTROL'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_ADR_CONTROL             ADDR
X$DIAG_ADR_CONTROL             INDX
X$DIAG_ADR_CONTROL             INST_ID
X$DIAG_ADR_CONTROL             ADR_PATH_IDX
X$DIAG_ADR_CONTROL             ADR_HOME
X$DIAG_ADR_CONTROL             ADRID
X$DIAG_ADR_CONTROL             SHORTP_POLICY
X$DIAG_ADR_CONTROL             LONGP_POLICY
X$DIAG_ADR_CONTROL             LAST_MOD_TIME
X$DIAG_ADR_CONTROL             LAST_AUTOPRG_TIME
X$DIAG_ADR_CONTROL             LAST_MANUPRG_TIME
X$DIAG_ADR_CONTROL             ADRDIR_VERSION
X$DIAG_ADR_CONTROL             ADRSCHM_VERSION
X$DIAG_ADR_CONTROL             ADRSCHMV_SUMMARY
X$DIAG_ADR_CONTROL             ADRALERT_VERSION
X$DIAG_ADR_CONTROL             CREATE_TIME

And this is correct again!

Now let’s verify the highest X$KQFTA.INDX number in the version of this database:

SYS@o112 AS SYSDBA> select max(indx) from x$kqfta;

 MAX(INDX)
----------
       927

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 927;

KQFTANAM
------------------------------
X$DIAG_VTEST_EXISTS

SYS@o112 AS SYSDBA> desc X$DIAG_VTEST_EXISTS
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 ADR_PATH_IDX                                                                                                               VARCHAR2(445)
 ADR_HOME                                                                                                                   VARCHAR2(445)
 PROBLEM_ID                                                                                                                 NUMBER
 PROBLEM_KEY                                                                                                                VARCHAR2(552)
 FIRST_INCIDENT                                                                                                             NUMBER
 FIRSTINC_TIME                                                                                                              TIMESTAMP(9) WITH TIME ZONE
 LAST_INCIDENT                                                                                                              NUMBER
 LASTINC_TIME                                                                                                               TIMESTAMP(9) WITH TIME ZONE
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and kqftanam = 'X$DIAG_VTEST_EXISTS'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_VTEST_EXISTS            ADDR
X$DIAG_VTEST_EXISTS            INDX
X$DIAG_VTEST_EXISTS            INST_ID
X$DIAG_VTEST_EXISTS            ADR_PATH_IDX
X$DIAG_VTEST_EXISTS            ADR_HOME
X$DIAG_VTEST_EXISTS            INCIDENT_ID
X$DIAG_VTEST_EXISTS            PROBLEM_ID
X$DIAG_VTEST_EXISTS            CREATE_TIME
X$DIAG_VTEST_EXISTS            CLOSE_TIME
X$DIAG_VTEST_EXISTS            STATUS
X$DIAG_VTEST_EXISTS            FLAGS
X$DIAG_VTEST_EXISTS            FLOOD_CONTROLLED
X$DIAG_VTEST_EXISTS            ERROR_FACILITY
X$DIAG_VTEST_EXISTS            ERROR_NUMBER
X$DIAG_VTEST_EXISTS            ERROR_ARG1
X$DIAG_VTEST_EXISTS            ERROR_ARG2
X$DIAG_VTEST_EXISTS            ERROR_ARG3
X$DIAG_VTEST_EXISTS            ERROR_ARG4
X$DIAG_VTEST_EXISTS            ERROR_ARG5
X$DIAG_VTEST_EXISTS            ERROR_ARG6
X$DIAG_VTEST_EXISTS            ERROR_ARG7
X$DIAG_VTEST_EXISTS            ERROR_ARG8
X$DIAG_VTEST_EXISTS            SIGNALLING_COMPONENT
X$DIAG_VTEST_EXISTS            SIGNALLING_SUBCOMPONENT
X$DIAG_VTEST_EXISTS            SUSPECT_COMPONENT
X$DIAG_VTEST_EXISTS            SUSPECT_SUBCOMPONENT
X$DIAG_VTEST_EXISTS            ECID
X$DIAG_VTEST_EXISTS            IMPACT
X$DIAG_VTEST_EXISTS            IMPACT_STR
X$DIAG_VTEST_EXISTS            STATUS_STR
X$DIAG_VTEST_EXISTS            ERROR_ARG9
X$DIAG_VTEST_EXISTS            ERROR_ARG10
X$DIAG_VTEST_EXISTS            ERROR_ARG11
X$DIAG_VTEST_EXISTS            ERROR_ARG12

So that’s wrong, let’s try subtracting by one:

SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab-1
  6* and kqftanam = 'X$DIAG_VTEST_EXISTS'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$DIAG_VTEST_EXISTS            ADDR
X$DIAG_VTEST_EXISTS            INDX
X$DIAG_VTEST_EXISTS            INST_ID
X$DIAG_VTEST_EXISTS            ADR_PATH_IDX
X$DIAG_VTEST_EXISTS            ADR_HOME
X$DIAG_VTEST_EXISTS            PROBLEM_ID
X$DIAG_VTEST_EXISTS            PROBLEM_KEY
X$DIAG_VTEST_EXISTS            FIRST_INCIDENT
X$DIAG_VTEST_EXISTS            FIRSTINC_TIME
X$DIAG_VTEST_EXISTS            LAST_INCIDENT
X$DIAG_VTEST_EXISTS            LASTINC_TIME

And correct again!

So at least starting from X$KQFTA.INDX = 841 up to the highest INDX number, 927, X$KQFCO.KQFCOTAB is off by one.

Starting from there, I decided to test INDX = 1, which was consistent with KQFCOTAB. Then I decided to go down in INDX number starting from 841 being the number of X$KQFTVRTTST0. In my database number 840 is X$KTCNQROW, which was off by one, then I found something weird:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 839;

KQFTANAM
------------------------------
X$SKGXP_MISC

SYS@o112 AS SYSDBA> desc X$SKGXP_MISC
ERROR:
ORA-00942: table or view does not exist
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1      select t.kqftanam, c.kqfconam
  2      from x$kqfta t, x$kqfco c
  3      where t.inst_id = 1
  4      and c.inst_id = 1
  5      and t.indx = c.kqfcotab
  6*    and kqftanam = 'X$SKGXP_MISC'
SYS@o112 AS SYSDBA> /

no rows selected

So the X$ view X$SKGXP_MISC exists in X$KQFTA, but doesn’t exist in reality, nor in X$KQFCO. Please mind this is a separate, different issue from the X$KQFCO.KQFCOTAB inconsistency this blogpost is about. My current thinking is the issue of existing in X$KQFTA and not queryable and (rightfully because of that), not available in X$KQFCO, might be either just a screwup/bug, or the X$ tables being reserved in the X$ table array, but not actually created, maybe because of setting during compilation. Just a hunch, I don’t know.

The same issue of being visible in X$KQFTA and not existing is true for:
838, X$SKGXP_CONNECTION
837, X$SKGXP_PORT
836, X$KSXP_STATS

INDX number 835 is X$KSXPTESTTBL. This table has the X$KQFCO.KQFCOTAB off by one issue:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 835;

KQFTANAM
------------------------------
X$KSXPTESTTBL
SYS@o112 AS SYSDBA> desc X$KSXPTESTTBL
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 KSXPTESTF1                                                                                                                 NUMBER
 KSXPTESTF2                                                                                                                 VARCHAR2(10)
 KSXPTESTF3                                                                                                                 NUMBER
 KSXPTESTF4                                                                                                                 NUMBER
 KSXPTESTF5                                                                                                                 RAW(8)
 KSXPTESTF6                                                                                                                 DATE
 KSXPTESTF7                                                                                                                 TIMESTAMP(3)
 KSXPTESTF8                                                                                                                 TIMESTAMP(3) WITH TIME ZONE
 KSXPTESTF9                                                                                                                 VARCHAR2(10)
 KSXPTESTF10                                                                                                                RAW(2)
 KSXPTESTF11                                                                                                                RAW(10)
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and t.kqftanam = 'X$KSXPTESTTBL'
SYS@o112 AS SYSDBA> /

no rows selected
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab-1
  6* and t.kqftanam = 'X$KSXPTESTTBL'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$KSXPTESTTBL                  ADDR
X$KSXPTESTTBL                  INDX
X$KSXPTESTTBL                  INST_ID
X$KSXPTESTTBL                  KSXPTESTF1
X$KSXPTESTTBL                  KSXPTESTF2
X$KSXPTESTTBL                  KSXPTESTF3
X$KSXPTESTTBL                  KSXPTESTF4
X$KSXPTESTTBL                  KSXPTESTF5
X$KSXPTESTTBL                  KSXPTESTF6
X$KSXPTESTTBL                  KSXPTESTF7
X$KSXPTESTTBL                  KSXPTESTF8
X$KSXPTESTTBL                  KSXPTESTF9
X$KSXPTESTTBL                  KSXPTESTF10
X$KSXPTESTTBL                  KSXPTESTF11

After X$KSXPTESTTBL, INDX number 834 is X$CELL_NAME:

SYS@o112 AS SYSDBA> select kqftanam from x$kqfta where indx = 834;

KQFTANAM
------------------------------
X$CELL_NAME
SYS@o112 AS SYSDBA> desc X$CELL_NAME
 Name                                                                                                              Null?    Type
 ----------------------------------------------------------------------------------------------------------------- -------- ----------------------------------------------------------------------------
 ADDR                                                                                                                       RAW(8)
 INDX                                                                                                                       NUMBER
 INST_ID                                                                                                                    NUMBER
 CELLPATH_CELLROW                                                                                                           VARCHAR2(400)
 CELLHASHVAL_CELLROW                                                                                                        NUMBER
SYS@o112 AS SYSDBA> ed
Wrote file afiedt.buf

  1  select t.kqftanam, c.kqfconam
  2  from x$kqfta t, x$kqfco c
  3  where t.inst_id = 1
  4  and c.inst_id = 1
  5  and t.indx = c.kqfcotab
  6* and t.kqftanam = 'X$CELL_NAME'
SYS@o112 AS SYSDBA> /

KQFTANAM                       KQFCONAM
------------------------------ ------------------------------
X$CELL_NAME                    ADDR
X$CELL_NAME                    INDX
X$CELL_NAME                    INST_ID
X$CELL_NAME                    CELLPATH_CELLROW
X$CELL_NAME                    CELLHASHVAL_CELLROW

And with X$CELL_NAME the X$KQFTA.INDX=X$KQFCO.KQFCOTAB link is restored.

So, this means that starting with X$KSXPTESTTBL the X$KQFCO.KQFCOTAB number is off by one. In fact, I tested version 18 (18.4), and found that the value X$KQFTA.INDX is 1217, so much higher, but still with X$KSXPTESTTBL the inconsistency of X$KQFCO.KQFCOTAB being one too high to correctly join with X$KQFTA.INDX.

Conclusion.
This means that based on my current investigations, it seems that starting from Oracle database version 11.2.0.1 up to version 19 and current PSU, starting with the X$KQFTA.INDX value of the table X$KSXPTESTTBL in X$KQFTA, the X$KQFCO.KQFCOTAB value needs to be subtracted by one to get the correct fields from X$KQFCO.KQFCOTAB to join to X$KQFTA.INDX.

A lot of X$ tables are not impacted, the majority of the tables have an X$KQFTA.INDX value lower than X$KSXPTESTTBL, but there are a reasonable amount of tables that are impacted, a lot of them related to ADR (starting with X$DIAG).

The reason for showing an additional oddity with X$SKGXP_MISC, X$SKGXP_CONNECTION, X$SKGXP_PORT and X$KSXP_STATS is that the X$ table where the join problem starts is X$KSXPTESTTBL. All these tables have something to do with cross instance IPC communication (skgxp/ksxp), and a lot of them have the weird property that they don’t exist whilst are visible in X$KQFTA, this might have something to do with X$KQFCO.KQFCOTAB being off by one. Of course this is something I can’t check, because it requires source code access, but I thought it would be good to mention.

It’s interesting to see that the join of X$KQFTA.INDX to X$KQFCO.KQFCOTAB is present in Oracle’s data dictionary with the view V$INDEXED_FIXED_COLUMN:

SQL> select view_definition from v$fixed_view_definition where view_name = 'GV$INDEXED_FIXED_COLUMN';

VIEW_DEFINITION
--------------------------------------------------------------------------------
select c.inst_id,kqftanam, kqfcoidx, kqfconam, kqfcoipo, c.con_id  from x$kqfco
c, x$kqfta t where t.indx = c.kqfcotab and kqfcoidx != 0

Which by now should be clear is not correct for X$ tables with an X$KQFTA.INDX number equal or higher than X$KSXPTESTTBL.

It seems that the most eye-catching difference between Oracle database versions 19.5 and 19.4 is three underscore parameters spare parameters being changed to specifically named underscore parameters, two of them have a bug number in them which is not publicly visible.

In v$sysstat/v$sesstat, a group of statistics are renamed from ‘OS %’ to ‘Server %’. All these statistics are about networking. One changed parameter directly points to networking (tcpinfo). These statistics are added in version 19.

One DBA view was added: DBA_DV_COMMON_OPERATION_STATUS. This view is specific and owned by to DVSYS.

NAME
--------------------------------------------------
_fourth_spare_parameter
_one-hundred-and-fifty-third_spare_parameter
_one-hundred-and-forty-sixth_spare_parameter

parameters unique in version 19.5 versus 19.4

NAME
--------------------------------------------------
_bug29302220_tcpinfo_statistics_save_atexit
_bug29903454_ksws_enable_alb
_session_modp_list

parameter values changed isdefault between 19.4 versus 19.5

parameter values unique to 19.4 versus 19.5

parameter values unique to 19.5 versus 19.4

waitevents unique in version 19.4 versus 19.5

waitevents unique in version 19.5 versus 19.4

waitevents changed parameter description between 19.4 versus 19.5

x$ tables unique to 19.4 versus 19.5

x$ tables unique to 19.5 versus 19.4

x$ tables columns unique to 19.4 versus 19.5

x$ tables columns unique to 19.5 versus 19.4

v$ tables unique to 19.4 versus 19.5

v$ tables unique to 19.5 versus 19.4

v$ tables columns unique to 19.4 versus 19.5

v$ tables columns unique to 19.5 versus 19.4

gv$ tables unique to 19.4 versus 19.5

gv$ tables unique to 19.5 versus 19.4

gv$ tables columns unique to 19.4 versus 19.5

gv$ tables columns unique to 19.5 versus 19.4

sysstat statistics unique to 19.4 versus 19.5

NAME
----------------------------------------------------------------------------------------------------
OS Advertised Receive Window
OS Advertised Send Window
OS Data Segments In
OS Data Segments Out
OS Lost Packets
OS Path Maximum Transmission Unit(MTU)
OS Send Congestion Window
OS Time (usec) Busy Sending Data
OS Time (usec) Busy Sending Data under Congestion
OS Time (usec) Last Ack Received
OS Time (usec) Last Ack Sent
OS Time (usec) Last Data Received
OS Time (usec) Last Data Sent
OS Time (usec) Limited by Receive Window
OS Time (usec) Limited by Send Buffer
OS Time (usec) Round Trip Time
OS Time (usec) Round Trip Time Variance
OS Total Bytes Acked
OS Total Bytes Received
OS Total Number of Retransmitted Packets

sysstat statistics unique to 19.5 versus 19.4

NAME
----------------------------------------------------------------------------------------------------
Server Advertised Receive Window
Server Advertised Send Window
Server Data Segments In
Server Data Segments Out
Server Elapsed Time (msec) Last Ack Received
Server Elapsed Time (msec) Last Ack Sent
Server Elapsed Time (msec) Last Data Received
Server Elapsed Time (msec) Last Data Sent
Server Lost Packets
Server Path Maximum Transmission Unit(MTU)
Server Send Congestion Window
Server Time (usec) Approximate Elapsed Time from the packet dro
Server Time (usec) Busy Sending Data
Server Time (usec) Limited by Receive Window
Server Time (usec) Limited by Send Buffer
Server Time (usec) Round Trip Time
Server Time (usec) Round Trip Time Variance
Server Total Bytes Acked
Server Total Bytes Received
Server Total Number of Retransmitted Packets

sys_time_model statistics unique to 19.4 versus 19.5

sys_time_model statistics unique to 19.5 versus 19.4

dba tables unique to 19.4 versus 19.5

dba tables unique to 19.5 versus 19.4

NAME
----------------------------------------------------------------------------------------------------
DBA_DV_COMMON_OPERATION_STATUS

dba tables columns unique to 19.5 versus 19.4

dba tables columns unique to 19.4 versus 19.5

cdb tables unique to 19.4 versus 19.5

cdb tables unique to 19.5 versus 19.4

cdb tables column unique to 19.4 versus 19.5

cdb tables column unique to 19.5 versus 19.4

And here are the differences in symbols (c functions).
The functions unique to 19.4 versus 19.5 (alias removed) are partly jon thus java related functions, which all have ‘domain’ in the name. Plus some diverse functions.

There is a significant amount number of functions added. The first ones starting with ‘aux_’ and ‘decode_krb5’ might have something to do with ‘krb5’, alias Kerberos version 5.

There is a ‘dbnest_dir_check’ function in between ‘aux’ and ‘decode’. It seems ‘dbnest’ is an entire layer in the database code at least, however I do not know if it relates to something that is public knowledge.

Among all kinds of singular added functions, what further is noticeable, is a high number of ASM related functions being added, of which a lot of them refer to ‘oss’ (oracle storage server?).

Additionally, a bunch of ‘ksp_modp’ functions are added which might have something to do with the ‘_session_modp_list’ parameter, and a bunch of parameters starting with ‘ksws_alb’ which might have something to do with the ‘_bug29903454_ksws_enable_alb’ parameter.

Further a group of functions starting with ‘soda’, which might be related to DBMS_SODA: a package implementing Simple Oracle Document Access (SODA), for managing document collections using PL/SQL.

code symbol names unique in version 19.4 versus 19.5

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
drexogidx                                                    (drex)ogidx                                                  text index query ??
joncl_at_domains                                             (jon)cl_at_domains                                           java ??
joncl_domains                                                (jon)cl_domains                                              java ??
joncl_set_domains                                            (jon)cl_set_domains                                          java ??
joncl_snor_domains                                           (jon)cl_snor_domains                                         java ??
joniul_at_domains                                            (jon)iul_at_domains                                          java ??
joniul_domains                                               (jon)iul_domains                                             java ??
joniul_set_domains                                           (jon)iul_set_domains                                         java ??
joniul_snor_domains                                          (jon)iul_snor_domains                                        java ??
jonndl_at_domains                                            (jon)ndl_at_domains                                          java ??
jonndl_domains                                               (jon)ndl_domains                                             java ??
jonndl_set_domains                                           (jon)ndl_set_domains                                         java ??
jonndl_snor_domains                                          (jon)ndl_snor_domains                                        java ??
jonoce_at_domains                                            (jon)oce_at_domains                                          java ??
jonoce_domains                                               (jon)oce_domains                                             java ??
jonoce_set_domains                                           (jon)oce_set_domains                                         java ??
jonoce_snor_domains                                          (jon)oce_snor_domains                                        java ??
jonosd_at_domains                                            (jon)osd_at_domains                                          java ??
jonosd_domains                                               (jon)osd_domains                                             java ??
jonosd_set_domains                                           (jon)osd_set_domains                                         java ??
jonosd_snor_domains                                          (jon)osd_snor_domains                                        java ??
jonvcl_at_domains                                            (jon)vcl_at_domains                                          java ??
jonvcl_domains                                               (jon)vcl_domains                                             java ??
jonvcl_set_domains                                           (jon)vcl_set_domains                                         java ??
jonvcl_snor_domains                                          (jon)vcl_snor_domains                                        java ??
jsv_try_scan_3choice                                         (js)v_try_scan_3choice                                       job scheduing ??
kcbtse_raise_tbs_datafile_enc_mismatch_error                 (kcbtse)_raise_tbs_datafile_enc_mismatch_error               kernel cache buffers tablespace encryption ??
kffgAttrMaxPxCb                                              (kffg)AttrMaxPxCb                                            kernel automatic storage management KFF file group ??
kffgVolClientCheck                                           (kffg)VolClientCheck                                         kernel automatic storage management KFF file group ??
kgiflini                                                     (kgi)flini                                                   kernel generic instantiation manager ??
kgiflpop                                                     (kgi)flpop                                                   kernel generic instantiation manager ??
kjgcr_GetTopCPU                                              (kj)gcr_GetTopCPU                                            kernel lock management ??
kjgcr_RunSyncTask                                            (kj)gcr_RunSyncTask                                          kernel lock management ??
kjgcr_SlaveReqGetSlot                                        (kj)gcr_SlaveReqGetSlot                                      kernel lock management ??
kkdlpGetObjn                                                 (kkdl)pGetObjn                                               kernel compile dictionary lookup ??
kkjcjchk                                                     (kkj)cjchk                                                   kernel compile job queue ??
kkoarFreeCtx                                                 (kkoar)FreeCtx                                               kernel compile optimizer automatic (sql) reoptimisation ??
kkoarFreeStats                                               (kkoar)FreeStats                                             kernel compile optimizer automatic (sql) reoptimisation ??
kkqojeFroNoRid                                               (kkqoj)eFroNoRid                                             kernel compile query  outer join ??
kkquIsCorrTransInList                                        (kkqu)IsCorrTransInList                                      kernel compile query  subquery unnesting ??
kslsesftcb_int                                               (ksl)sesftcb_int                                             kernel service (VOS) latching and post-wait ??
ktcsnaInitMinActScnInEnv                                     (ktc)snaInitMinActScnInEnv                                   kernel transaction control component ??
nsmplxhash                                                   (ns)mplxhash                                                 network session ??
pdy7F89_Create_Frame_Descriptor                              (pdy)7F89_Create_Frame_Descriptor                            PLSQL code execution codegen MCode gen ??
pdzdM60_Make_RBST                                            (pdz)dM60_Make_RBST                                          PLSQL code execution codegen infrastructure ??
pdzdM66_Insert_At_Root                                       (pdz)dM66_Insert_At_Root                                     PLSQL code execution codegen infrastructure ??
pdzdM67_Create                                               (pdz)dM67_Create                                             PLSQL code execution codegen infrastructure ??
qerocnStart                                                  (qeroc)nStart                                                query execute rowsource object collection iterator ??
qesmmCUpdProfile                                             (qesmm)CUpdProfile                                           query execute services memory manager (pga) ??
qosdMonNonCGCol                                              (qosd)MonNonCGCol                                            query optimizer statistics sql plan directive ??
qsodasqlGetInsertOneSQL                                      (qsoda)sqlGetInsertOneSQL                                    query system DBMS_SODA ??

code symbol names unique in version 19.5 versus 19.4

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
aux_loadptr_external_principal_identifier_ptr                aux_loadptr_external_principal_identifier_ptr                ??
aux_loadptr_pkinit_krb5_principal_name                       aux_loadptr_pkinit_krb5_principal_name                       ??
aux_loadptr_ptr_kdf_alg_id                                   aux_loadptr_ptr_kdf_alg_id                                   ??
aux_loadptr_ptr_seqof_external_principal_identifier          aux_loadptr_ptr_seqof_external_principal_identifier          ??
aux_loadptr_ptr_supported_kdfs                               aux_loadptr_ptr_supported_kdfs                               ??
aux_loadptr_subject_pk_info_ptr                              aux_loadptr_subject_pk_info_ptr                              ??
aux_present_opt_ptr_kdf_alg_id                               aux_present_opt_ptr_kdf_alg_id                               ??
aux_present_opt_ptr_seqof_external_principal_identifier      aux_present_opt_ptr_seqof_external_principal_identifier      ??
aux_present_opt_ptr_supported_kdfs                           aux_present_opt_ptr_supported_kdfs                           ??
aux_present_opt_subject_pk_info_ptr                          aux_present_opt_subject_pk_info_ptr                          ??
aux_storeptr_external_principal_identifier_ptr               aux_storeptr_external_principal_identifier_ptr               ??
aux_storeptr_pkinit_krb5_principal_name                      aux_storeptr_pkinit_krb5_principal_name                      ??
aux_storeptr_ptr_kdf_alg_id                                  aux_storeptr_ptr_kdf_alg_id                                  ??
aux_storeptr_ptr_seqof_external_principal_identifier         aux_storeptr_ptr_seqof_external_principal_identifier         ??
aux_storeptr_ptr_supported_kdfs                              aux_storeptr_ptr_supported_kdfs                              ??
aux_storeptr_subject_pk_info_ptr                             aux_storeptr_subject_pk_info_ptr                             ??
dbnest_dir_check                                             dbnest_dir_check                                             ??
decode_krb5_auth_pack                                        decode_krb5_auth_pack                                        ??
decode_krb5_auth_pack_draft9                                 decode_krb5_auth_pack_draft9                                 ??
decode_krb5_kdc_dh_key_info                                  decode_krb5_kdc_dh_key_info                                  ??
decode_krb5_pa_pk_as_rep                                     decode_krb5_pa_pk_as_rep                                     ??
decode_krb5_pa_pk_as_req                                     decode_krb5_pa_pk_as_req                                     ??
decode_krb5_pa_pk_as_req_draft9                              decode_krb5_pa_pk_as_req_draft9                              ??
decode_krb5_principal_name                                   decode_krb5_principal_name                                   ??
decode_krb5_reply_key_pack                                   decode_krb5_reply_key_pack                                   ??
decode_krb5_reply_key_pack_draft9                            decode_krb5_reply_key_pack_draft9                            ??
decode_krb5_td_dh_parameters                                 decode_krb5_td_dh_parameters                                 ??
decode_krb5_td_trusted_certifiers                            decode_krb5_td_trusted_certifiers                            ??
encode_krb5_auth_pack                                        encode_krb5_auth_pack                                        ??
encode_krb5_auth_pack_draft9                                 encode_krb5_auth_pack_draft9                                 ??
encode_krb5_kdc_dh_key_info                                  encode_krb5_kdc_dh_key_info                                  ??
encode_krb5_pa_pk_as_rep                                     encode_krb5_pa_pk_as_rep                                     ??
encode_krb5_pa_pk_as_rep_draft9                              encode_krb5_pa_pk_as_rep_draft9                              ??
encode_krb5_pa_pk_as_req                                     encode_krb5_pa_pk_as_req                                     ??
encode_krb5_pa_pk_as_req_draft9                              encode_krb5_pa_pk_as_req_draft9                              ??
encode_krb5_reply_key_pack                                   encode_krb5_reply_key_pack                                   ??
encode_krb5_reply_key_pack_draft9                            encode_krb5_reply_key_pack_draft9                            ??
encode_krb5_td_dh_parameters                                 encode_krb5_td_dh_parameters                                 ??
encode_krb5_td_trusted_certifiers                            encode_krb5_td_trusted_certifiers                            ??
kafcpy_one_row                                               (kaf)cpy_one_row                                             kernel access fetch ??
kcbflc_reap_internal                                         (kcb)flc_reap_internal                                       kernel cache buffers ??
kcbo_incr_doc                                                (kcbo)_incr_doc                                              kernel cache buffers object queue ??
kcbtse_raise_tbs_datafile_enc_mismatch                       (kcbtse)_raise_tbs_datafile_enc_mismatch                     kernel cache buffers tablespace encryption ??
kcfis_clear_lob                                              (kcfis)_clear_lob                                            kernel cache file management intelligent storage ??
kcslcu8                                                      (kcs)lcu8                                                    kernel cache scn management ??
kdilm_row_diskcompress_policy_type                           (kdil)m_row_diskcompress_policy_type                         kernel data index load ??
kdiss_optimize                                               (kdiss)_optimize                                             kernel data index split skip scan ??
kdlfForcedType                                               (kdl)fForcedType                                             kernel data lob ??
kdmrIMCULoadAvg                                              (kdmr)IMCULoadAvg                                            kernel data in-memory data layer background ??
kewraxc_attribute_xcb                                        (kewr)axc_attribute_xcb                                      kernel event AWR repository ??
kewxgenstbcb                                                 (kewx)genstbcb                                               kernel event AWR sysaux? ??
kewxgsc_generic_space_cbf                                    (kewx)gsc_generic_space_cbf                                  kernel event AWR sysaux? ??
kfIsZdlra                                                    (kf)IsZdlra                                                  kernel automatic storage management ??
kfdDskComputeHashForRow                                      (kfd)DskComputeHashForRow                                    kernel automatic storage management disk ??
kfdDskTableCbInternal                                        (kfd)DskTableCbInternal                                      kernel automatic storage management disk ??
kffgAddChildBlk                                              (kffg)AddChildBlk                                            kernel automatic storage management KFF file group ??
kffgAddDefault                                               (kffg)AddDefault                                             kernel automatic storage management KFF file group ??
kffgAttrMaxParCb                                             (kffg)AttrMaxParCb                                           kernel automatic storage management KFF file group ??
kffgAttrParSchmOvrdCb                                        (kffg)AttrParSchmOvrdCb                                      kernel automatic storage management KFF file group ??
kffgAuthenticate                                             (kffg)Authenticate                                           kernel automatic storage management KFF file group ??
kffgCalDeltaRedChg                                           (kffg)CalDeltaRedChg                                         kernel automatic storage management KFF file group ??
kffgCalRealQuota                                             (kffg)CalRealQuota                                           kernel automatic storage management KFF file group ??
kffgCalTargetRedun                                           (kffg)CalTargetRedun                                         kernel automatic storage management KFF file group ??
kffgCmp                                                      (kffg)Cmp                                                    kernel automatic storage management KFF file group ??
kffgFillFg                                                   (kffg)FillFg                                                 kernel automatic storage management KFF file group ??
kffgFindFree                                                 (kffg)FindFree                                               kernel automatic storage management KFF file group ??
kffgGetParOvrd                                               (kffg)GetParOvrd                                             kernel automatic storage management KFF file group ??
kffgIsParitySupported                                        (kffg)IsParitySupported                                      kernel automatic storage management KFF file group ??
kffgIsRedunAvailable                                         (kffg)IsRedunAvailable                                       kernel automatic storage management KFF file group ??
kffgIsUnsplit                                                (kffg)IsUnsplit                                              kernel automatic storage management KFF file group ??
kffgMakeName                                                 (kffg)MakeName                                               kernel automatic storage management KFF file group ??
kffgParseParOvrd                                             (kffg)ParseParOvrd                                           kernel automatic storage management KFF file group ??
kffgVerifyRedunChg                                           (kffg)VerifyRedunChg                                         kernel automatic storage management KFF file group ??
kffgcSyncDBLazy                                              (kffg)cSyncDBLazy                                            kernel automatic storage management KFF file group ??
kffgcSyncEnd                                                 (kffg)cSyncEnd                                               kernel automatic storage management KFF file group ??
kffgftMap                                                    (kffg)ftMap                                                  kernel automatic storage management KFF file group ??
kffgptAddDefault                                             (kffg)ptAddDefault                                           kernel automatic storage management KFF file group ??
kffgptFixRedun                                               (kffg)ptFixRedun                                             kernel automatic storage management KFF file group ??
kffgptGetStrVal                                              (kffg)ptGetStrVal                                            kernel automatic storage management KFF file group ??
kffgptLookup                                                 (kffg)ptLookup                                               kernel automatic storage management KFF file group ??
kffgptLookupDump                                             (kffg)ptLookupDump                                           kernel automatic storage management KFF file group ??
kffgptParseNum                                               (kffg)ptParseNum                                             kernel automatic storage management KFF file group ??
kfgComputeGrpProp                                            (kfg)ComputeGrpProp                                          kernel automatic storage management diskgroups ??
kfgGrpTableCbInternal                                        (kfg)GrpTableCbInternal                                      kernel automatic storage management diskgroups ??
kfiQuantize2                                                 (kf)iQuantize2                                               kernel automatic storage management ??
kfioComputeStripeIndex                                       (kfio)ComputeStripeIndex                                     kernel automatic storage management translation I/O layer ??
kfioGetXSetSize                                              (kfio)GetXSetSize                                            kernel automatic storage management translation I/O layer ??
kfioLoadParity                                               (kfio)LoadParity                                             kernel automatic storage management translation I/O layer ??
kfioParityWriteBegin                                         (kfio)ParityWriteBegin                                       kernel automatic storage management translation I/O layer ??
kfk_alloc_kfkio_freelist                                     (kfk)_alloc_kfkio_freelist                                   kernel automatic storage management KFK ??
kfk_asm_ioerror                                              (kfk)_asm_ioerror                                            kernel automatic storage management KFK ??
kfk_bld_sbmtptr_arry                                         (kfk)_bld_sbmtptr_arry                                       kernel automatic storage management KFK ??
kfk_bld_waitptr_arry                                         (kfk)_bld_waitptr_arry                                       kernel automatic storage management KFK ??
kfk_block_asmlib                                             (kfk)_block_asmlib                                           kernel automatic storage management KFK ??
kfk_block_oss                                                (kfk)_block_oss                                              kernel automatic storage management KFK ??
kfk_block_ufs                                                (kfk)_block_ufs                                              kernel automatic storage management KFK ??
kfk_build_osdstruct                                          (kfk)_build_osdstruct                                        kernel automatic storage management KFK ??
kfk_chk_xor_valid                                            (kfk)_chk_xor_valid                                          kernel automatic storage management KFK ??
kfk_do_abn_xor_chk                                           (kfk)_do_abn_xor_chk                                         kernel automatic storage management KFK ??
kfk_error_emulate                                            (kfk)_error_emulate                                          kernel automatic storage management KFK ??
kfk_fill_ksfdrq                                              (kfk)_fill_ksfdrq                                            kernel automatic storage management KFK ??
kfk_find_io_subsys                                           (kfk)_find_io_subsys                                         kernel automatic storage management KFK ??
kfk_get_asmioc_oper                                          (kfk)_get_asmioc_oper                                        kernel automatic storage management KFK ??
kfk_get_free_kfkrq                                           (kfk)_get_free_kfkrq                                         kernel automatic storage management KFK ??
kfk_get_unreaped_ios                                         (kfk)_get_unreaped_ios                                       kernel automatic storage management KFK ??
kfk_hard_abn_set                                             (kfk)_hard_abn_set                                           kernel automatic storage management KFK ??
kfk_hard_getoffs                                             (kfk)_hard_getoffs                                           kernel automatic storage management KFK ??
kfk_iodone_merge_slave                                       (kfk)_iodone_merge_slave                                     kernel automatic storage management KFK ??
kfk_lib_abn_check                                            (kfk)_lib_abn_check                                          kernel automatic storage management KFK ??
kfk_lib_xor_check                                            (kfk)_lib_xor_check                                          kernel automatic storage management KFK ??
kfk_oss_cell_timedout                                        (kfk)_oss_cell_timedout                                      kernel automatic storage management KFK ??
kfk_oss_dump_orq                                             (kfk)_oss_dump_orq                                           kernel automatic storage management KFK ??
kfk_oss_get_orq                                              (kfk)_oss_get_orq                                            kernel automatic storage management KFK ??
kfk_oss_offload_fill_orq                                     (kfk)_oss_offload_fill_orq                                   kernel automatic storage management KFK ??
kfk_oss_offload_submit                                       (kfk)_oss_offload_submit                                     kernel automatic storage management KFK ??
kfk_oss_post_wait                                            (kfk)_oss_post_wait                                          kernel automatic storage management KFK ??
kfk_oss_put_orq                                              (kfk)_oss_put_orq                                            kernel automatic storage management KFK ??
kfk_parse_disk_path                                          (kfk)_parse_disk_path                                        kernel automatic storage management KFK ??
kfk_proc_submitted_lib_reqs                                  (kfk)_proc_submitted_lib_reqs                                kernel automatic storage management KFK ??
kfk_process_an_ioq                                           (kfk)_process_an_ioq                                         kernel automatic storage management KFK ??
kfk_put_free_kfkios                                          (kfk)_put_free_kfkios                                        kernel automatic storage management KFK ??
kfk_reap_io_from_kfks                                        (kfk)_reap_io_from_kfks                                      kernel automatic storage management KFK ??
kfk_reap_lib_async_io                                        (kfk)_reap_lib_async_io                                      kernel automatic storage management KFK ??
kfk_reap_oss_async_io                                        (kfk)_reap_oss_async_io                                      kernel automatic storage management KFK ??
kfk_reap_ufs_async_io                                        (kfk)_reap_ufs_async_io                                      kernel automatic storage management KFK ??
kfk_rebuild_kfdrqs                                           (kfk)_rebuild_kfdrqs                                         kernel automatic storage management KFK ??
kfk_retry_submit_oss_io                                      (kfk)_retry_submit_oss_io                                    kernel automatic storage management KFK ??
kfk_submit_lib_io                                            (kfk)_submit_lib_io                                          kernel automatic storage management KFK ??
kfk_submit_oss_io                                            (kfk)_submit_oss_io                                          kernel automatic storage management KFK ??
kfk_submit_oss_io_no_wtevt                                   (kfk)_submit_oss_io_no_wtevt                                 kernel automatic storage management KFK ??
kfk_submit_oss_io_wtevt                                      (kfk)_submit_oss_io_wtevt                                    kernel automatic storage management KFK ??
kfk_submit_ufs_io                                            (kfk)_submit_ufs_io                                          kernel automatic storage management KFK ??
kfk_transit_fetchIO                                          (kfk)_transit_fetchIO                                        kernel automatic storage management KFK ??
kfk_transit_waitIO                                           (kfk)_transit_waitIO                                         kernel automatic storage management KFK ??
kfk_update_iodone_stats                                      (kfk)_update_iodone_stats                                    kernel automatic storage management KFK ??
kfk_zero_reap_array                                          (kfk)_zero_reap_array                                        kernel automatic storage management KFK ??
kfmdIsDGMountedCluster                                       (kfmd)IsDGMountedCluster                                     kernel automatic storage management node monitor interface implementation layer
                                                                                                                          for diskgroup registration ??

kfmsGetGrpMemCnt                                             (kfms)GetGrpMemCnt                                           kernel automatic storage management node monitor interface implementation layers
                                                                                                                           support function interface ??

kgfdDiscoverSubmit                                           (kgfd)DiscoverSubmit                                         kernel generic ASM io subsystem driver ??
kgh_size_sanity_check                                        (kgh)_size_sanity_check                                      kernel generic heap manager ??
kjac_allow_tg                                                (kjac)_allow_tg                                              kernel lock management application continuity  ??
kjccspbat                                                    (kjc)cspbat                                                  kernel lock management communication ??
kkocfbFindOldActual                                          (kkocfb)FindOldActual                                        kernel compile optimizer cardinality feedback ??
kkopinlpInt                                                  (kko)pinlpInt                                                kernel compile optimizer ??
kkscsGetPQSharingParameters                                  (kkscs)GetPQSharingParameters                                kernel compile shared objects (cursor) child cursor ??
kkscsGetSessionLimitDOP                                      (kkscs)GetSessionLimitDOP                                    kernel compile shared objects (cursor) child cursor ??
kokegAddOneGarbageItem                                       (koke)gAddOneGarbageItem                                     kernel objects kernel side OBJECTS images ??
kokegRepinLob                                                (koke)gRepinLob                                              kernel objects kernel side OBJECTS images ??
kpochalts                                                    (kpo)chalts                                                  kernel programmatic interface oracle ??
kpossGetEncNonTemplateOverflow                               (kpo)ssGetEncNonTemplateOverflow                             kernel programmatic interface oracle ??
kpossSetEncNonTemplateOverflow                               (kpo)ssSetEncNonTemplateOverflow                             kernel programmatic interface oracle ??
kpossTemplateMatch_                                          (kpo)ssTemplateMatch_                                        kernel programmatic interface oracle ??
kpossTemplateSet                                             (kpo)ssTemplateSet                                           kernel programmatic interface oracle ??
kpueONSTraceWrite                                            (kpu)eONSTraceWrite                                          kernel programmatic interface user ??
kpueONSTrcCb                                                 (kpu)eONSTrcCb                                               kernel programmatic interface user ??
kpussTemplateSend                                            (kpu)ssTemplateSend                                          kernel programmatic interface user ??
kpuxcOpenCurCntDecr                                          (kpu)xcOpenCurCntDecr                                        kernel programmatic interface user ??
kpuxcOpenCurCntIncr                                          (kpu)xcOpenCurCntIncr                                        kernel programmatic interface user ??
kpuxcRefCurDfnMarkOpen                                       (kpu)xcRefCurDfnMarkOpen                                     kernel programmatic interface user ??
kpuxcRefCurOutBndMarkOpen                                    (kpu)xcRefCurOutBndMarkOpen                                  kernel programmatic interface user ??
kpuxcSessionSignatureRecv                                    (kpu)xcSessionSignatureRecv                                  kernel programmatic interface user ??
kpuxcSessionTemplateSend                                     (kpu)xcSessionTemplateSend                                   kernel programmatic interface user ??
kpuxcSessionTemplatesFree                                    (kpu)xcSessionTemplatesFree                                  kernel programmatic interface user ??
kpuxcTACService                                              (kpu)xcTACService                                            kernel programmatic interface user ??
kpuxsoCurPrefetchFree                                        (kpu)xsoCurPrefetchFree                                      kernel programmatic interface user ??
kpuxsoOperInit                                               (kpu)xsoOperInit                                             kernel programmatic interface user ??
krbbCountThreads                                             (krbb)CountThreads                                           kernel redo backup/restore creation of a backuppiece ??
ksdhng_wdat_rem_dup                                          (ksdhng)_wdat_rem_dup                                        kernel service (VOS) debug internal errors hang analyze ??
ksfdafGetXsetSize                                            (ksfdaf)GetXsetSize                                          kernel service (VOS) functions disk IO asm files ??
ksfdafParityWrite                                            (ksfdaf)ParityWrite                                          kernel service (VOS) functions disk IO asm files ??
ksfdafParityWriteAbort                                       (ksfdaf)ParityWriteAbort                                     kernel service (VOS) functions disk IO asm files ??
ksfdafParityWriteBegin                                       (ksfdaf)ParityWriteBegin                                     kernel service (VOS) functions disk IO asm files ??
ksfdafParityWriteEnd                                         (ksfdaf)ParityWriteEnd                                       kernel service (VOS) functions disk IO asm files ??
kslwt_event_stats                                            (kslwt)_event_stats                                          kernel service (VOS) latching and post-wait wait interface ??
kslwt_push_stats                                             (kslwt)_push_stats                                           kernel service (VOS) latching and post-wait wait interface ??
ksmsq_spendingi                                              (ksmsq)_spendingi                                            kernel service (VOS) memory sga heap message queue services ??
ksp_init_modp_send                                           (ksp)_init_modp_send                                         kernel service (VOS) parameter ??
ksp_modp_get_enckeyvals                                      (ksp)_modp_get_enckeyvals                                    kernel service (VOS) parameter ??
ksp_modp_get_keyvals                                         (ksp)_modp_get_keyvals                                       kernel service (VOS) parameter ??
ksp_modp_set_enckeyvals                                      (ksp)_modp_set_enckeyvals                                    kernel service (VOS) parameter ??
ksp_modp_set_keyvals                                         (ksp)_modp_set_keyvals                                       kernel service (VOS) parameter ??
ksp_modp_update_sign                                         (ksp)_modp_update_sign                                       kernel service (VOS) parameter ??
kspdecbuf                                                    (ksp)decbuf                                                  kernel service (VOS) parameter ??
kspencbuf                                                    (ksp)encbuf                                                  kernel service (VOS) parameter ??
ksucln_expired_cleanup_timer                                 (ksucln)_expired_cleanup_timer                               kernel service (VOS) user cleanup detached process ??
kswrListDelete                                               (ksw)rListDelete                                             kernel service (VOS) workgroup ??
ksws_alb_alloc_pga_memory                                    (ksws)_alb_alloc_pga_memory                                  kernel service (VOS) workgroup services ??
ksws_alb_block_service                                       (ksws)_alb_block_service                                     kernel service (VOS) workgroup services ??
ksws_alb_check_sessions_to_mark                              (ksws)_alb_check_sessions_to_mark                            kernel service (VOS) workgroup services ??
ksws_alb_check_threshold                                     (ksws)_alb_check_threshold                                   kernel service (VOS) workgroup services ??
ksws_alb_clear_mark_drain_session                            (ksws)_alb_clear_mark_drain_session                          kernel service (VOS) workgroup services ??
ksws_alb_dump_service                                        (ksws)_alb_dump_service                                      kernel service (VOS) workgroup services ??
ksws_alb_dump_session                                        (ksws)_alb_dump_session                                      kernel service (VOS) workgroup services ??
ksws_alb_filter_session                                      (ksws)_alb_filter_session                                    kernel service (VOS) workgroup services ??
ksws_alb_mark_target_sessions                                (ksws)_alb_mark_target_sessions                              kernel service (VOS) workgroup services ??
ksws_alb_process_directives                                  (ksws)_alb_process_directives                                kernel service (VOS) workgroup services ??
ksws_alb_run_gv_sql                                          (ksws)_alb_run_gv_sql                                        kernel service (VOS) workgroup services ??
ksws_alb_safe_session                                        (ksws)_alb_safe_session                                      kernel service (VOS) workgroup services ??
ksws_alb_safe_to_block_service                               (ksws)_alb_safe_to_block_service                             kernel service (VOS) workgroup services ??
ksws_alb_selection_bias_apply                                (ksws)_alb_selection_bias_apply                              kernel service (VOS) workgroup services ??
ksws_alb_set_mark_drain_session                              (ksws)_alb_set_mark_drain_session                            kernel service (VOS) workgroup services ??
ksws_alb_unblock_fix_delta_services                          (ksws)_alb_unblock_fix_delta_services                        kernel service (VOS) workgroup services ??
ksws_alb_unblock_service                                     (ksws)_alb_unblock_service                                   kernel service (VOS) workgroup services ??
ksws_alb_update_service_session_metrics                      (ksws)_alb_update_service_session_metrics                    kernel service (VOS) workgroup services ??
ksws_alb_update_threshold                                    (ksws)_alb_update_threshold                                  kernel service (VOS) workgroup services ??
ksws_alb_valid_session                                       (ksws)_alb_valid_session                                     kernel service (VOS) workgroup services ??
kwraChkOptdef                                                (k)wraChkOptdef                                              kernel ??
kwrachkopnTree                                               (k)wrachkopnTree                                             kernel ??
kwraplschkLogdef                                             (k)wraplschkLogdef                                           kernel ??
kxccXRkxck                                                   (kxcc)XRkxck                                                 kernel execution constraints catch all ??
kzctxgIsGlobal                                               (kzctxg)IsGlobal                                             kernel security Code-Based Access Control context routines global_context ??
kzrtGetAndCheckTriggerCache                                  (kzrt)GetAndCheckTriggerCache                                kernel security virtual private database privileges ??
kzrtTriggerPresent                                           (kzrt)TriggerPresent                                         kernel security virtual private database privileges ??
kzvdveCommOpBypass                                           (kzvd)veCommOpBypass                                         kernel security data vault ??
kzvdveIsAppCommonUserId                                      (kzvd)veIsAppCommonUserId                                    kernel security data vault ??
kzvradmAllowCommOp                                           (kz)vradmAllowCommOp                                         kernel security ??
kzvradmAllowCommOpCB                                         (kz)vradmAllowCommOpCB                                       kernel security ??
kzvradmAllowCommOp_int                                       (kz)vradmAllowCommOp_int                                     kernel security ??
kzvutilGetACOStatus                                          (kz)vutilGetACOStatus                                        kernel security ??
kzvutilGetACOStatus0                                         (kz)vutilGetACOStatus0                                       kernel security ??
psdvcn                                                       (psd)vcn                                                     PLSQL infrastructure programmatic interfaces to/from  ??
qesrcTol_New                                                 (qesrc)Tol_New                                               query execute services result cache ??
qsodasqlGetContentDefine                                     (qsoda)sqlGetContentDefine                                   query system DBMS_SODA ??
qsodasqlGetCreatedOnDefine                                   (qsoda)sqlGetCreatedOnDefine                                 query system DBMS_SODA ??
qsodasqlGetInsertSQL                                         (qsoda)sqlGetInsertSQL                                       query system DBMS_SODA ??
qsodasqlGetKeyDefine                                         (qsoda)sqlGetKeyDefine                                       query system DBMS_SODA ??
qsodasqlGetLastModDefine                                     (qsoda)sqlGetLastModDefine                                   query system DBMS_SODA ??
qsodasqlGetMediaDefine                                       (qsoda)sqlGetMediaDefine                                     query system DBMS_SODA ??
qsodasqlGetVersionDefine                                     (qsoda)sqlGetVersionDefine                                   query system DBMS_SODA ??
qsodasqlPrefetchPrepare                                      (qsoda)sqlPrefetchPrepare                                    query system DBMS_SODA ??
qsodasqlSelectDocFieldsSQL                                   (qsoda)sqlSelectDocFieldsSQL                                 query system DBMS_SODA ??
qsodaxArrayFetchDocuments                                    (qsoda)xArrayFetchDocuments                                  query system DBMS_SODA ??
qsodaxDefineDocumentFields                                   (qsoda)xDefineDocumentFields                                 query system DBMS_SODA ??
qsodaxGetPrefetchDocument                                    (qsoda)xGetPrefetchDocument                                  query system DBMS_SODA ??
sskgds_dw_is_context_valid                                   (sskg)ds_dw_is_context_valid                                 operating system dependent system kernel generic ??
sskgds_dw_is_pccontext_valid                                 (sskg)ds_dw_is_pccontext_valid                               operating system dependent system kernel generic ??

In the article oracle memory troubleshooting using analysis on heap dumps I introduced heap_analyze.awk.

The reason the tool exists is because I am using it myself. Therefore, I ran into additional things that I wanted the tool to do. I added some stuff, which is that significant, that I decided to make another blogpost to introduce the new features.

1. Percentages
In order to get an idea of the relative size of the summarised topic, I added a percentage. For example:

top 5 allocation by total size per alloc reason per heap
==================================================================================================
heap             alloc_reason            #chunks       total_size   %
--------------------------------------------------------------------------------------------------
sga heap(1,0)    perm                         19        182947456  72
sga heap(1,0)                                 43         14488960   5
sga heap(1,0)    SO private sga               18         14284168   5
sga heap(1,0)    KGLHD                      5787          4318400   1
sga heap(1,0)    KSFD SGA I/O b                1          4190416   1

2. Enhanced perm (permanent) memory descriptions
It seems that for PGA heap dumps, sometimes there is a description for memory area’s that are perm (permanent memory, memory allocated for the lifetime of the process). This is how that’s visible in the dump:

PERMANENT CHUNKS:
  Chunk     7fcbcad6c020 sz=    20632    perm      "perm           "  alo=20632
            7fcbcad6c020 sz=    20632    cpmlst    "callback hsn   "

I must say I don’t know what ‘cpmlst’ means, so if anyone knows or has a good guess, please let me know. However, the two addresses and sizes are an exact match, so I now change the alloc_reason for the cpmlst text.
This is helpful because there is a quite some memory allocated as perm. Sadly, this is not done for SGA dumps.

3. Excel mode
If you want to store and compare different dumps, one way of doing that is pasting the output in Microsoft excel. Once you set the ‘Text To Columns’ option to space as a separator, it will put the information in its own cells. But there are a few problems with that, one of them is that the heap names and alloc_reasons can have spaces in them, so that the placement of the figures can vary. I created excel mode for that.
If you set excel mode (set the variable excel_mode to 1 on line 5):

#!/bin/awk -f
BEGIN {
  printf "Heap dump analyzer v0.2 by Frits Hoogland\n";
  group_sum_nr=5;
  excel_mode=1;
}

In this mode, the horizontal lines (with ‘-‘ and ‘=’) are omitted in the output, and all spaces are changed to underscores, so a table stays consistent when pasted in excel:

heap             alloc_reason            #chunks       total_size   %
sga_heap(1,0)    perm___________              19        182947456  72
sga_heap(1,0)    _______________              43         14488960   5
sga_heap(1,0)    SO_private_sga_              18         14284168   5
sga_heap(1,0)    KGLHD__________            5787          4318400   1
sga_heap(1,0)    KSFD_SGA_I/O_b_               1          4190416   1

4. New table which shows memory by type
Another way of looking at memory in a heap is by grouping it by type. This allows you to very quickly see if a certain type of chunk is dominating a heap:

heap                        type    #chunks         min_size         max_size       total_size   %
--------------------------------------------------------------------------------------------------
top call heap               free          7            16408            65512           390064  99
top call heap           recreate          2              992              992             1984   0
top call heap               perm          2              120              904             1024   0

This is an overview of the top call heap of a session that is not active, and therefore most of it should be empty, which is true for this dump.

Once again, get the awk script here: https://gitlab.com/FritsHoogland/oracle_memory_analyze/blob/master/heap_analyze.awk

This blogpost is about analysing Oracle heap dumps. It is an extension to earlier work, Tanel Poder’s heap dump analyzer. So hat tip to Tanel, he’s done the hard work, I merely increased the presentation options. The heap analyser script that I wrote analyses Oracle heapdumps from the trace file that the dump was written to by the Oracle database. Because the heap dump representation is the same between PGA and SGA memory, it can work on both. The reason for this is that memory management is done by the same memory manager, and is commonly called ‘kgh’ (kernel generic heap) managed memory.

Please mind that for PGA analysis, not all memory is managed by the kgh memory manager. For example memory used for networking (sqlnet) is allocated totally outside of the kgh memory manager.

Let’s take the output of a PGA heap dump at level 29 (PGA, UGA, CGA, top call heaps, call heaps, session heap; executed via ‘alter session set events ‘immediate trace name heapdump level 29”):

$ ./heap_analyze.awk /u01/app/oracle/diag/rdbms/tt/tt/trace/tt_ora_6515.trc
Heap dump analyzer v0.1 by Frits Hoogland
heap size totals (all sizes in bytes)
==============================================================================================
heap name                       total size
----------------------------------------------------------------------------------------------
top call heap                       393096
callheap                             36200
pga heap                           1016816
top uga heap                        262048
session heap                        261744
------------------------------------------
total                              1969904

top 5 allocation by total size per alloc reason per heap
==============================================================================================
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
top call heap                                  5           354456
top call heap    callheap                     11            37616
top call heap    perm                          2             1024
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
callheap                                      10            35920
callheap         ctxtcds: kksQui               1              160
callheap         perm                          1              120
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
pga heap         perm                         44           290256
pga heap         Fixed UGA heap                1           226232
pga heap         diag pga                     21           214048
pga heap                                       7            62616
pga heap         kfkio bucket                  1            40984
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
top uga heap     session heap                  4           261928
top uga heap     perm                          1              120
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
session heap     perm                          3            63608
session heap                                   8            63528
session heap     koh-kghu sessi                5            28288
session heap     kxsFrame4kPage                6            24912
session heap     kxsFrame8kPage                3            24744

top 5 allocations by total size per alloc reason, type, chunk size per heap
==============================================================================================
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
top call heap                                  1       free           131048           131048
top call heap                                  2       free            65512           131024
top call heap                                  1       free            63496            63496
top call heap                                  1       free            28888            28888
top call heap    callheap                      6   freeable             4224            25344
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
callheap                                       6       free             4184            25104
callheap                                       2       free             4192             8384
callheap                                       1       free             1784             1784
callheap                                       1       free              648              648
callheap         ctxtcds: kksQui               1   freeable              160              160
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
pga heap         Fixed UGA heap                1   recreate           226232           226232
pga heap         perm                          1       perm            71720            71720
pga heap         diag pga                      1   freeable            55048            55048
pga heap         diag pga                      2   freeable            27456            54912
pga heap                                       1       free            47296            47296
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
session heap     perm                          1       perm            43584            43584
session heap                                   1       free            36544            36544
session heap     kxsFrame4kPage                6   freeable             4152            24912
session heap     kxsFrame8kPage                3   freeable             8248            24744
session heap                                   1       free            20352            20352
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
top uga heap     session heap                  3   freeable            65512           196536
top uga heap     session heap                  1   recreate            65392            65392
top uga heap     perm                          1       perm              120              120

top 5 allocations by total size per heap, alloc reason, type, chunk size
==============================================================================================
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
pga heap         Fixed UGA heap                1   recreate           226232           226232
top uga heap     session heap                  3   freeable            65512           196536
top call heap                                  1       free           131048           131048
top call heap                                  2       free            65512           131024
pga heap         perm                          1       perm            71720            71720

I figured that the first thing you want to see, are the heaps that are in the dump, and their sizes. That’s what is visible in rows 2-12.
I summed the heap sizes, which might make sense, or it might not.
In this case, with a heap dump that includes PGA, UGA and CGA plus session and call heaps, it means there are heaps in the dump that are part of another heap that is in the dump. So the total size here is bogus. This means that you need to have an understanding of what is actually dumped.

The next section, top 5 allocations by total size per alloc reason per dump, shows a per-heap summary by the allocation reason in the dump. If there’s no alloc reason, it’s free memory. Because this is a heap dump of a session that has done nothing (I just started sqlplus and ran the dump of its own PGA memory), you see that a lot of memory chunks are free memory. If you look closely to the allocation reasons, you can see that the ‘top call heap’ has a memory section that is called ‘callheap’ which is slightly larger than the ‘callheap’ section in the heap totals, and the ‘top uga heap’ section has a memory section called ‘session heap’ that is slightly larger than the ‘session heap’ section in the heap totals. In this case, it means that you can actually see the subheads in the parent heap allocation totals. The subheap size must be slightly larger in the parent heap because of headers in the memory allocations which are needed memory management. Please mind that this is based on my knowledge of how process memory is created, the only way to be absolutely sure is that a heap is part of another heap is to look at the memory addresses. This output only shows the heap names, not the addresses. The purpose of this section is to have an understanding of where memory is allocated to in a heap.

The following section, top 5 allocations by total size per alloc reason, type, chunk size per heap shows a per-heap summary by reason, type and chunksize, so you can investigate if specific types and sizes of chunks are causing issues or weird behaviour.

The last section is the same as the previous section, but doesn’t do it per heap. This is identical to what Tanel’s heapdump_analyzer shows.

You can find the heap_analyze.awk script here: https://gitlab.com/FritsHoogland/oracle_memory_analyze/blob/master/heap_analyze.awk

This is the output of a dump of the SGA of my small test database (oradebug dump heapdump 2):

$ ./heap_analyze.awk /u01/app/oracle/diag/rdbms/tt/tt/trace/tt_ora_9461.trc
Heap dump analyzer v0.1 by Frits Hoogland
heap size totals (all sizes in bytes)
==============================================================================================
heap name                       total size
----------------------------------------------------------------------------------------------
sga heap(1,0)                    285210904
sga heap(1,3)                     83885560
------------------------------------------
total                            369096464

top 5 allocation by total size per alloc reason per heap
==============================================================================================
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
sga heap(1,0)    perm                         22        188068904
sga heap(1,0)                                141         17975320
sga heap(1,0)    SO private sga               17         14268008
sga heap(1,0)    KQR PO                     8271          7568912
sga heap(1,0)    KGLHD                      9381          7023960
heap             alloc reason            #chunks       total size
----------------------------------------------------------------------------------------------
sga heap(1,3)                                 51         14120288
sga heap(1,3)    SQLA^6d9b8a7e               337          1380352
sga heap(1,3)    SQLA^31cc505b               167           684032
sga heap(1,3)    SQLA^aab93e92               162           663552
sga heap(1,3)    PLDIA^191e0a8d              155           634880

top 5 allocations by total size per alloc reason, type, chunk size per heap
==============================================================================================
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
sga heap(1,0)    perm                          1       perm         15937496         15937496
sga heap(1,0)    perm                          1       perm         15931312         15931312
sga heap(1,0)    perm                          1       perm         15811464         15811464
sga heap(1,0)    perm                          1       perm         15741952         15741952
sga heap(1,0)    perm                          1       perm         15723584         15723584
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
sga heap(1,3)                                  1       free          9883208          9883208
sga heap(1,3)                                  4     R-free           839480          3357920
sga heap(1,3)    SQLA^6d9b8a7e               336  freeableU             4096          1376256
sga heap(1,3)                                  1     R-free           839360           839360
sga heap(1,3)    SQLA^31cc505b               166  freeableU             4096           679936

top 5 allocations by total size per heap, alloc reason, type, chunk size
==============================================================================================
heap             alloc reason            #chunks       type       chunk size       total size
----------------------------------------------------------------------------------------------
sga heap(1,0)    perm                          1       perm         15937496         15937496
sga heap(1,0)    perm                          1       perm         15931312         15931312
sga heap(1,0)    perm                          1       perm         15811464         15811464
sga heap(1,0)    perm                          1       perm         15741952         15741952
sga heap(1,0)    perm                          1       perm         15723584         15723584

It’s interesting to see only subpool 1 sub-sub pool 1 and 3 are used. Subpool 1,1 contains a lot of permanent allocations and a lot of allocations that might have a more permanent nature, like KQR (dictionary cache) allocations, Subpool 1,3 seems to have allocations that are deemed more transient in nature, like SQLA (sql area) allocations. This might be wildly different in databases that are actually heavily used, this is an idle lab database.

Please mind it’s important to understand that dumping the shared pool requires obtaining the respective latches, so doing that on a live production system might lead to (severe) issues. Only do this if you know what you are doing. For the PGA there can only be one using process by definition, but be careful there too, if you request a PGA dump you are interacting with memory that is deemed private by the process that is using that.

If you require more rows to be shown than the 5 that are shown, set the ‘group_sum_nr’ variable to the amount you need on row 4 of the script.

There are a couple of underscore parameters changed from spare to named ones.
It’s interesting to see that in sysstat, ‘spare statistic 2’ changed to ‘cell XT granule IO bytes saved by HDFS tbs extent map scan’. This obviously has to do with big data access via cell servers. What is weird is that this is the only version where this had happened.

parameters unique in version 12.2.0.1.190416 versus 12.2.0.1.190716

NAME
--------------------------------------------------
_eleventh_spare_parameter
_one-hundred-and-forty-seventh_spare_parameter
_one-hundred-and-forty-sixth_spare_parameter
_tenth_spare_parameter

parameters unique in version 12.2.0.1.190716 versus 12.2.0.1.190416

NAME
--------------------------------------------------
_bug27693416_kgh_free_list_min_effort
_cell_offload_hybrid_processing
_dskm_single_instance
_seventh_spare_parameter

parameter values changed isdefault between 12.2.0.1.190416 versus 12.2.0.1.190716

parameter values unique to 12.2.0.1.190416 versus 12.2.0.1.190716

parameter values unique to 12.2.0.1.190716 versus 12.2.0.1.190416

waitevents unique in version 12.2.0.1.190416 versus 12.2.0.1.190716

waitevents unique in version 12.2.0.1.190716 versus 12.2.0.1.190416

waitevents changed parameter description between 12.2.0.1.190416 versus 12.2.0.1.190716

x$ tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

x$ tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

x$ tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

x$ tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

v$ tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

v$ tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

v$ tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

v$ tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

gv$ tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

gv$ tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

gv$ tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

gv$ tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

sysstat statistics unique to 12.2.0.1.190416 versus 12.2.0.1.190716

NAME
----------------------------------------------------------------------------------------------------
spare statistic 2

sysstat statistics unique to 12.2.0.1.190716 versus 12.2.0.1.190416

NAME
----------------------------------------------------------------------------------------------------
cell XT granule IO bytes saved by HDFS tbs extent map scan

sys_time_model statistics unique to 12.2.0.1.190416 versus 12.2.0.1.190716

sys_time_model statistics unique to 12.2.0.1.190716 versus 12.2.0.1.190416

dba tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

dba tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

dba tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

dba tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

cdb tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

cdb tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

cdb tables column unique to 12.2.0.1.190416 versus 12.2.0.1.190716

cdb tables column unique to 12.2.0.1.190716 versus 12.2.0.1.190416

And here are the differences in symbols (c functions).
Most notable:
– a lot of functions have been added that seem to be part of ‘little cms’, which seems to be a color management suite. I have no idea currently what these are doing in the oracle database executable.
– a significant amount of kola* (kernel objects lob) and prs* (parse) related functions have been added.

code symbol names unique in version 12.2.0.1.190416 versus 12.2.0.1.190716

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
free_ex_data                                                 free_ex_data                                                 ??
kcrfw_alfs_use_polling                                       (kcrfw_alfs)_use_polling                                     kernel cache recovery file write/broadcast SCN adaptive log file sync ??
kdmoBuildOperand                                             (kdmo)BuildOperand                                           kernel data in-memory data layer optimizer ??
kdmoDecompAvgCostPerSeg                                      (kdmo)DecompAvgCostPerSeg                                    kernel data in-memory data layer optimizer ??
kdmoDecompGetRate                                            (kdmo)DecompGetRate                                          kernel data in-memory data layer optimizer ??
kdmoGetStatHT                                                (kdmo)GetStatHT                                              kernel data in-memory data layer optimizer ??
kdmoIsValidTableCol                                          (kdmo)IsValidTableCol                                        kernel data in-memory data layer optimizer ??
kdmoMinMaxCountPrunableCUsSeg                                (kdmo)MinMaxCountPrunableCUsSeg                              kernel data in-memory data layer optimizer ??
kdmoMinMaxProcessPreds                                       (kdmo)MinMaxProcessPreds                                     kernel data in-memory data layer optimizer ??
kdmoPredEvalAvgCostPerSeg                                    (kdmo)PredEvalAvgCostPerSeg                                  kernel data in-memory data layer optimizer ??
kdmoPredEvalGetRate                                          (kdmo)PredEvalGetRate                                        kernel data in-memory data layer optimizer ??
kdpSizeOfGbyKdst                                             (kdp)SizeOfGbyKdst                                           kernel data archive compression: pcode ??
kdp_op_is_char                                               (kdp)_op_is_char                                             kernel data archive compression: pcode ??
kdp_pcode_calc_constant_area                                 (kdp)_pcode_calc_constant_area                               kernel data archive compression: pcode ??
kdp_pcode_setup_cbks                                         (kdp)_pcode_setup_cbks                                       kernel data archive compression: pcode ??
kdzdcol_agg_cols_imc_dict_dgk_update                         (kdzd)col_agg_cols_imc_dict_dgk_update                       kernel data archive compression decompression ??
kdzdcol_gby_dgk_create                                       (kdzd)col_gby_dgk_create                                     kernel data archive compression decompression ??
kdzdcol_gby_find_groups                                      (kdzd)col_gby_find_groups                                    kernel data archive compression decompression ??
kewmcomprmdif                                                (kewm)comprmdif                                              kernel event AWR metrics ??
kkesrc                                                       (kke)src                                                     kernel compile cost engine ??
kkfdGetQueReason                                             (kkfd)GetQueReason                                           kernel compile fast dataflow (PQ DFO) ??
kkqgCompareTTWAndWoGby                                       (kkq)gCompareTTWAndWoGby                                     kernel compile query  ??
kkqgMsqAggAppNampOrArgp                                      (kkq)gMsqAggAppNampOrArgp                                    kernel compile query  ??
kkqgMsqUnprs                                                 (kkq)gMsqUnprs                                               kernel compile query  ??
kkqjfCopyFroNoChn                                            (kkqjf)CopyFroNoChn                                          kernel compile query  join analysis join factorization ??
kkqvtOpnInView                                               (kkqvt)OpnInView                                             kernel compile query  vector transformation ??
krbr3sf                                                      (krbr)3sf                                                    kernel redo backup/restore restore and recovery ??
ksunetstat                                                   (ksu)netstat                                                 kernel service (VOS) user ??
ksws_map_fo_restore_to_text                                  (ksws)_map_fo_restore_to_text                                kernel service (VOS) workgroup services ??
kubsxiFetchOpenSetCpx                                        (ku)bsxiFetchOpenSetCpx                                      kernel utility ??
kzvdvechk_ownerid                                            (kzvd)vechk_ownerid                                          kernel security data vault ??
qerxaSetXVMCtx                                               (qer)xaSetXVMCtx                                             query execute rowsource ??
qkexrXC_SetOptTyp                                            (qke)xrXC_SetOptTyp                                          query kernel expressions ??
qksbgUnderOFE                                                (qksbg)UnderOFE                                              query kernel sql bind (variable) management(?) ??
qmxqcSetUDFVar2Parm                                          (qmxq)cSetUDFVar2Parm                                        query XDB XML Objects XML ??
qmxqcTreeAplyOp                                              (qmxq)cTreeAplyOp                                            query XDB XML Objects XML ??
qmxqcXPathNeedInputRetNodeRef                                (qmxq)cXPathNeedInputRetNodeRef                              query XDB XML Objects XML ??
qmxqtcPathHasPred                                            (qmxq)tcPathHasPred                                          query XDB XML Objects XML ??
qmxtr2IsFroRefdInFUP                                         (qmxt)r2IsFroRefdInFUP                                       query XDB XML Objects XML ??
qmxtr2MrkGrpChains                                           (qmxt)r2MrkGrpChains                                         query XDB XML Objects XML ??
set_dh                                                       set_dh                                                       ??

code symbol names unique in version 12.2.0.1.190716 versus 12.2.0.1.190416

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
AddConversion                                                AddConversion                                                ??
AllocArray                                                   AllocArray                                                   ??
AllocEmptyTransform                                          AllocEmptyTransform                                          ??
AllocateToneCurveStruct                                      AllocateToneCurveStruct                                      ??
BilinearInterp16                                             BilinearInterp16                                             ??
BilinearInterpFloat                                          BilinearInterpFloat                                          ??
BlackPreservingGrayOnlySampler                               BlackPreservingGrayOnlySampler                               ??
BlackPreservingKOnlyIntents                                  BlackPreservingKOnlyIntents                                  ??
BlackPreservingKPlaneIntents                                 BlackPreservingKPlaneIntents                                 ??
BlackPreservingSampler                                       BlackPreservingSampler                                       ??
CLUTElemDup                                                  CLUTElemDup                                                  ??
CLutElemTypeFree                                             CLutElemTypeFree                                             ??
CachedXFORM                                                  CachedXFORM                                                  ??
CachedXFORMGamutCheck                                        CachedXFORMGamutCheck                                        ??
Clipper                                                      Clipper                                                      ??
ComputeAbsoluteIntent                                        ComputeAbsoluteIntent                                        ??
CurveSetDup                                                  CurveSetDup                                                  ??
CurveSetElemTypeFree                                         CurveSetElemTypeFree                                         ??
CurvesDup                                                    CurvesDup                                                    ??
CurvesFree                                                   CurvesFree                                                   ??
DecideCurveType                                              DecideCurveType                                              ??
DecideLUTtypeA2B                                             DecideLUTtypeA2B                                             ??
DecideLUTtypeB2A                                             DecideLUTtypeB2A                                             ??
DecideTextDescType                                           DecideTextDescType                                           ??
DecideTextType                                               DecideTextType                                               ??
DecideXYZtype                                                DecideXYZtype                                                ??
DefaultEvalParametricFn                                      DefaultEvalParametricFn                                      ??
DefaultICCintents                                            DefaultICCintents                                            ??
DefaultLogErrorHandlerFunction                               DefaultLogErrorHandlerFunction                               ??
DupMatShaper                                                 DupMatShaper                                                 ??
DupNamedColorList                                            DupNamedColorList                                            ??
EstimateTAC                                                  EstimateTAC                                                  ??
Eval16nop1D                                                  Eval16nop1D                                                  ??
Eval1Input                                                   Eval1Input                                                   ??
Eval1InputFloat                                              Eval1InputFloat                                              ??
Eval4Inputs                                                  Eval4Inputs                                                  ??
Eval4InputsFloat                                             Eval4InputsFloat                                             ??
Eval5Inputs                                                  Eval5Inputs                                                  ??
Eval5InputsFloat                                             Eval5InputsFloat                                             ??
Eval6Inputs                                                  Eval6Inputs                                                  ??
Eval6InputsFloat                                             Eval6InputsFloat                                             ??
Eval7Inputs                                                  Eval7Inputs                                                  ??
Eval7InputsFloat                                             Eval7InputsFloat                                             ??
Eval8Inputs                                                  Eval8Inputs                                                  ??
Eval8InputsFloat                                             Eval8InputsFloat                                             ??
EvalNamedColor                                               EvalNamedColor                                               ??
EvalNamedColorPCS                                            EvalNamedColorPCS                                            ??
EvaluateCLUTfloat                                            EvaluateCLUTfloat                                            ??
EvaluateCLUTfloatIn16                                        EvaluateCLUTfloatIn16                                        ??
EvaluateCurves                                               EvaluateCurves                                               ??
EvaluateIdentity                                             EvaluateIdentity                                             ??
EvaluateLab2XYZ                                              EvaluateLab2XYZ                                              ??
EvaluateMatrix                                               EvaluateMatrix                                               ??
EvaluateXYZ2Lab                                              EvaluateXYZ2Lab                                              ??
FastEvaluateCurves16                                         FastEvaluateCurves16                                         ??
FastEvaluateCurves8                                          FastEvaluateCurves8                                          ??
FastIdentity16                                               FastIdentity16                                               ??
FileClose                                                    FileClose                                                    ??
FileRead                                                     FileRead                                                     ??
FileSeek                                                     FileSeek                                                     ??
FileTell                                                     FileTell                                                     ??
FileWrite                                                    FileWrite                                                    ??
FixWhiteMisalignment                                         FixWhiteMisalignment                                         ??
FloatXFORM                                                   FloatXFORM                                                   ??
FreeMatShaper                                                FreeMatShaper                                                ??
FreeNamedColorList                                           FreeNamedColorList                                           ??
GamutSampler                                                 GamutSampler                                                 ??
GenericMPEdup                                                GenericMPEdup                                                ??
GenericMPEfree                                               GenericMPEfree                                               ??
IdentitySampler                                              IdentitySampler                                              ??
InkLimitingSampler                                           InkLimitingSampler                                           ??
Java_sun_java2d_cmm_lcms_LCMS_colorConvert                   Java_sun_java2d_cmm_lcms_LCMS_colorConvert                   ??
Java_sun_java2d_cmm_lcms_LCMS_createNativeTransform          Java_sun_java2d_cmm_lcms_LCMS_createNativeTransform          ??
Java_sun_java2d_cmm_lcms_LCMS_getProfileDataNative           Java_sun_java2d_cmm_lcms_LCMS_getProfileDataNative           ??
Java_sun_java2d_cmm_lcms_LCMS_getProfileID                   Java_sun_java2d_cmm_lcms_LCMS_getProfileID                   ??
Java_sun_java2d_cmm_lcms_LCMS_getProfileSizeNative           Java_sun_java2d_cmm_lcms_LCMS_getProfileSizeNative           ??
Java_sun_java2d_cmm_lcms_LCMS_getTagNative                   Java_sun_java2d_cmm_lcms_LCMS_getTagNative                   ??
Java_sun_java2d_cmm_lcms_LCMS_initLCMS                       Java_sun_java2d_cmm_lcms_LCMS_initLCMS                       ??
Java_sun_java2d_cmm_lcms_LCMS_loadProfileNative              Java_sun_java2d_cmm_lcms_LCMS_loadProfileNative              ??
Java_sun_java2d_cmm_lcms_LCMS_setTagDataNative               Java_sun_java2d_cmm_lcms_LCMS_setTagDataNative               ??
LCMS_freeProfile                                             LCMS_freeProfile                                             ??
LCMS_freeTransform                                           LCMS_freeTransform                                           ??
LinLerp1D                                                    LinLerp1D                                                    ??
LinLerp1Dfloat                                               LinLerp1Dfloat                                               ??
MatShaperEval16                                              MatShaperEval16                                              ??
MatrixElemDup                                                MatrixElemDup                                                ??
MatrixElemTypeFree                                           MatrixElemTypeFree                                           ??
MemoryClose                                                  MemoryClose                                                  ??
MemoryRead                                                   MemoryRead                                                   ??
MemorySeek                                                   MemorySeek                                                   ??
MemoryTell                                                   MemoryTell                                                   ??
MemoryWrite                                                  MemoryWrite                                                  ??
NULLClose                                                    NULLClose                                                    ??
NULLRead                                                     NULLRead                                                     ??
NULLSeek                                                     NULLSeek                                                     ??
NULLTell                                                     NULLTell                                                     ??
NULLWrite                                                    NULLWrite                                                    ??
NullFloatXFORM                                               NullFloatXFORM                                               ??
NullXFORM                                                    NullXFORM                                                    ??
OptimizeByComputingLinearization                             OptimizeByComputingLinearization                             ??
OptimizeByJoiningCurves                                      OptimizeByJoiningCurves                                      ??
OptimizeByResampling                                         OptimizeByResampling                                         ??
OptimizeMatrixShaper                                         OptimizeMatrixShaper                                         ??
Pack1Byte                                                    Pack1Byte                                                    ??
Pack1ByteReversed                                            Pack1ByteReversed                                            ??
Pack1ByteSkip1                                               Pack1ByteSkip1                                               ??
Pack1ByteSkip1SwapFirst                                      Pack1ByteSkip1SwapFirst                                      ??
Pack1Word                                                    Pack1Word                                                    ??
Pack1WordBigEndian                                           Pack1WordBigEndian                                           ??
Pack1WordReversed                                            Pack1WordReversed                                            ??
Pack1WordSkip1                                               Pack1WordSkip1                                               ??
Pack1WordSkip1SwapFirst                                      Pack1WordSkip1SwapFirst                                      ??
Pack3Bytes                                                   Pack3Bytes                                                   ??
Pack3BytesAndSkip1                                           Pack3BytesAndSkip1                                           ??
Pack3BytesAndSkip1Optimized                                  Pack3BytesAndSkip1Optimized                                  ??
Pack3BytesAndSkip1Swap                                       Pack3BytesAndSkip1Swap                                       ??
Pack3BytesAndSkip1SwapFirst                                  Pack3BytesAndSkip1SwapFirst                                  ??
Pack3BytesAndSkip1SwapFirstOptimized                         Pack3BytesAndSkip1SwapFirstOptimized                         ??
Pack3BytesAndSkip1SwapOptimized                              Pack3BytesAndSkip1SwapOptimized                              ??
Pack3BytesAndSkip1SwapSwapFirst                              Pack3BytesAndSkip1SwapSwapFirst                              ??
Pack3BytesAndSkip1SwapSwapFirstOptimized                     Pack3BytesAndSkip1SwapSwapFirstOptimized                     ??
Pack3BytesOptimized                                          Pack3BytesOptimized                                          ??
Pack3BytesSwap                                               Pack3BytesSwap                                               ??
Pack3BytesSwapOptimized                                      Pack3BytesSwapOptimized                                      ??
Pack3Words                                                   Pack3Words                                                   ??
Pack3WordsAndSkip1                                           Pack3WordsAndSkip1                                           ??
Pack3WordsAndSkip1Swap                                       Pack3WordsAndSkip1Swap                                       ??
Pack3WordsAndSkip1SwapFirst                                  Pack3WordsAndSkip1SwapFirst                                  ??
Pack3WordsAndSkip1SwapSwapFirst                              Pack3WordsAndSkip1SwapSwapFirst                              ??
Pack3WordsBigEndian                                          Pack3WordsBigEndian                                          ??
Pack3WordsSwap                                               Pack3WordsSwap                                               ??
Pack4Bytes                                                   Pack4Bytes                                                   ??
Pack4BytesReverse                                            Pack4BytesReverse                                            ??
Pack4BytesSwap                                               Pack4BytesSwap                                               ??
Pack4BytesSwapFirst                                          Pack4BytesSwapFirst                                          ??
Pack4BytesSwapSwapFirst                                      Pack4BytesSwapSwapFirst                                      ??
Pack4Words                                                   Pack4Words                                                   ??
Pack4WordsBigEndian                                          Pack4WordsBigEndian                                          ??
Pack4WordsReverse                                            Pack4WordsReverse                                            ??
Pack4WordsSwap                                               Pack4WordsSwap                                               ??
Pack6Bytes                                                   Pack6Bytes                                                   ??
Pack6BytesSwap                                               Pack6BytesSwap                                               ??
Pack6Words                                                   Pack6Words                                                   ??
Pack6WordsSwap                                               Pack6WordsSwap                                               ??
PackALabV2_8                                                 PackALabV2_8                                                 ??
PackAnyBytes                                                 PackAnyBytes                                                 ??
PackAnyWords                                                 PackAnyWords                                                 ??
PackDoubleFrom16                                             PackDoubleFrom16                                             ??
PackDoublesFromFloat                                         PackDoublesFromFloat                                         ??
PackFloatFrom16                                              PackFloatFrom16                                              ??
PackFloatsFromFloat                                          PackFloatsFromFloat                                          ??
PackHalfFrom16                                               PackHalfFrom16                                               ??
PackHalfFromFloat                                            PackHalfFromFloat                                            ??
PackLabDoubleFrom16                                          PackLabDoubleFrom16                                          ??
PackLabDoubleFromFloat                                       PackLabDoubleFromFloat                                       ??
PackLabFloatFrom16                                           PackLabFloatFrom16                                           ??
PackLabFloatFromFloat                                        PackLabFloatFromFloat                                        ??
PackLabV2_16                                                 PackLabV2_16                                                 ??
PackLabV2_8                                                  PackLabV2_8                                                  ??
PackPlanarBytes                                              PackPlanarBytes                                              ??
PackPlanarWords                                              PackPlanarWords                                              ??
PackXYZDoubleFrom16                                          PackXYZDoubleFrom16                                          ??
PackXYZDoubleFromFloat                                       PackXYZDoubleFromFloat                                       ??
PackXYZFloatFrom16                                           PackXYZFloatFrom16                                           ??
PackXYZFloatFromFloat                                        PackXYZFloatFromFloat                                        ??
PreOptimize                                                  PreOptimize                                                  ??
PrecalculatedXFORM                                           PrecalculatedXFORM                                           ??
PrecalculatedXFORMGamutCheck                                 PrecalculatedXFORMGamutCheck                                 ??
Prelin16dup                                                  Prelin16dup                                                  ??
Prelin8dup                                                   Prelin8dup                                                   ??
Prelin8free                                                  Prelin8free                                                  ??
PrelinEval16                                                 PrelinEval16                                                 ??
PrelinEval8                                                  PrelinEval8                                                  ??
PrelinOpt16free                                              PrelinOpt16free                                              ??
R_SSL_clear_options                                          R_SSL_clear_options                                          ??
R_SSL_clear_options_by_type                                  R_SSL_clear_options_by_type                                  ??
ReadMPECurve                                                 ReadMPECurve                                                 ??
ReadMPEElem                                                  ReadMPEElem                                                  ??
ReadSeqID                                                    ReadSeqID                                                    ??
ReadSetOfCurves                                              ReadSetOfCurves                                              ??
SaveTags                                                     SaveTags                                                     ??
TetrahedralInterp16                                          TetrahedralInterp16                                          ??
TetrahedralInterpFloat                                       TetrahedralInterpFloat                                       ??
TrilinearInterp16                                            TrilinearInterp16                                            ??
TrilinearInterpFloat                                         TrilinearInterpFloat                                         ??
Type_Chromaticity_Dup                                        Type_Chromaticity_Dup                                        ??
Type_Chromaticity_Free                                       Type_Chromaticity_Free                                       ??
Type_Chromaticity_Read                                       Type_Chromaticity_Read                                       ??
Type_Chromaticity_Write                                      Type_Chromaticity_Write                                      ??
Type_ColorantOrderType_Dup                                   Type_ColorantOrderType_Dup                                   ??
Type_ColorantOrderType_Free                                  Type_ColorantOrderType_Free                                  ??
Type_ColorantOrderType_Read                                  Type_ColorantOrderType_Read                                  ??
Type_ColorantOrderType_Write                                 Type_ColorantOrderType_Write                                 ??
Type_ColorantTable_Dup                                       Type_ColorantTable_Dup                                       ??
Type_ColorantTable_Free                                      Type_ColorantTable_Free                                      ??
Type_ColorantTable_Read                                      Type_ColorantTable_Read                                      ??
Type_ColorantTable_Write                                     Type_ColorantTable_Write                                     ??
Type_CrdInfo_Dup                                             Type_CrdInfo_Dup                                             ??
Type_CrdInfo_Free                                            Type_CrdInfo_Free                                            ??
Type_CrdInfo_Read                                            Type_CrdInfo_Read                                            ??
Type_CrdInfo_Write                                           Type_CrdInfo_Write                                           ??
Type_Curve_Dup                                               Type_Curve_Dup                                               ??
Type_Curve_Free                                              Type_Curve_Free                                              ??
Type_Curve_Read                                              Type_Curve_Read                                              ??
Type_Curve_Write                                             Type_Curve_Write                                             ??
Type_Data_Dup                                                Type_Data_Dup                                                ??
Type_Data_Free                                               Type_Data_Free                                               ??
Type_Data_Read                                               Type_Data_Read                                               ??
Type_Data_Write                                              Type_Data_Write                                              ??
Type_DateTime_Dup                                            Type_DateTime_Dup                                            ??
Type_DateTime_Free                                           Type_DateTime_Free                                           ??
Type_DateTime_Read                                           Type_DateTime_Read                                           ??
Type_DateTime_Write                                          Type_DateTime_Write                                          ??
Type_Dictionary_Dup                                          Type_Dictionary_Dup                                          ??
Type_Dictionary_Free                                         Type_Dictionary_Free                                         ??
Type_Dictionary_Read                                         Type_Dictionary_Read                                         ??
Type_Dictionary_Write                                        Type_Dictionary_Write                                        ??
Type_LUT16_Dup                                               Type_LUT16_Dup                                               ??
Type_LUT16_Free                                              Type_LUT16_Free                                              ??
Type_LUT16_Read                                              Type_LUT16_Read                                              ??
Type_LUT16_Write                                             Type_LUT16_Write                                             ??
Type_LUT8_Dup                                                Type_LUT8_Dup                                                ??
Type_LUT8_Free                                               Type_LUT8_Free                                               ??
Type_LUT8_Read                                               Type_LUT8_Read                                               ??
Type_LUT8_Write                                              Type_LUT8_Write                                              ??
Type_LUTA2B_Dup                                              Type_LUTA2B_Dup                                              ??
Type_LUTA2B_Free                                             Type_LUTA2B_Free                                             ??
Type_LUTA2B_Read                                             Type_LUTA2B_Read                                             ??
Type_LUTA2B_Write                                            Type_LUTA2B_Write                                            ??
Type_LUTB2A_Dup                                              Type_LUTB2A_Dup                                              ??
Type_LUTB2A_Free                                             Type_LUTB2A_Free                                             ??
Type_LUTB2A_Read                                             Type_LUTB2A_Read                                             ??
Type_LUTB2A_Write                                            Type_LUTB2A_Write                                            ??
Type_MLU_Dup                                                 Type_MLU_Dup                                                 ??
Type_MLU_Free                                                Type_MLU_Free                                                ??
Type_MLU_Read                                                Type_MLU_Read                                                ??
Type_MLU_Write                                               Type_MLU_Write                                               ??
Type_MPE_Dup                                                 Type_MPE_Dup                                                 ??
Type_MPE_Free                                                Type_MPE_Free                                                ??
Type_MPE_Read                                                Type_MPE_Read                                                ??
Type_MPE_Write                                               Type_MPE_Write                                               ??
Type_MPEclut_Read                                            Type_MPEclut_Read                                            ??
Type_MPEclut_Write                                           Type_MPEclut_Write                                           ??
Type_MPEcurve_Read                                           Type_MPEcurve_Read                                           ??
Type_MPEcurve_Write                                          Type_MPEcurve_Write                                          ??
Type_MPEmatrix_Read                                          Type_MPEmatrix_Read                                          ??
Type_MPEmatrix_Write                                         Type_MPEmatrix_Write                                         ??
Type_Measurement_Dup                                         Type_Measurement_Dup                                         ??
Type_Measurement_Free                                        Type_Measurement_Free                                        ??
Type_Measurement_Read                                        Type_Measurement_Read                                        ??
Type_Measurement_Write                                       Type_Measurement_Write                                       ??
Type_NamedColor_Dup                                          Type_NamedColor_Dup                                          ??
Type_NamedColor_Free                                         Type_NamedColor_Free                                         ??
Type_NamedColor_Read                                         Type_NamedColor_Read                                         ??
Type_NamedColor_Write                                        Type_NamedColor_Write                                        ??
Type_ParametricCurve_Dup                                     Type_ParametricCurve_Dup                                     ??
Type_ParametricCurve_Free                                    Type_ParametricCurve_Free                                    ??
Type_ParametricCurve_Read                                    Type_ParametricCurve_Read                                    ??
Type_ParametricCurve_Write                                   Type_ParametricCurve_Write                                   ??
Type_ProfileSequenceDesc_Dup                                 Type_ProfileSequenceDesc_Dup                                 ??
Type_ProfileSequenceDesc_Free                                Type_ProfileSequenceDesc_Free                                ??
Type_ProfileSequenceDesc_Read                                Type_ProfileSequenceDesc_Read                                ??
Type_ProfileSequenceDesc_Write                               Type_ProfileSequenceDesc_Write                               ??
Type_ProfileSequenceId_Dup                                   Type_ProfileSequenceId_Dup                                   ??
Type_ProfileSequenceId_Free                                  Type_ProfileSequenceId_Free                                  ??
Type_ProfileSequenceId_Read                                  Type_ProfileSequenceId_Read                                  ??
Type_ProfileSequenceId_Write                                 Type_ProfileSequenceId_Write                                 ??
Type_S15Fixed16_Dup                                          Type_S15Fixed16_Dup                                          ??
Type_S15Fixed16_Free                                         Type_S15Fixed16_Free                                         ??
Type_S15Fixed16_Read                                         Type_S15Fixed16_Read                                         ??
Type_S15Fixed16_Write                                        Type_S15Fixed16_Write                                        ??
Type_Screening_Dup                                           Type_Screening_Dup                                           ??
Type_Screening_Free                                          Type_Screening_Free                                          ??
Type_Screening_Read                                          Type_Screening_Read                                          ??
Type_Screening_Write                                         Type_Screening_Write                                         ??
Type_Signature_Dup                                           Type_Signature_Dup                                           ??
Type_Signature_Free                                          Type_Signature_Free                                          ??
Type_Signature_Read                                          Type_Signature_Read                                          ??
Type_Signature_Write                                         Type_Signature_Write                                         ??
Type_Text_Description_Dup                                    Type_Text_Description_Dup                                    ??
Type_Text_Description_Free                                   Type_Text_Description_Free                                   ??
Type_Text_Description_Read                                   Type_Text_Description_Read                                   ??
Type_Text_Description_Write                                  Type_Text_Description_Write                                  ??
Type_Text_Dup                                                Type_Text_Dup                                                ??
Type_Text_Free                                               Type_Text_Free                                               ??
Type_Text_Read                                               Type_Text_Read                                               ??
Type_Text_Write                                              Type_Text_Write                                              ??
Type_U16Fixed16_Dup                                          Type_U16Fixed16_Dup                                          ??
Type_U16Fixed16_Free                                         Type_U16Fixed16_Free                                         ??
Type_U16Fixed16_Read                                         Type_U16Fixed16_Read                                         ??
Type_U16Fixed16_Write                                        Type_U16Fixed16_Write                                        ??
Type_UcrBg_Dup                                               Type_UcrBg_Dup                                               ??
Type_UcrBg_Free                                              Type_UcrBg_Free                                              ??
Type_UcrBg_Read                                              Type_UcrBg_Read                                              ??
Type_UcrBg_Write                                             Type_UcrBg_Write                                             ??
Type_ViewingConditions_Dup                                   Type_ViewingConditions_Dup                                   ??
Type_ViewingConditions_Free                                  Type_ViewingConditions_Free                                  ??
Type_ViewingConditions_Read                                  Type_ViewingConditions_Read                                  ??
Type_ViewingConditions_Write                                 Type_ViewingConditions_Write                                 ??
Type_XYZ_Dup                                                 Type_XYZ_Dup                                                 ??
Type_XYZ_Free                                                Type_XYZ_Free                                                ??
Type_XYZ_Read                                                Type_XYZ_Read                                                ??
Type_XYZ_Write                                               Type_XYZ_Write                                               ??
Type_vcgt_Dup                                                Type_vcgt_Dup                                                ??
Type_vcgt_Free                                               Type_vcgt_Free                                               ??
Type_vcgt_Read                                               Type_vcgt_Read                                               ??
Type_vcgt_Write                                              Type_vcgt_Write                                              ??
Unroll1Byte                                                  Unroll1Byte                                                  ??
Unroll1ByteReversed                                          Unroll1ByteReversed                                          ??
Unroll1ByteSkip1                                             Unroll1ByteSkip1                                             ??
Unroll1ByteSkip2                                             Unroll1ByteSkip2                                             ??
Unroll1Word                                                  Unroll1Word                                                  ??
Unroll1WordReversed                                          Unroll1WordReversed                                          ??
Unroll1WordSkip3                                             Unroll1WordSkip3                                             ??
Unroll2Bytes                                                 Unroll2Bytes                                                 ??
Unroll2Words                                                 Unroll2Words                                                 ??
Unroll3Bytes                                                 Unroll3Bytes                                                 ??
Unroll3BytesSkip1Swap                                        Unroll3BytesSkip1Swap                                        ??
Unroll3BytesSkip1SwapFirst                                   Unroll3BytesSkip1SwapFirst                                   ??
Unroll3BytesSkip1SwapSwapFirst                               Unroll3BytesSkip1SwapSwapFirst                               ??
Unroll3BytesSwap                                             Unroll3BytesSwap                                             ??
Unroll3Words                                                 Unroll3Words                                                 ??
Unroll3WordsSkip1Swap                                        Unroll3WordsSkip1Swap                                        ??
Unroll3WordsSkip1SwapFirst                                   Unroll3WordsSkip1SwapFirst                                   ??
Unroll3WordsSwap                                             Unroll3WordsSwap                                             ??
Unroll4Bytes                                                 Unroll4Bytes                                                 ??
Unroll4BytesReverse                                          Unroll4BytesReverse                                          ??
Unroll4BytesSwap                                             Unroll4BytesSwap                                             ??
Unroll4BytesSwapFirst                                        Unroll4BytesSwapFirst                                        ??
Unroll4BytesSwapSwapFirst                                    Unroll4BytesSwapSwapFirst                                    ??
Unroll4Words                                                 Unroll4Words                                                 ??
Unroll4WordsReverse                                          Unroll4WordsReverse                                          ??
Unroll4WordsSwap                                             Unroll4WordsSwap                                             ??
Unroll4WordsSwapFirst                                        Unroll4WordsSwapFirst                                        ??
Unroll4WordsSwapSwapFirst                                    Unroll4WordsSwapSwapFirst                                    ??
UnrollALabV2_8                                               UnrollALabV2_8                                               ??
UnrollAnyWords                                               UnrollAnyWords                                               ??
UnrollChunkyBytes                                            UnrollChunkyBytes                                            ??
UnrollDouble1Chan                                            UnrollDouble1Chan                                            ??
UnrollDoubleTo16                                             UnrollDoubleTo16                                             ??
UnrollDoublesToFloat                                         UnrollDoublesToFloat                                         ??
UnrollFloatTo16                                              UnrollFloatTo16                                              ??
UnrollFloatsToFloat                                          UnrollFloatsToFloat                                          ??
UnrollHalfTo16                                               UnrollHalfTo16                                               ??
UnrollHalfToFloat                                            UnrollHalfToFloat                                            ??
UnrollLabDoubleTo16                                          UnrollLabDoubleTo16                                          ??
UnrollLabDoubleToFloat                                       UnrollLabDoubleToFloat                                       ??
UnrollLabFloatTo16                                           UnrollLabFloatTo16                                           ??
UnrollLabFloatToFloat                                        UnrollLabFloatToFloat                                        ??
UnrollLabV2_16                                               UnrollLabV2_16                                               ??
UnrollLabV2_8                                                UnrollLabV2_8                                                ??
UnrollPlanarBytes                                            UnrollPlanarBytes                                            ??
UnrollPlanarWords                                            UnrollPlanarWords                                            ??
UnrollXYZDoubleTo16                                          UnrollXYZDoubleTo16                                          ??
UnrollXYZDoubleToFloat                                       UnrollXYZDoubleToFloat                                       ??
UnrollXYZFloatTo16                                           UnrollXYZFloatTo16                                           ??
UnrollXYZFloatToFloat                                        UnrollXYZFloatToFloat                                        ??
WriteMPECurve                                                WriteMPECurve                                                ??
WriteSeqID                                                   WriteSeqID                                                   ??
WriteSetOfCurves                                             WriteSetOfCurves                                             ??
XFormSampler16                                               XFormSampler16                                               ??
_LUTeval16                                                   _LUTeval16                                                   ??
_LUTevalFloat                                                _LUTevalFloat                                                ??
_cms15Fixed16toDouble                                        (_cms)15Fixed16toDouble                                      color management system (little cms) ??
_cms8Fixed8toDouble                                          (_cms)8Fixed8toDouble                                        color management system (little cms) ??
_cmsAdaptationMatrix                                         (_cms)AdaptationMatrix                                       color management system (little cms) ??
_cmsAdjustEndianess16                                        (_cms)AdjustEndianess16                                      color management system (little cms) ??
_cmsAdjustEndianess32                                        (_cms)AdjustEndianess32                                      color management system (little cms) ??
_cmsAdjustEndianess64                                        (_cms)AdjustEndianess64                                      color management system (little cms) ??
_cmsAllocAdaptationStateChunk                                (_cms)AllocAdaptationStateChunk                              color management system (little cms) ??
_cmsAllocAlarmCodesChunk                                     (_cms)AllocAlarmCodesChunk                                   color management system (little cms) ??
_cmsAllocCurvesPluginChunk                                   (_cms)AllocCurvesPluginChunk                                 color management system (little cms) ??
_cmsAllocFormattersPluginChunk                               (_cms)AllocFormattersPluginChunk                             color management system (little cms) ??
_cmsAllocIntentsPluginChunk                                  (_cms)AllocIntentsPluginChunk                                color management system (little cms) ??
_cmsAllocInterpPluginChunk                                   (_cms)AllocInterpPluginChunk                                 color management system (little cms) ??
_cmsAllocLogErrorChunk                                       (_cms)AllocLogErrorChunk                                     color management system (little cms) ??
_cmsAllocMPETypePluginChunk                                  (_cms)AllocMPETypePluginChunk                                color management system (little cms) ??
_cmsAllocMemPluginChunk                                      (_cms)AllocMemPluginChunk                                    color management system (little cms) ??
_cmsAllocMutexPluginChunk                                    (_cms)AllocMutexPluginChunk                                  color management system (little cms) ??
_cmsAllocOptimizationPluginChunk                             (_cms)AllocOptimizationPluginChunk                           color management system (little cms) ??
_cmsAllocTagPluginChunk                                      (_cms)AllocTagPluginChunk                                    color management system (little cms) ??
_cmsAllocTagTypePluginChunk                                  (_cms)AllocTagTypePluginChunk                                color management system (little cms) ??
_cmsAllocTransformPluginChunk                                (_cms)AllocTransformPluginChunk                              color management system (little cms) ??
_cmsBuildKToneCurve                                          (_cms)BuildKToneCurve                                        color management system (little cms) ??
_cmsBuildRGB2XYZtransferMatrix                               (_cms)BuildRGB2XYZtransferMatrix                             color management system (little cms) ??
_cmsCalloc                                                   (_cms)Calloc                                                 color management system (little cms) ??
_cmsCallocDefaultFn                                          (_cms)CallocDefaultFn                                        color management system (little cms) ??
_cmsChain2Lab                                                (_cms)Chain2Lab                                              color management system (little cms) ??
_cmsCompileProfileSequence                                   (_cms)CompileProfileSequence                                 color management system (little cms) ??
_cmsComputeInterpParams                                      (_cms)ComputeInterpParams                                    color management system (little cms) ??
_cmsComputeInterpParamsEx                                    (_cms)ComputeInterpParamsEx                                  color management system (little cms) ??
_cmsContextGetClientChunk                                    (_cms)ContextGetClientChunk                                  color management system (little cms) ??
_cmsCreateGamutCheckPipeline                                 (_cms)CreateGamutCheckPipeline                               color management system (little cms) ??
_cmsCreateMutex                                              (_cms)CreateMutex                                            color management system (little cms) ??
_cmsCreateSubAlloc                                           (_cms)CreateSubAlloc                                         color management system (little cms) ??
_cmsDecodeDateTimeNumber                                     (_cms)DecodeDateTimeNumber                                   color management system (little cms) ??
_cmsDefaultICCintents                                        (_cms)DefaultICCintents                                      color management system (little cms) ??
_cmsDestroyMutex                                             (_cms)DestroyMutex                                           color management system (little cms) ??
_cmsDoubleTo15Fixed16                                        (_cms)DoubleTo15Fixed16                                      color management system (little cms) ??
_cmsDoubleTo8Fixed8                                          (_cms)DoubleTo8Fixed8                                        color management system (little cms) ??
_cmsDupDefaultFn                                             (_cms)DupDefaultFn                                           color management system (little cms) ??
_cmsDupMem                                                   (_cms)DupMem                                                 color management system (little cms) ??
_cmsEncodeDateTimeNumber                                     (_cms)EncodeDateTimeNumber                                   color management system (little cms) ??
_cmsEndPointsBySpace                                         (_cms)EndPointsBySpace                                       color management system (little cms) ??
_cmsFloat2Half                                               (_cms)Float2Half                                             color management system (little cms) ??
_cmsFormatterIs8bit                                          (_cms)FormatterIs8bit                                        color management system (little cms) ??
_cmsFormatterIsFloat                                         (_cms)FormatterIsFloat                                       color management system (little cms) ??
_cmsFree                                                     (_cms)Free                                                   color management system (little cms) ??
_cmsFreeDefaultFn                                            (_cms)FreeDefaultFn                                          color management system (little cms) ??
_cmsFreeInterpParams                                         (_cms)FreeInterpParams                                       color management system (little cms) ??
_cmsGetContext                                               (_cms)GetContext                                             color management system (little cms) ??
_cmsGetFormatter                                             (_cms)GetFormatter                                           color management system (little cms) ??
_cmsGetTagDescriptor                                         (_cms)GetTagDescriptor                                       color management system (little cms) ??
_cmsGetTagTrueType                                           (_cms)GetTagTrueType                                         color management system (little cms) ??
_cmsGetTagTypeHandler                                        (_cms)GetTagTypeHandler                                      color management system (little cms) ??
_cmsGetTransformFormatters16                                 (_cms)GetTransformFormatters16                               color management system (little cms) ??
_cmsGetTransformFormattersFloat                              (_cms)GetTransformFormattersFloat                            color management system (little cms) ??
_cmsGetTransformUserData                                     (_cms)GetTransformUserData                                   color management system (little cms) ??
_cmsHalf2Float                                               (_cms)Half2Float                                             color management system (little cms) ??
_cmsHandleExtraChannels                                      (_cms)HandleExtraChannels                                    color management system (little cms) ??
_cmsICCcolorSpace                                            (_cms)ICCcolorSpace                                          color management system (little cms) ??
_cmsIOPrintf                                                 (_cms)IOPrintf                                               color management system (little cms) ??
_cmsInstallAllocFunctions                                    (_cms)InstallAllocFunctions                                  color management system (little cms) ??
_cmsLCMScolorSpace                                           (_cms)LCMScolorSpace                                         color management system (little cms) ??
_cmsLinkProfiles                                             (_cms)LinkProfiles                                           color management system (little cms) ??
_cmsLockMutex                                                (_cms)LockMutex                                              color management system (little cms) ??
_cmsMAT3eval                                                 (_cms)MAT3eval                                               color management system (little cms) ??
_cmsMAT3identity                                             (_cms)MAT3identity                                           color management system (little cms) ??
_cmsMAT3inverse                                              (_cms)MAT3inverse                                            color management system (little cms) ??
_cmsMAT3isIdentity                                           (_cms)MAT3isIdentity                                         color management system (little cms) ??
_cmsMAT3per                                                  (_cms)MAT3per                                                color management system (little cms) ??
_cmsMAT3solve                                                (_cms)MAT3solve                                              color management system (little cms) ??
_cmsMalloc                                                   (_cms)Malloc                                                 color management system (little cms) ??
_cmsMallocDefaultFn                                          (_cms)MallocDefaultFn                                        color management system (little cms) ??
_cmsMallocZero                                               (_cms)MallocZero                                             color management system (little cms) ??
_cmsMallocZeroDefaultFn                                      (_cms)MallocZeroDefaultFn                                    color management system (little cms) ??
_cmsOptimizePipeline                                         (_cms)OptimizePipeline                                       color management system (little cms) ??
_cmsPipelineSetOptimizationParameters                        (_cms)PipelineSetOptimizationParameters                      color management system (little cms) ??
_cmsPluginMalloc                                             (_cms)PluginMalloc                                           color management system (little cms) ??
_cmsQuantizeVal                                              (_cms)QuantizeVal                                            color management system (little cms) ??
_cmsRead15Fixed16Number                                      (_cms)Read15Fixed16Number                                    color management system (little cms) ??
_cmsReadAlignment                                            (_cms)ReadAlignment                                          color management system (little cms) ??
_cmsReadCHAD                                                 (_cms)ReadCHAD                                               color management system (little cms) ??
_cmsReadDevicelinkLUT                                        (_cms)ReadDevicelinkLUT                                      color management system (little cms) ??
_cmsReadFloat32Number                                        (_cms)ReadFloat32Number                                      color management system (little cms) ??
_cmsReadHeader                                               (_cms)ReadHeader                                             color management system (little cms) ??
_cmsReadInputLUT                                             (_cms)ReadInputLUT                                           color management system (little cms) ??
_cmsReadMediaWhitePoint                                      (_cms)ReadMediaWhitePoint                                    color management system (little cms) ??
_cmsReadOutputLUT                                            (_cms)ReadOutputLUT                                          color management system (little cms) ??
_cmsReadProfileSequence                                      (_cms)ReadProfileSequence                                    color management system (little cms) ??
_cmsReadTypeBase                                             (_cms)ReadTypeBase                                           color management system (little cms) ??
_cmsReadUInt16Array                                          (_cms)ReadUInt16Array                                        color management system (little cms) ??
_cmsReadUInt16Number                                         (_cms)ReadUInt16Number                                       color management system (little cms) ??
_cmsReadUInt32Number                                         (_cms)ReadUInt32Number                                       color management system (little cms) ??
_cmsReadUInt64Number                                         (_cms)ReadUInt64Number                                       color management system (little cms) ??
_cmsReadUInt8Number                                          (_cms)ReadUInt8Number                                        color management system (little cms) ??
_cmsReadXYZNumber                                            (_cms)ReadXYZNumber                                          color management system (little cms) ??
_cmsRealloc                                                  (_cms)Realloc                                                color management system (little cms) ??
_cmsReallocDefaultFn                                         (_cms)ReallocDefaultFn                                       color management system (little cms) ??
_cmsReasonableGridpointsByColorspace                         (_cms)ReasonableGridpointsByColorspace                       color management system (little cms) ??
_cmsRegisterFormattersPlugin                                 (_cms)RegisterFormattersPlugin                               color management system (little cms) ??
_cmsRegisterInterpPlugin                                     (_cms)RegisterInterpPlugin                                   color management system (little cms) ??
_cmsRegisterMemHandlerPlugin                                 (_cms)RegisterMemHandlerPlugin                               color management system (little cms) ??
_cmsRegisterMultiProcessElementPlugin                        (_cms)RegisterMultiProcessElementPlugin                      color management system (little cms) ??
_cmsRegisterMutexPlugin                                      (_cms)RegisterMutexPlugin                                    color management system (little cms) ??
_cmsRegisterOptimizationPlugin                               (_cms)RegisterOptimizationPlugin                             color management system (little cms) ??
_cmsRegisterParametricCurvesPlugin                           (_cms)RegisterParametricCurvesPlugin                         color management system (little cms) ??
_cmsRegisterRenderingIntentPlugin                            (_cms)RegisterRenderingIntentPlugin                          color management system (little cms) ??
_cmsRegisterTagPlugin                                        (_cms)RegisterTagPlugin                                      color management system (little cms) ??
_cmsRegisterTagTypePlugin                                    (_cms)RegisterTagTypePlugin                                  color management system (little cms) ??
_cmsRegisterTransformPlugin                                  (_cms)RegisterTransformPlugin                                color management system (little cms) ??
_cmsSearchTag                                                (_cms)SearchTag                                              color management system (little cms) ??
_cmsSetInterpolationRoutine                                  (_cms)SetInterpolationRoutine                                color management system (little cms) ??
_cmsSetTransformUserData                                     (_cms)SetTransformUserData                                   color management system (little cms) ??
_cmsStageAllocIdentityCLut                                   (_cms)StageAllocIdentityCLut                                 color management system (little cms) ??
_cmsStageAllocIdentityCurves                                 (_cms)StageAllocIdentityCurves                               color management system (little cms) ??
_cmsStageAllocLab2XYZ                                        (_cms)StageAllocLab2XYZ                                      color management system (little cms) ??
_cmsStageAllocLabPrelin                                      (_cms)StageAllocLabPrelin                                    color management system (little cms) ??
_cmsStageAllocLabV2ToV4                                      (_cms)StageAllocLabV2ToV4                                    color management system (little cms) ??
_cmsStageAllocLabV2ToV4curves                                (_cms)StageAllocLabV2ToV4curves                              color management system (little cms) ??
_cmsStageAllocLabV4ToV2                                      (_cms)StageAllocLabV4ToV2                                    color management system (little cms) ??
_cmsStageAllocNamedColor                                     (_cms)StageAllocNamedColor                                   color management system (little cms) ??
_cmsStageAllocPlaceholder                                    (_cms)StageAllocPlaceholder                                  color management system (little cms) ??
_cmsStageAllocXYZ2Lab                                        (_cms)StageAllocXYZ2Lab                                      color management system (little cms) ??
_cmsStageClipNegatives                                       (_cms)StageClipNegatives                                     color management system (little cms) ??
_cmsStageGetPtrToCurveSet                                    (_cms)StageGetPtrToCurveSet                                  color management system (little cms) ??
_cmsStageNormalizeFromLabFloat                               (_cms)StageNormalizeFromLabFloat                             color management system (little cms) ??
_cmsStageNormalizeFromXyzFloat                               (_cms)StageNormalizeFromXyzFloat                             color management system (little cms) ??
_cmsStageNormalizeToLabFloat                                 (_cms)StageNormalizeToLabFloat                               color management system (little cms) ??
_cmsStageNormalizeToXyzFloat                                 (_cms)StageNormalizeToXyzFloat                               color management system (little cms) ??
_cmsSubAlloc                                                 (_cms)SubAlloc                                               color management system (little cms) ??
_cmsSubAllocDestroy                                          (_cms)SubAllocDestroy                                        color management system (little cms) ??
_cmsSubAllocDup                                              (_cms)SubAllocDup                                            color management system (little cms) ??
_cmsTagSignature2String                                      (_cms)TagSignature2String                                    color management system (little cms) ??
_cmsTransform2toTransformAdaptor                             (_cms)Transform2toTransformAdaptor                           color management system (little cms) ??
_cmsUnlockMutex                                              (_cms)UnlockMutex                                            color management system (little cms) ??
_cmsVEC3cross                                                (_cms)VEC3cross                                              color management system (little cms) ??
_cmsVEC3distance                                             (_cms)VEC3distance                                           color management system (little cms) ??
_cmsVEC3dot                                                  (_cms)VEC3dot                                                color management system (little cms) ??
_cmsVEC3init                                                 (_cms)VEC3init                                               color management system (little cms) ??
_cmsVEC3length                                               (_cms)VEC3length                                             color management system (little cms) ??
_cmsVEC3minus                                                (_cms)VEC3minus                                              color management system (little cms) ??
_cmsWrite15Fixed16Number                                     (_cms)Write15Fixed16Number                                   color management system (little cms) ??
_cmsWriteAlignment                                           (_cms)WriteAlignment                                         color management system (little cms) ??
_cmsWriteFloat32Number                                       (_cms)WriteFloat32Number                                     color management system (little cms) ??
_cmsWriteHeader                                              (_cms)WriteHeader                                            color management system (little cms) ??
_cmsWriteProfileSequence                                     (_cms)WriteProfileSequence                                   color management system (little cms) ??
_cmsWriteTypeBase                                            (_cms)WriteTypeBase                                          color management system (little cms) ??
_cmsWriteUInt16Array                                         (_cms)WriteUInt16Array                                       color management system (little cms) ??
_cmsWriteUInt16Number                                        (_cms)WriteUInt16Number                                      color management system (little cms) ??
_cmsWriteUInt32Number                                        (_cms)WriteUInt32Number                                      color management system (little cms) ??
_cmsWriteUInt64Number                                        (_cms)WriteUInt64Number                                      color management system (little cms) ??
_cmsWriteUInt8Number                                         (_cms)WriteUInt8Number                                       color management system (little cms) ??
_cmsWriteXYZNumber                                           (_cms)WriteXYZNumber                                         color management system (little cms) ??
bchswSampler                                                 bchswSampler                                                 ??
cmsAdaptToIlluminant                                         (cms)AdaptToIlluminant                                       color management system (little cms) ??
cmsAllocNamedColorList                                       (cms)AllocNamedColorList                                     color management system (little cms) ??
cmsAllocProfileSequenceDescription                           (cms)AllocProfileSequenceDescription                         color management system (little cms) ??
cmsAppendNamedColor                                          (cms)AppendNamedColor                                        color management system (little cms) ??
cmsBFDdeltaE                                                 (cms)BFDdeltaE                                               color management system (little cms) ??
cmsBuildGamma                                                (cms)BuildGamma                                              color management system (little cms) ??
cmsBuildParametricToneCurve                                  (cms)BuildParametricToneCurve                                color management system (little cms) ??
cmsBuildSegmentedToneCurve                                   (cms)BuildSegmentedToneCurve                                 color management system (little cms) ??
cmsBuildTabulatedToneCurve16                                 (cms)BuildTabulatedToneCurve16                               color management system (little cms) ??
cmsBuildTabulatedToneCurveFloat                              (cms)BuildTabulatedToneCurveFloat                            color management system (little cms) ??
cmsCIE2000DeltaE                                             (cms)CIE2000DeltaE                                           color management system (little cms) ??
cmsCIE94DeltaE                                               (cms)CIE94DeltaE                                             color management system (little cms) ??
cmsCMCdeltaE                                                 (cms)CMCdeltaE                                               color management system (little cms) ??
cmsChangeBuffersFormat                                       (cms)ChangeBuffersFormat                                     color management system (little cms) ??
cmsChannelsOf                                                (cms)ChannelsOf                                              color management system (little cms) ??
cmsCloseIOhandler                                            (cms)CloseIOhandler                                          color management system (little cms) ??
cmsCloseProfile                                              (cms)CloseProfile                                            color management system (little cms) ??
cmsCreateBCHSWabstractProfile                                (cms)CreateBCHSWabstractProfile                              color management system (little cms) ??
cmsCreateBCHSWabstractProfileTHR                             (cms)CreateBCHSWabstractProfileTHR                           color management system (little cms) ??
cmsCreateContext                                             (cms)CreateContext                                           color management system (little cms) ??
cmsCreateExtendedTransform                                   (cms)CreateExtendedTransform                                 color management system (little cms) ??
cmsCreateGrayProfile                                         (cms)CreateGrayProfile                                       color management system (little cms) ??
cmsCreateGrayProfileTHR                                      (cms)CreateGrayProfileTHR                                    color management system (little cms) ??
cmsCreateInkLimitingDeviceLink                               (cms)CreateInkLimitingDeviceLink                             color management system (little cms) ??
cmsCreateInkLimitingDeviceLinkTHR                            (cms)CreateInkLimitingDeviceLinkTHR                          color management system (little cms) ??
cmsCreateLab2Profile                                         (cms)CreateLab2Profile                                       color management system (little cms) ??
cmsCreateLab2ProfileTHR                                      (cms)CreateLab2ProfileTHR                                    color management system (little cms) ??
cmsCreateLab4Profile                                         (cms)CreateLab4Profile                                       color management system (little cms) ??
cmsCreateLab4ProfileTHR                                      (cms)CreateLab4ProfileTHR                                    color management system (little cms) ??
cmsCreateLinearizationDeviceLink                             (cms)CreateLinearizationDeviceLink                           color management system (little cms) ??
cmsCreateLinearizationDeviceLinkTHR                          (cms)CreateLinearizationDeviceLinkTHR                        color management system (little cms) ??
cmsCreateMultiprofileTransform                               (cms)CreateMultiprofileTransform                             color management system (little cms) ??
cmsCreateMultiprofileTransformTHR                            (cms)CreateMultiprofileTransformTHR                          color management system (little cms) ??
cmsCreateNULLProfile                                         (cms)CreateNULLProfile                                       color management system (little cms) ??
cmsCreateNULLProfileTHR                                      (cms)CreateNULLProfileTHR                                    color management system (little cms) ??
cmsCreateProfilePlaceholder                                  (cms)CreateProfilePlaceholder                                color management system (little cms) ??
cmsCreateProofingTransform                                   (cms)CreateProofingTransform                                 color management system (little cms) ??
cmsCreateProofingTransformTHR                                (cms)CreateProofingTransformTHR                              color management system (little cms) ??
cmsCreateRGBProfile                                          (cms)CreateRGBProfile                                        color management system (little cms) ??
cmsCreateRGBProfileTHR                                       (cms)CreateRGBProfileTHR                                     color management system (little cms) ??
cmsCreateTransform                                           (cms)CreateTransform                                         color management system (little cms) ??
cmsCreateTransformTHR                                        (cms)CreateTransformTHR                                      color management system (little cms) ??
cmsCreateXYZProfile                                          (cms)CreateXYZProfile                                        color management system (little cms) ??
cmsCreateXYZProfileTHR                                       (cms)CreateXYZProfileTHR                                     color management system (little cms) ??
cmsCreate_sRGBProfile                                        (cms)Create_sRGBProfile                                      color management system (little cms) ??
cmsCreate_sRGBProfileTHR                                     (cms)Create_sRGBProfileTHR                                   color management system (little cms) ??
cmsD50_XYZ                                                   (cms)D50_XYZ                                                 color management system (little cms) ??
cmsD50_xyY                                                   (cms)D50_xyY                                                 color management system (little cms) ??
cmsDeleteContext                                             (cms)DeleteContext                                           color management system (little cms) ??
cmsDeleteTransform                                           (cms)DeleteTransform                                         color management system (little cms) ??
cmsDeltaE                                                    (cms)DeltaE                                                  color management system (little cms) ??
cmsDesaturateLab                                             (cms)DesaturateLab                                           color management system (little cms) ??
cmsDetectBlackPoint                                          (cms)DetectBlackPoint                                        color management system (little cms) ??
cmsDetectDestinationBlackPoint                               (cms)DetectDestinationBlackPoint                             color management system (little cms) ??
cmsDetectTAC                                                 (cms)DetectTAC                                               color management system (little cms) ??
cmsDictAddEntry                                              (cms)DictAddEntry                                            color management system (little cms) ??
cmsDictAlloc                                                 (cms)DictAlloc                                               color management system (little cms) ??
cmsDictDup                                                   (cms)DictDup                                                 color management system (little cms) ??
cmsDictFree                                                  (cms)DictFree                                                color management system (little cms) ??
cmsDictGetEntryList                                          (cms)DictGetEntryList                                        color management system (little cms) ??
cmsDictNextEntry                                             (cms)DictNextEntry                                           color management system (little cms) ??
cmsDoTransform                                               (cms)DoTransform                                             color management system (little cms) ??
cmsDoTransformLineStride                                     (cms)DoTransformLineStride                                   color management system (little cms) ??
cmsDoTransformStride                                         (cms)DoTransformStride                                       color management system (little cms) ??
cmsDupContext                                                (cms)DupContext                                              color management system (little cms) ??
cmsDupNamedColorList                                         (cms)DupNamedColorList                                       color management system (little cms) ??
cmsDupProfileSequenceDescription                             (cms)DupProfileSequenceDescription                           color management system (little cms) ??
cmsDupToneCurve                                              (cms)DupToneCurve                                            color management system (little cms) ??
cmsEstimateGamma                                             (cms)EstimateGamma                                           color management system (little cms) ??
cmsEvalToneCurve16                                           (cms)EvalToneCurve16                                         color management system (little cms) ??
cmsEvalToneCurveFloat                                        (cms)EvalToneCurveFloat                                      color management system (little cms) ??
cmsFloat2LabEncoded                                          (cms)Float2LabEncoded                                        color management system (little cms) ??
cmsFloat2LabEncodedV2                                        (cms)Float2LabEncodedV2                                      color management system (little cms) ??
cmsFloat2XYZEncoded                                          (cms)Float2XYZEncoded                                        color management system (little cms) ??
cmsFormatterForColorspaceOfProfile                           (cms)FormatterForColorspaceOfProfile                         color management system (little cms) ??
cmsFormatterForPCSOfProfile                                  (cms)FormatterForPCSOfProfile                                color management system (little cms) ??
cmsFreeNamedColorList                                        (cms)FreeNamedColorList                                      color management system (little cms) ??
cmsFreeProfileSequenceDescription                            (cms)FreeProfileSequenceDescription                          color management system (little cms) ??
cmsFreeToneCurve                                             (cms)FreeToneCurve                                           color management system (little cms) ??
cmsFreeToneCurveTriple                                       (cms)FreeToneCurveTriple                                     color management system (little cms) ??
cmsGetAlarmCodes                                             (cms)GetAlarmCodes                                           color management system (little cms) ??
cmsGetAlarmCodesTHR                                          (cms)GetAlarmCodesTHR                                        color management system (little cms) ??
cmsGetColorSpace                                             (cms)GetColorSpace                                           color management system (little cms) ??
cmsGetContextUserData                                        (cms)GetContextUserData                                      color management system (little cms) ??
cmsGetDeviceClass                                            (cms)GetDeviceClass                                          color management system (little cms) ??
cmsGetEncodedCMMversion                                      (cms)GetEncodedCMMversion                                    color management system (little cms) ??
cmsGetEncodedICCversion                                      (cms)GetEncodedICCversion                                    color management system (little cms) ??
cmsGetHeaderAttributes                                       (cms)GetHeaderAttributes                                     color management system (little cms) ??
cmsGetHeaderCreationDateTime                                 (cms)GetHeaderCreationDateTime                               color management system (little cms) ??
cmsGetHeaderCreator                                          (cms)GetHeaderCreator                                        color management system (little cms) ??
cmsGetHeaderFlags                                            (cms)GetHeaderFlags                                          color management system (little cms) ??
cmsGetHeaderManufacturer                                     (cms)GetHeaderManufacturer                                   color management system (little cms) ??
cmsGetHeaderModel                                            (cms)GetHeaderModel                                          color management system (little cms) ??
cmsGetHeaderProfileID                                        (cms)GetHeaderProfileID                                      color management system (little cms) ??
cmsGetHeaderRenderingIntent                                  (cms)GetHeaderRenderingIntent                                color management system (little cms) ??
cmsGetNamedColorList                                         (cms)GetNamedColorList                                       color management system (little cms) ??
cmsGetPCS                                                    (cms)GetPCS                                                  color management system (little cms) ??
cmsGetPipelineContextID                                      (cms)GetPipelineContextID                                    color management system (little cms) ??
cmsGetProfileContextID                                       (cms)GetProfileContextID                                     color management system (little cms) ??
cmsGetProfileIOhandler                                       (cms)GetProfileIOhandler                                     color management system (little cms) ??
cmsGetProfileInfo                                            (cms)GetProfileInfo                                          color management system (little cms) ??
cmsGetProfileInfoASCII                                       (cms)GetProfileInfoASCII                                     color management system (little cms) ??
cmsGetProfileVersion                                         (cms)GetProfileVersion                                       color management system (little cms) ??
cmsGetSupportedIntents                                       (cms)GetSupportedIntents                                     color management system (little cms) ??
cmsGetSupportedIntentsTHR                                    (cms)GetSupportedIntentsTHR                                  color management system (little cms) ??
cmsGetTagCount                                               (cms)GetTagCount                                             color management system (little cms) ??
cmsGetTagSignature                                           (cms)GetTagSignature                                         color management system (little cms) ??
cmsGetToneCurveEstimatedTable                                (cms)GetToneCurveEstimatedTable                              color management system (little cms) ??
cmsGetToneCurveEstimatedTableEntries                         (cms)GetToneCurveEstimatedTableEntries                       color management system (little cms) ??
cmsGetToneCurveParametricType                                (cms)GetToneCurveParametricType                              color management system (little cms) ??
cmsGetTransformContextID                                     (cms)GetTransformContextID                                   color management system (little cms) ??
cmsGetTransformInputFormat                                   (cms)GetTransformInputFormat                                 color management system (little cms) ??
cmsGetTransformOutputFormat                                  (cms)GetTransformOutputFormat                                color management system (little cms) ??
cmsIsCLUT                                                    (cms)IsCLUT                                                  color management system (little cms) ??
cmsIsIntentSupported                                         (cms)IsIntentSupported                                       color management system (little cms) ??
cmsIsMatrixShaper                                            (cms)IsMatrixShaper                                          color management system (little cms) ??
cmsIsTag                                                     (cms)IsTag                                                   color management system (little cms) ??
cmsIsToneCurveDescending                                     (cms)IsToneCurveDescending                                   color management system (little cms) ??
cmsIsToneCurveLinear                                         (cms)IsToneCurveLinear                                       color management system (little cms) ??
cmsIsToneCurveMonotonic                                      (cms)IsToneCurveMonotonic                                    color management system (little cms) ??
cmsIsToneCurveMultisegment                                   (cms)IsToneCurveMultisegment                                 color management system (little cms) ??
cmsJoinToneCurve                                             (cms)JoinToneCurve                                           color management system (little cms) ??
cmsLCh2Lab                                                   (cms)LCh2Lab                                                 color management system (little cms) ??
cmsLab2LCh                                                   (cms)Lab2LCh                                                 color management system (little cms) ??
cmsLab2XYZ                                                   (cms)Lab2XYZ                                                 color management system (little cms) ??
cmsLabEncoded2Float                                          (cms)LabEncoded2Float                                        color management system (little cms) ??
cmsLabEncoded2FloatV2                                        (cms)LabEncoded2FloatV2                                      color management system (little cms) ??
cmsLinkTag                                                   (cms)LinkTag                                                 color management system (little cms) ??
cmsMLUalloc                                                  (cms)MLUalloc                                                color management system (little cms) ??
cmsMLUdup                                                    (cms)MLUdup                                                  color management system (little cms) ??
cmsMLUfree                                                   (cms)MLUfree                                                 color management system (little cms) ??
cmsMLUgetASCII                                               (cms)MLUgetASCII                                             color management system (little cms) ??
cmsMLUgetTranslation                                         (cms)MLUgetTranslation                                       color management system (little cms) ??
cmsMLUgetWide                                                (cms)MLUgetWide                                              color management system (little cms) ??
cmsMLUsetASCII                                               (cms)MLUsetASCII                                             color management system (little cms) ??
cmsMLUsetWide                                                (cms)MLUsetWide                                              color management system (little cms) ??
cmsMLUtranslationsCodes                                      (cms)MLUtranslationsCodes                                    color management system (little cms) ??
cmsMLUtranslationsCount                                      (cms)MLUtranslationsCount                                    color management system (little cms) ??
cmsNamedColorCount                                           (cms)NamedColorCount                                         color management system (little cms) ??
cmsNamedColorIndex                                           (cms)NamedColorIndex                                         color management system (little cms) ??
cmsNamedColorInfo                                            (cms)NamedColorInfo                                          color management system (little cms) ??
cmsOpenIOhandlerFromFile                                     (cms)OpenIOhandlerFromFile                                   color management system (little cms) ??
cmsOpenIOhandlerFromMem                                      (cms)OpenIOhandlerFromMem                                    color management system (little cms) ??
cmsOpenIOhandlerFromNULL                                     (cms)OpenIOhandlerFromNULL                                   color management system (little cms) ??
cmsOpenIOhandlerFromStream                                   (cms)OpenIOhandlerFromStream                                 color management system (little cms) ??
cmsOpenProfileFromFile                                       (cms)OpenProfileFromFile                                     color management system (little cms) ??
cmsOpenProfileFromFileTHR                                    (cms)OpenProfileFromFileTHR                                  color management system (little cms) ??
cmsOpenProfileFromIOhandler2THR                              (cms)OpenProfileFromIOhandler2THR                            color management system (little cms) ??
cmsOpenProfileFromIOhandlerTHR                               (cms)OpenProfileFromIOhandlerTHR                             color management system (little cms) ??
cmsOpenProfileFromMem                                        (cms)OpenProfileFromMem                                      color management system (little cms) ??
cmsOpenProfileFromMemTHR                                     (cms)OpenProfileFromMemTHR                                   color management system (little cms) ??
cmsOpenProfileFromStream                                     (cms)OpenProfileFromStream                                   color management system (little cms) ??
cmsOpenProfileFromStreamTHR                                  (cms)OpenProfileFromStreamTHR                                color management system (little cms) ??
cmsPipelineAlloc                                             (cms)PipelineAlloc                                           color management system (little cms) ??
cmsPipelineCat                                               (cms)PipelineCat                                             color management system (little cms) ??
cmsPipelineCheckAndRetreiveStages                            (cms)PipelineCheckAndRetreiveStages                          color management system (little cms) ??
cmsPipelineDup                                               (cms)PipelineDup                                             color management system (little cms) ??
cmsPipelineEval16                                            (cms)PipelineEval16                                          color management system (little cms) ??
cmsPipelineEvalFloat                                         (cms)PipelineEvalFloat                                       color management system (little cms) ??
cmsPipelineEvalReverseFloat                                  (cms)PipelineEvalReverseFloat                                color management system (little cms) ??
cmsPipelineFree                                              (cms)PipelineFree                                            color management system (little cms) ??
cmsPipelineGetPtrToFirstStage                                (cms)PipelineGetPtrToFirstStage                              color management system (little cms) ??
cmsPipelineGetPtrToLastStage                                 (cms)PipelineGetPtrToLastStage                               color management system (little cms) ??
cmsPipelineInputChannels                                     (cms)PipelineInputChannels                                   color management system (little cms) ??
cmsPipelineInsertStage                                       (cms)PipelineInsertStage                                     color management system (little cms) ??
cmsPipelineOutputChannels                                    (cms)PipelineOutputChannels                                  color management system (little cms) ??
cmsPipelineSetSaveAs8bitsFlag                                (cms)PipelineSetSaveAs8bitsFlag                              color management system (little cms) ??
cmsPipelineStageCount                                        (cms)PipelineStageCount                                      color management system (little cms) ??
cmsPipelineUnlinkStage                                       (cms)PipelineUnlinkStage                                     color management system (little cms) ??
cmsPlugin                                                    (cms)Plugin                                                  color management system (little cms) ??
cmsPluginTHR                                                 (cms)PluginTHR                                               color management system (little cms) ??
cmsReadRawTag                                                (cms)ReadRawTag                                              color management system (little cms) ??
cmsReadTag                                                   (cms)ReadTag                                                 color management system (little cms) ??
cmsReverseToneCurve                                          (cms)ReverseToneCurve                                        color management system (little cms) ??
cmsReverseToneCurveEx                                        (cms)ReverseToneCurveEx                                      color management system (little cms) ??
cmsSaveProfileToFile                                         (cms)SaveProfileToFile                                       color management system (little cms) ??
cmsSaveProfileToIOhandler                                    (cms)SaveProfileToIOhandler                                  color management system (little cms) ??
cmsSaveProfileToMem                                          (cms)SaveProfileToMem                                        color management system (little cms) ??
cmsSaveProfileToStream                                       (cms)SaveProfileToStream                                     color management system (little cms) ??
cmsSetAdaptationState                                        (cms)SetAdaptationState                                      color management system (little cms) ??
cmsSetAdaptationStateTHR                                     (cms)SetAdaptationStateTHR                                   color management system (little cms) ??
cmsSetAlarmCodes                                             (cms)SetAlarmCodes                                           color management system (little cms) ??
cmsSetAlarmCodesTHR                                          (cms)SetAlarmCodesTHR                                        color management system (little cms) ??
cmsSetColorSpace                                             (cms)SetColorSpace                                           color management system (little cms) ??
cmsSetDeviceClass                                            (cms)SetDeviceClass                                          color management system (little cms) ??
cmsSetEncodedICCversion                                      (cms)SetEncodedICCversion                                    color management system (little cms) ??
cmsSetHeaderAttributes                                       (cms)SetHeaderAttributes                                     color management system (little cms) ??
cmsSetHeaderFlags                                            (cms)SetHeaderFlags                                          color management system (little cms) ??
cmsSetHeaderManufacturer                                     (cms)SetHeaderManufacturer                                   color management system (little cms) ??
cmsSetHeaderModel                                            (cms)SetHeaderModel                                          color management system (little cms) ??
cmsSetHeaderProfileID                                        (cms)SetHeaderProfileID                                      color management system (little cms) ??
cmsSetHeaderRenderingIntent                                  (cms)SetHeaderRenderingIntent                                color management system (little cms) ??
cmsSetLogErrorHandler                                        (cms)SetLogErrorHandler                                      color management system (little cms) ??
cmsSetLogErrorHandlerTHR                                     (cms)SetLogErrorHandlerTHR                                   color management system (little cms) ??
cmsSetPCS                                                    (cms)SetPCS                                                  color management system (little cms) ??
cmsSetProfileVersion                                         (cms)SetProfileVersion                                       color management system (little cms) ??
cmsSignalError                                               (cms)SignalError                                             color management system (little cms) ??
cmsSliceSpace16                                              (cms)SliceSpace16                                            color management system (little cms) ??
cmsSliceSpaceFloat                                           (cms)SliceSpaceFloat                                         color management system (little cms) ??
cmsSmoothToneCurve                                           (cms)SmoothToneCurve                                         color management system (little cms) ??
cmsStageAllocCLut16bit                                       (cms)StageAllocCLut16bit                                     color management system (little cms) ??
cmsStageAllocCLut16bitGranular                               (cms)StageAllocCLut16bitGranular                             color management system (little cms) ??
cmsStageAllocCLutFloat                                       (cms)StageAllocCLutFloat                                     color management system (little cms) ??
cmsStageAllocCLutFloatGranular                               (cms)StageAllocCLutFloatGranular                             color management system (little cms) ??
cmsStageAllocIdentity                                        (cms)StageAllocIdentity                                      color management system (little cms) ??
cmsStageAllocMatrix                                          (cms)StageAllocMatrix                                        color management system (little cms) ??
cmsStageAllocToneCurves                                      (cms)StageAllocToneCurves                                    color management system (little cms) ??
cmsStageData                                                 (cms)StageData                                               color management system (little cms) ??
cmsStageDup                                                  (cms)StageDup                                                color management system (little cms) ??
cmsStageFree                                                 (cms)StageFree                                               color management system (little cms) ??
cmsStageInputChannels                                        (cms)StageInputChannels                                      color management system (little cms) ??
cmsStageNext                                                 (cms)StageNext                                               color management system (little cms) ??
cmsStageOutputChannels                                       (cms)StageOutputChannels                                     color management system (little cms) ??
cmsStageSampleCLut16bit                                      (cms)StageSampleCLut16bit                                    color management system (little cms) ??
cmsStageSampleCLutFloat                                      (cms)StageSampleCLutFloat                                    color management system (little cms) ??
cmsStageType                                                 (cms)StageType                                               color management system (little cms) ??
cmsTagLinkedTo                                               (cms)TagLinkedTo                                             color management system (little cms) ??
cmsTempFromWhitePoint                                        (cms)TempFromWhitePoint                                      color management system (little cms) ??
cmsTransform2DeviceLink                                      (cms)Transform2DeviceLink                                    color management system (little cms) ??
cmsUnregisterPlugins                                         (cms)UnregisterPlugins                                       color management system (little cms) ??
cmsUnregisterPluginsTHR                                      (cms)UnregisterPluginsTHR                                    color management system (little cms) ??
cmsWhitePointFromTemp                                        (cms)WhitePointFromTemp                                      color management system (little cms) ??
cmsWriteRawTag                                               (cms)WriteRawTag                                             color management system (little cms) ??
cmsWriteTag                                                  (cms)WriteTag                                                color management system (little cms) ??
cmsXYZ2Lab                                                   (cms)XYZ2Lab                                                 color management system (little cms) ??
cmsXYZ2xyY                                                   (cms)XYZ2xyY                                                 color management system (little cms) ??
cmsXYZEncoded2Float                                          (cms)XYZEncoded2Float                                        color management system (little cms) ??
cmsfilelength                                                (cms)filelength                                              color management system (little cms) ??
cmsstrcasecmp                                                (cms)strcasecmp                                              color management system (little cms) ??
cmsxyY2XYZ                                                   (cms)xyY2XYZ                                                 color management system (little cms) ??
copy16                                                       copy16                                                       ??
copy32                                                       copy32                                                       ??
copy64                                                       copy64                                                       ??
copy8                                                        copy8                                                        ??
defMtxCreate                                                 defMtxCreate                                                 ??
defMtxDestroy                                                defMtxDestroy                                                ??
defMtxLock                                                   defMtxLock                                                   ??
defMtxUnlock                                                 defMtxUnlock                                                 ??
errorHandler                                                 (err)orHandler                                               error recovery ??
free_ex_data_arg                                             free_ex_data_arg                                             ??
from16to8                                                    from16to8                                                    ??
from16toDBL                                                  from16toDBL                                                  ??
from16toFLT                                                  from16toFLT                                                  ??
from16toHLF                                                  from16toHLF                                                  ??
from8to16                                                    from8to16                                                    ??
from8toDBL                                                   from8toDBL                                                   ??
from8toFLT                                                   from8toFLT                                                   ??
from8toHLF                                                   from8toHLF                                                   ??
fromDBLto16                                                  fromDBLto16                                                  ??
fromDBLto8                                                   fromDBLto8                                                   ??
fromDBLtoFLT                                                 fromDBLtoFLT                                                 ??
fromDBLtoHLF                                                 fromDBLtoHLF                                                 ??
fromFLTto16                                                  fromFLTto16                                                  ??
fromFLTto8                                                   fromFLTto8                                                   ??
fromFLTtoDBL                                                 fromFLTtoDBL                                                 ??
fromFLTtoHLF                                                 fromFLTtoHLF                                                 ??
fromHLFto16                                                  fromHLFto16                                                  ??
fromHLFto8                                                   fromHLFto8                                                   ??
fromHLFtoDBL                                                 fromHLFtoDBL                                                 ??
fromHLFtoFLT                                                 fromHLFtoFLT                                                 ??
getILData                                                    getILData                                                    ??
kcfis_oss_wait_per_ctx                                       (kcfis)_oss_wait_per_ctx                                     kernel cache file management intelligent storage ??
kcfis_set_appliance_ctx                                      (kcfis)_set_appliance_ctx                                    kernel cache file management intelligent storage ??
kcfis_tablespace_is_smart_scannable                          (kcfis)_tablespace_is_smart_scannable                        kernel cache file management intelligent storage ??
kdmoCheckGbyAgg                                              (kdmo)CheckGbyAgg                                            kernel data in-memory data layer optimizer ??
kdmoCheckGbyOpn                                              (kdmo)CheckGbyOpn                                            kernel data in-memory data layer optimizer ??
kdmoColsUlvlIsDict                                           (kdmo)ColsUlvlIsDict                                         kernel data in-memory data layer optimizer ??
kdmoGByPushdownCardEst                                       (kdmo)GByPushdownCardEst                                     kernel data in-memory data layer optimizer ??
kdmoGByPushdownValid                                         (kdmo)GByPushdownValid                                       kernel data in-memory data layer optimizer ??
kdmoGenericPcodeCheck                                        (kdmo)GenericPcodeCheck                                      kernel data in-memory data layer optimizer ??
kdmoStorageColCheck                                          (kdmo)StorageColCheck                                        kernel data in-memory data layer optimizer ??
kdmoStorageConstantCheck                                     (kdmo)StorageConstantCheck                                   kernel data in-memory data layer optimizer ??
kdmoStorageOpnCheckAux                                       (kdmo)StorageOpnCheckAux                                     kernel data in-memory data layer optimizer ??
kdmoTabUlvlIsDict                                            (kdmo)TabUlvlIsDict                                          kernel data in-memory data layer optimizer ??
kdmoValidGbyPushdownPredCB                                   (kdmo)ValidGbyPushdownPredCB                                 kernel data in-memory data layer optimizer ??
kdzfCheckDbaMatch                                            (kdz)fCheckDbaMatch                                          kernel data archive compression ??
kdzsDumpKaf                                                  (kdzs)DumpKaf                                                kernel data archive compression decompression ??
kfgbCheckInterrupt                                           (kfgb)CheckInterrupt                                         kernel automatic storage management diskgroups background ??
kgh_check_simple_free_canary                                 (kgh)_check_simple_free_canary                               kernel generic heap manager ??
kgh_get_max_size_on_list                                     (kgh)_get_max_size_on_list                                   kernel generic heap manager ??
kgh_get_visit_limit                                          (kgh)_get_visit_limit                                        kernel generic heap manager ??
kgh_set_fl_effort                                            (kgh)_set_fl_effort                                          kernel generic heap manager ??
kghsrch_best_fit                                             (kghsrch)_best_fit                                           kernel generic heap manager search freelists for a memory chunk ??
kghunalo                                                     (kgh)unalo                                                   kernel generic heap manager ??
kgzm_encode_identify_ctype                                   (kg)zm_encode_identify_ctype                                 kernel generic ??
kgzm_encode_version_with_reid                                (kg)zm_encode_version_with_reid                              kernel generic ??
kjbmisgdbabast                                               (kjb)misgdbabast                                             kernel lock management global cache service ??
kjbmrejectbast                                               (kjb)mrejectbast                                             kernel lock management global cache service ??
kkesrcCard                                                   (kke)srcCard                                                 kernel compile cost engine ??
kkqjpdrace                                                   (kkqjpd)race                                                 kernel compile query  join analysis predicate push down ??
kkqoreIsUnsupported                                          (kkqore)IsUnsupported                                        kernel compile query  or-expansion ??
klxGetBuffer                                                 (kl)xGetBuffer                                               kernel loader ??
klxGetLength                                                 (kl)xGetLength                                               kernel loader ??
koklc_screate                                                (kokl)c_screate                                              kernel objects kernel side lob access ??
kolaCreateFromExternalSrc                                    (kola)CreateFromExternalSrc                                  kernel objects lob ??
kolaetAssign                                                 (kola)etAssign                                               kernel objects lob ??
kolaetChkSize                                                (kola)etChkSize                                              kernel objects lob ??
kolaetCreate                                                 (kola)etCreate                                               kernel objects lob ??
kolaetCreateCtx                                              (kola)etCreateCtx                                            kernel objects lob ??
kolaetCreateExternalInline                                   (kola)etCreateExternalInline                                 kernel objects lob ??
kolaetCreateFromExternalInline                               (kola)etCreateFromExternalInline                             kernel objects lob ??
kolaetDmpData                                                (kola)etDmpData                                              kernel objects lob ??
kolaetFree                                                   (kola)etFree                                                 kernel objects lob ??
kolaetGetLength                                              (kola)etGetLength                                            kernel objects lob ??
kolaetGetRawData                                             (kola)etGetRawData                                           kernel objects lob ??
kolaetIsExternalInlineLob                                    (kola)etIsExternalInlineLob                                  kernel objects lob ??
kolaetRead                                                   (kola)etRead                                                 kernel objects lob ??
kolaetWrite                                                  (kola)etWrite                                                kernel objects lob ??
kolamalSage                                                  (kola)malSage                                                kernel objects lob ??
kolamfrSage                                                  (kola)mfrSage                                                kernel objects lob ??
kolasugc                                                     (kola)sugc                                                   kernel objects lob ??
kolasugi                                                     (kola)sugi                                                   kernel objects lob ??
kolrsdesht                                                   (kolr)sdesht                                                 kernel objects lob refcount ??
kolrsugi                                                     (kolr)sugi                                                   kernel objects lob refcount ??
kqltprom                                                     (kql)tprom                                                   kernel query library cache ??
krbCreateGuidStr                                             (krb)CreateGuidStr                                           kernel redo backup/restore ??
krbrCheckLogical                                             (krbr)CheckLogical                                           kernel redo backup/restore restore and recovery ??
krvxdsr                                                      (krvx)dsr                                                    kernel redo recovery extract ??
ksmsq_memlock_limit_alert_error                              (ksmsq)_memlock_limit_alert_error                            kernel service (VOS) memory sga heap message queue services ??
ksmsq_transport_require_memlock                              (ksmsq)_transport_require_memlock                            kernel service (VOS) memory sga heap message queue services ??
ksz_oss_get_ip_info                                          (ksz)_oss_get_ip_info                                        kernel service (VOS) oracle storage server (OSS) server layer ??
ksz_skgxp_check_default_query                                (ksz)_skgxp_check_default_query                              kernel service (VOS) oracle storage server (OSS) server layer ??
ktmaLgwrHeartbeat                                            (ktma)LgwrHeartbeat                                          kernel transaction transaction monitor (smon) IM transaction ADG ??
ktsttsn_to_tsnpdb                                            (ktst)tsn_to_tsnpdb                                          kernel transaction segment management sort management ??
ktsttsnpdb_to_pdb                                            (ktst)tsnpdb_to_pdb                                          kernel transaction segment management sort management ??
ktsttsnpdb_to_tsn                                            (ktst)tsnpdb_to_tsn                                          kernel transaction segment management sort management ??
kubsjniSkip                                                  (ku)bsjniSkip                                                kernel utility ??
kubsxiMapCluUser                                             (ku)bsxiMapCluUser                                           kernel utility ??
kubsxiSetCpx                                                 (ku)bsxiSetCpx                                               kernel utility ??
kxfrGetNextSplit                                             (kxfr)GetNextSplit                                           kernel execution parallel query granules ??
kzvdvechk_ownerid_1                                          (kzvd)vechk_ownerid_1                                        kernel security data vault ??
oracle_storidx_recompute_ridx_summary                        oracle_storidx_recompute_ridx_summary                        ??
prsUnusable                                                  (prs)Unusable                                                parse ??
prscViewColAlias                                             (prs)cViewColAlias                                           parse ??
prsc_edition                                                 (prs)c_edition                                               parse ??
prscal                                                       (prs)cal                                                     parse ??
prscap                                                       (prs)cap                                                     parse ??
prscbl                                                       (prs)cbl                                                     parse ??
prsccc                                                       (prs)ccc                                                     parse ??
prscct                                                       (prs)cct                                                     parse ??
prscdg                                                       (prs)cdg                                                     parse ??
prscdr                                                       (prs)cdr                                                     parse ??
prscet_parse_extab                                           (prs)cet_parse_extab                                         parse ??
prscipc_index_placement_clause                               (prs)cipc_index_placement_clause                             parse ??
prscisfbdrp                                                  (prs)cisfbdrp                                                parse ??
prscisfbtbl                                                  (prs)cisfbtbl                                                parse ??
prscnd                                                       (prs)cnd                                                     parse ??
prscpfil                                                     (prs)cpfil                                                   parse ??
prscrefv                                                     (prs)crefv                                                   parse ??
prscrely                                                     (prs)crely                                                   parse ??
prscspfi                                                     (prs)cspfi                                                   parse ??
prscvac                                                      (prs)cvac                                                    parse ??
prsdfl                                                       (prs)dfl                                                     parse ??
prsfbdrp                                                     (prs)fbdrp                                                   parse ??
prsfbdrptab                                                  (prs)fbdrptab                                                parse ??
prsfla                                                       (prs)fla                                                     parse ??
prshve_high_val_expr                                         (prs)hve_high_val_expr                                       parse ??
prsicsl                                                      (prs)icsl                                                    parse ??
prsident                                                     (prs)ident                                                   parse ??
prsini_iot_init                                              (prs)ini_iot_init                                            parse ??
prsref                                                       (prs)ref                                                     parse ??
prssoasn                                                     (prs)soasn                                                   parse ??
prssptoa                                                     (prs)sptoa                                                   parse ??
qcpiSetFnInOpn                                               (qcpi)SetFnInOpn                                             query compile parse interim ??
qcpiSetFnInSel                                               (qcpi)SetFnInSel                                             query compile parse interim ??
qertbFilter_qertbs                                           (qertb)Filter_qertbs                                         query execute rowsource table access ??
qertbInitXTFilter                                            (qertb)InitXTFilter                                          query execute rowsource table access ??
qertbQueryHXTFilter                                          (qertb)QueryHXTFilter                                        query execute rowsource table access ??
qertbQueryLXTFilter                                          (qertb)QueryLXTFilter                                        query execute rowsource table access ??
qertbUpdateXTFilter                                          (qertb)UpdateXTFilter                                        query execute rowsource table access ??
qertbXTCbk                                                   (qertb)XTCbk                                                 query execute rowsource table access ??
qesSageIsHDFSTsn                                             (qes)SageIsHDFSTsn                                           query execute services ??
qesxtcCloseScan                                              (qes)xtcCloseScan                                            query execute services ??
qesxtcFetchCurrScan                                          (qes)xtcFetchCurrScan                                        query execute services ??
qesxtcFetchNextScan                                          (qes)xtcFetchNextScan                                        query execute services ??
qesxtcOpenScan                                               (qes)xtcOpenScan                                             query execute services ??
qjsnGenerateCapabilitySet                                    (qjsn)GenerateCapabilitySet                                  query json ??
qjsngCloseSageUga                                            (qjsn)gCloseSageUga                                          query json ??
qjsngCloseUga                                                (qjsn)gCloseUga                                              query json ??
qjsngInitSageUga                                             (qjsn)gInitSageUga                                           query json ??
qjsngStreamFromLobInSage                                     (qjsn)gStreamFromLobInSage                                   query json ??
qjsngStreamFromLob_h                                         (qjsn)gStreamFromLob_h                                       query json ??
qkaQknSTPruneKaf                                             (qka)QknSTPruneKaf                                           query kernel allocation ??
qksSageFlushSgaCIC                                           (qksSage)FlushSgaCIC                                         query kernel sql exadata ??
qksSageFlushSgaCICDump                                       (qksSage)FlushSgaCICDump                                     query kernel sql exadata ??
qksSageFlushSgaImpl                                          (qksSage)FlushSgaImpl                                        query kernel sql exadata ??
qksSageLoadInternalExternalMD                                (qksSage)LoadInternalExternalMD                              query kernel sql exadata ??
qksSageMakeTbsHDFS                                           (qksSage)MakeTbsHDFS                                         query kernel sql exadata ??
qkssage_kkxbct                                               (qkssa)ge_kkxbct                                             query kernel sql sampling ??
r_ex_data_clear                                              r_ex_data_clear                                              ??
r_ex_data_update                                             r_ex_data_update                                             ??
r_ssl_ctx_ex_data_clear                                      r_ssl_ctx_ex_data_clear                                      ??
r_ssl_ctx_get_dh_uses                                        r_ssl_ctx_get_dh_uses                                        ??
r_ssl_ctx_set_dh_uses                                        r_ssl_ctx_set_dh_uses                                        ??
r_ssl_ec_cert_algs_are_equal                                 r_ssl_ec_cert_algs_are_equal                                 ??
r_ssl_get_dh_uses                                            r_ssl_get_dh_uses                                            ??
r_ssl_set_dh_uses                                            r_ssl_set_dh_uses                                            ??
releaseILData                                                releaseILData                                                ??
ri_ssl3_base                                                 ri_ssl3_base                                                 ??
ri_ssl3_ctx_dh_tmp                                           ri_ssl3_ctx_dh_tmp                                           ??
ri_ssl3_dh_tmp                                               ri_ssl3_dh_tmp                                               ??
ri_ssl_cert_dup_params                                       ri_ssl_cert_dup_params                                       ??
ri_ssl_cipher_ctx_cipher_size                                ri_ssl_cipher_ctx_cipher_size                                ??
ri_ssl_cipher_ctx_is_aead                                    ri_ssl_cipher_ctx_is_aead                                    ??
sageDataReInitCtx                                            (sageData)ReInitCtx                                          exadata specific data layer ??
sageetProcessInternalTable                                   (sage)etProcessInternalTable                                 exadata specific ??
sageet_fp_buf_corrupt                                        (sage)et_fp_buf_corrupt                                      exadata specific ??
sageet_fp_buf_noalloc                                        (sage)et_fp_buf_noalloc                                      exadata specific ??
sageet_fp_buf_noresize                                       (sage)et_fp_buf_noresize                                     exadata specific ??
sagesqlFindCidInStoridx                                      (sagesql)FindCidInStoridx                                    exadata specific sql ??
seqSetStartWith                                              (seq)SetStartWith                                            sequence numbers ??
seqclp                                                       (seq)clp                                                     sequence numbers ??
seqmmv                                                       (seq)mmv                                                     sequence numbers ??
seqsetmin                                                    (seq)setmin                                                  sequence numbers ??

I was asked some time ago what the Oracle database event ‘TCP socket (KGAS)’ means. This blogpost is a deep dive into what this event times in Oracle database 12.1.0.2.180717.

This event is not normally seen, only when TCP connections are initiated from the database using packages like UTL_TCP, UTL_SMTP and the one used in this article, UTL_HTTP.

A very basic explanation is this event times the time that a database foreground session spends on TCP connection management and communicating over TCP, excluding client and database link (sqlnet) networking. If you trace the system calls, you see that mostly that is working with a (network) socket. Part of the code in the oracle database that is managing that, sits in the kernel code layer kgas, kernel generic (of which I am quite sure, and then my guess:) asynchronous services, which explains the naming of the event.

This is what the Oracle online manual (https://docs.oracle.com/database/121/REFRN/GUID-203ACA60-9912-4493-9B79-EA4CDE89A78D.htm#REFRN00642 – Oracle is notorious for breaking links) says about ‘TCP socket (KGAS)’:

C.3.157 TCP Socket (KGAS)
A session is waiting for an external host to provide requested data over a network socket. The time that this wait event tracks does not indicate a problem, and even a long wait time is not a reason to contact Oracle Support. It naturally takes time for data to flow between hosts over a network, and for the remote aspect of an application to process any request made to it. An application that communicates with a remote host must wait until the data it will read has arrived. In addition, on Microsoft Windows, a separate thread monitors the arrival of traffic. This thread spends most of its life in waits tracked by the TCP Socket (KGAS) wait event.

Wait Time: The total elapsed time for the network connection to be established or for data to arrive from over the network

Parameter Description
P0 For Oracle internal use only. Values 8, 9, and 10 occur within the special thread present on Microsoft Windows; other P0 values occur in normal user sessions.

P1 For Oracle internal use only

Overall, the basic explanation that Oracle provides is mostly correct. I think the text saying to not contact Oracle support is not relevant, but maybe there is a need to relieve Oracle support. In my tests, I found that the full TCP connection lifecycle (creation, usage and removal) is put under this event, for which the text seems to emphasise on waiting for a remote host, which would be the most logical culprit for wait times, but other issues could lead to wait times additionally. This means the wait event itself is not explaining what it is showing, outside of TCP connection management.

The wait time explanation is nearly complete. If it would say something like ‘all TCP connection management and usage’ it would have fully covered it, it now excludes disconnecting and sending, because it explicitly mentions creating the connecting and receiving (waiting for data).

I do not understand what is meant with P0 and P1. I think it is p1 and p2 of the wait event, but naming it P0 and P1 is weird. When looking at the explanation it reads to me ‘we do not wish to explain anything to you’.

So, that means I am going to find this out myself….

If you are interested in this, or do want to write articles like this too, I urge you to replay this on your own system.

First of all, create a small setup which you can use to actually execute UTL_HTTP. The example essentially is taken from father of code examples, Tim Hall/Oracle base. Thank you Tim!
First setup the database to allow a user (in my case ‘ts’) to create the objects and use the network:

grant create sequence to ts;
grant create procedure to ts;
grant create table to ts;
grant alter session to ts;
begin
  dbms_network_acl_admin.create_acl (
    acl => 'anything.xml',
    description => 'allow anything',
    principal => 'TS',
    is_grant => true,
    privilege => 'connect'
  );
end;
begin
  dbms_network_acl_admin.assign_acl (
    acl => 'anything.xml',
    host => '*'
  );
end;

Then connect as the actual user (ts), and create the objects and the procedure that uses UTL_HTTP:

drop table http_clob_test;
create table http_clob_test (
        id number(10),
        url varchar2(255),
        data clob,
        constraint http_clob_test_pk primary key (id)
);
drop sequence http_clob_test_seq;
create sequence http_clob_test_seq;
CREATE OR REPLACE PROCEDURE load_html_from_url (p_url  IN  VARCHAR2) AS
  -- examples by tim hall
  -- https://oracle-base.com/articles/misc/retrieving-html-and-binaries-into-tables-over-http
  l_http_request   UTL_HTTP.req;
  l_http_response  UTL_HTTP.resp;
  l_clob           CLOB;
  l_text           VARCHAR2(32767);
BEGIN
  DBMS_LOB.createtemporary(l_clob, FALSE);

  -- Make a HTTP request and get the response.
  l_http_request  := UTL_HTTP.begin_request(p_url);
  l_http_response := UTL_HTTP.get_response(l_http_request);

  -- Copy the response into the CLOB.
  BEGIN
    LOOP
      UTL_HTTP.read_text(l_http_response, l_text, 32766);
      DBMS_LOB.writeappend (l_clob, LENGTH(l_text), l_text);
    END LOOP;
  EXCEPTION
    WHEN UTL_HTTP.end_of_body THEN
      UTL_HTTP.end_response(l_http_response);
  END;

  -- Insert the data into the table.
  INSERT INTO http_clob_test (id, url, data)
  VALUES (http_clob_test_seq.NEXTVAL, p_url, l_clob);

  -- Relase the resources associated with the temporary LOB.
  DBMS_LOB.freetemporary(l_clob);
EXCEPTION
  WHEN OTHERS THEN
    UTL_HTTP.end_response(l_http_response);
    DBMS_LOB.freetemporary(l_clob);
    RAISE;
END load_html_from_url;
/

The first thing to do is turn on sql_trace at level 8 to include waits:

set serverout on
alter session set events 'sql_trace level 8';
exec load_html_from_url('http://orafun.info/');
alter session set events 'sql_trace off';

If you look at the relevant piece, which means where it shows the wait events, it looks like this:

WAIT #139864521752120: nam='TCP Socket (KGAS)' ela= 128265  =2  =0  =0 obj#=662 tim=86395107497
WAIT #139864521752120: nam='TCP Socket (KGAS)' ela= 395  =5  =0  =0 obj#=662 tim=86395110191
WAIT #139864521752120: nam='TCP Socket (KGAS)' ela= 150  =6  =0  =0 obj#=662 tim=86395111115
WAIT #139864521752120: nam='TCP Socket (KGAS)' ela= 131998  =6  =0  =0 obj#=662 tim=86395243764
WAIT #139864521752120: nam='TCP Socket (KGAS)' ela= 269  =4  =0  =0 obj#=662 tim=86395245182
WAIT #139864521752120: nam='direct path write temp' ela= 4137 file number=201 first dba=257795 block cnt=1 obj#=662 tim=86395250494
WAIT #139864521752120: nam='TCP Socket (KGAS)' ela= 352  =3  =2  =0 obj#=662 tim=86395251294

What is shown here is some quite spectacular differences in elapsed time. Also, the only way to understand what is actually done flagged as ‘TCP Socket (KGAS)’ is the value following ‘ela’, which is the event p1 value.
The pattern is:

- 2
- 5
- 6
- 6
- 4
- 3

It’s relatively simple to guess what a few of these are:

- 2 - connect
- 5 - send
- 6 - \
- 6 - |   receiving ?
- 4 - /
- 3 - close

But if you include the timing, there must be more into play:

- 2 - ela= 128265   connect
- 5 - ela= 395      send
- 6 - ela= 150      \
- 6 - ela= 131998   |   receiving ?
- 4 - ela= 14       /
- 3 - ela= 177     close

2/connect: In order to build up a connection, a tcp connection needs to be created and established. That takes some time.
5/send: Sending from the perspective of a userland process is writing into a socket, which will get send by the operating system independently. This means sending from a userland process normally takes relative little time, because it’s not waiting for actually sending it.
6,4/receive: At this time, this doesn’t make sense to me.
3/close: Closing for a userland process is a simple, swift task. The operating system will keep the port open for some time, etc. but this is not visible for the user land application.

Let’s pull an old trick out of the hat: use strace (system call tracing) with an emphasis on writing on an oracle session that has SQL trace with waits enabled set. This will show the system calls executed, and show exactly when the oracle engine ends a wait, so we can reasonably well establish a relation between wait events and system calls. I say “reasonably well”, because we can’t see when Oracle truly started timing the wait event (kslwtbctx), only the output to trace file as part of ending the wait event (kslwtectx).

The way I done it, is using the command ‘strace -e write=all -p 18513 -o utl_http_test.txt’. Obviously 18513 is the process ID of the database foreground process. The results of the strace are in utl_http_test.txt.

Now open utl_http_test.txt and search for KGAS. The full output is way too much text, let me show some of the output which I think is noteworthy. Again: this is selective, partial output.
I do maintain the order in which the calls are visible.

1. TCP Socket (KGAS) p1=2, earlier annotated as ‘connect’

-- try to find a socket that has been created by nscd (name server caching deamon)
-- two times?
--
socket(AF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 9
connect(9, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(9)                                = 0
socket(AF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 9
connect(9, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(9)                                = 0
--
-- obtain file status of resolv.conf (hostname resolving configuration file)
--
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=93, ...}) = 0
--
-- open and read host.conf (another hostname resolving configuration file)
--
open("/etc/host.conf", O_RDONLY|O_CLOEXEC) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=9, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f34bf377000
read(9, "multi on\n", 4096)             = 9
read(9, "", 4096)                       = 0
close(9)                                = 0
--
-- open and read resolv.conf (hostname resolving configuration)
--
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=93, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f34bf377000
read(9, "# Generated by NetworkManager\nse"..., 4096) = 93
read(9, "", 4096)                       = 0
close(9)                                = 0
--
-- open /etc/hosts (ip address to hostname mapping locally)
--
open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=200, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f34bf377000
read(9, "127.0.0.1   localhost localhost."..., 4096) = 200
read(9, "", 4096)                       = 0
close(9)
--
-- at this point two dynamic loadable libraries are read: libnss_dns.so.2 and libresolv.so.2
--
-- this is the DNS lookup of orafun.info
-- again, this is done twice, just like the use of /var/run/nscd/socket above?
--
socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 9
connect(9, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.0.2.3")}, 16) = 0
poll([{fd=9, events=POLLOUT}], 1, 0)    = 1 ([{fd=9, revents=POLLOUT}])
sendto(9, "=#\1\0\0\1\0\0\0\0\0\0\6orafun\4info\0\0\1\0\1", 29, MSG_NOSIGNAL, NULL, 0) = 29
 | 00000  3d 23 01 00 00 01 00 00  00 00 00 00 06 6f 72 61  =#...........ora |
 | 00010  66 75 6e 04 69 6e 66 6f  00 00 01 00 01           fun.info.....    |
poll([{fd=9, events=POLLIN}], 1, 5000)  = 1 ([{fd=9, revents=POLLIN}])
ioctl(9, FIONREAD, [45])                = 0
recvfrom(9, "=#\201\200\0\1\0\1\0\0\0\0\6orafun\4info\0\0\1\0\1\300\f\0"..., 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.0.2.3")}, [16]) = 45
close(9)                                = 0
socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 9
connect(9, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.0.2.3")}, 16) = 0
poll([{fd=9, events=POLLOUT}], 1, 4971) = 1 ([{fd=9, revents=POLLOUT}])
sendto(9, "o=\1\0\0\1\0\0\0\0\0\0\6orafun\4info\0\0\34\0\1", 29, MSG_NOSIGNAL, NULL, 0) = 29
 | 00000  6f 3d 01 00 00 01 00 00  00 00 00 00 06 6f 72 61  o=...........ora |
 | 00010  66 75 6e 04 69 6e 66 6f  00 00 1c 00 01           fun.info.....    |
poll([{fd=9, events=POLLIN}], 1, 4970)  = 1 ([{fd=9, revents=POLLIN}])
ioctl(9, FIONREAD, [109])               = 0
recvfrom(9, "o=\201\200\0\1\0\0\0\1\0\0\6orafun\4info\0\0\34\0\1\300\f\0"..., 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.0.2.3")}, [16]) = 109
close(9)                                = 0
--
-- an epoll is created at file descriptor 9 (epoll: I/O event notification facility)
--
epoll_create(82)                        = 9
fcntl(9, F_SETFD, FD_CLOEXEC)           = 0
--
-- an IPV6 socket is created at file descriptor 11, 
-- bound to the IPV6 equivalent of localhost (::1),
-- destination port 0, source port 63257,
-- and is NOT connected.
--
socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 11
bind(11, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
getsockname(11, {sa_family=AF_INET6, sin6_port=htons(63257), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
getpeername(11, 0x7ffdea6ba0f8, 0x7ffdea6ba1c8) = -1 ENOTCONN (Transport endpoint is not connected)
getsockopt(11, SOL_SOCKET, SO_SNDBUF, [262144], [4]) = 0
getsockopt(11, SOL_SOCKET, SO_RCVBUF, [262144], [4]) = 0
fcntl(11, F_SETFD, FD_CLOEXEC)          = 0
fcntl(11, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
--
-- File descriptor 11 is added to the epoll at file descriptor 9.
--
epoll_ctl(9, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=3110993336, u64=139864426020280}}) = 0
--
-- A connection is created to the true destination (orafun.info/18.218.92.122).
-- This connection gets file descriptor 12.
-- Destination port 80 (http), source port 11751.
--
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 12
fcntl(12, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
connect(12, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("18.218.92.122")}, 16) = -1 EINPROGRESS (Operation now in progress)
times(NULL)                             = 438106227
mmap(NULL, 786432, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f34b959b000
poll([{fd=12, events=POLLOUT}], 1, 60000) = 1 ([{fd=12, revents=POLLOUT}])
getsockopt(12, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
fcntl(12, F_GETFL)                      = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(12, F_SETFL, O_RDWR)              = 0
getsockname(12, {sa_family=AF_INET, sin_port=htons(11751), sin_addr=inet_addr("10.0.2.15")}, [16]) = 0
getsockopt(12, SOL_SOCKET, SO_SNDBUF, [87040], [4]) = 0
getsockopt(12, SOL_SOCKET, SO_RCVBUF, [374400], [4]) = 0
setsockopt(12, SOL_TCP, TCP_NODELAY, [1], 4) = 0
fcntl(12, F_SETFD, FD_CLOEXEC)          = 0
--
-- And this is the wait event written by the process: TCP Socket (KGAS), p1=2
--
write(7, "WAIT #139864521752120: nam='TCP "..., 95) = 95
 | 00000  57 41 49 54 20 23 31 33  39 38 36 34 35 32 31 37  WAIT #1398645217 |
 | 00010  35 32 31 32 30 3a 20 6e  61 6d 3d 27 54 43 50 20  52120: nam='TCP  |
 | 00020  53 6f 63 6b 65 74 20 28  4b 47 41 53 29 27 20 65  Socket (KGAS)' e |
 | 00030  6c 61 3d 20 31 32 38 32  36 35 20 20 3d 32 20 20  la= 128265  =2   |
 | 00040  3d 30 20 20 3d 30 20 6f  62 6a 23 3d 36 36 32 20  =0  =0 obj#=662  |
 | 00050  74 69 6d 3d 38 36 33 39  35 31 30 37 34 39 37     tim=86395107497  |

So yes, I am not sure if all of this is in the wait event, but there is a lot of stuff happening to build a connection to the remote server.

In order to find out why the lookup which is tried via the NSCD socket at the beginning, and later via DNS, is done twice, I ran the same procedure again and used tcpdump to look at the actual network traffic. This explained a lot:

# tcpdump -n host 10.0.2.3 and port 53
09:14:02.923389 IP 10.0.2.15.16819 > 10.0.2.3.domain: 15651+ A? orafun.info. (29)
09:14:02.948791 IP 10.0.2.3.domain > 10.0.2.15.16819: 15651 1/0/0 A 18.218.92.122 (45)
09:14:02.952304 IP 10.0.2.15.54590 > 10.0.2.3.domain: 28477+ AAAA? orafun.info. (29)
09:14:02.979534 IP 10.0.2.3.domain > 10.0.2.15.54590: 28477 0/1/0 (109)

In other words: first a DNS A record is requested (TCPv4 DNS name lookup), which results in the IPv4 ip address, then a DNS AAAA record is requested (TCPv6 DNS name lookup), which resulted in no ip address. In other words: orafun.info only has an IPv4 ip address. So the two lookups actually do have a function.

2. TCP Socket (KGAS) p1=5, earlier annotated as ‘send’
These are the systemcalls that are visible and quite probably related to the send wait event:

--
-- file descriptor 12 holding the connection to the destination server is added to the epoll at file descriptor 9
--
epoll_ctl(9, EPOLL_CTL_ADD, 12, {EPOLLIN, {u32=3110998864, u64=139864426025808}}) = 0
--
-- Then the http get request is sent to the destination server at its normal file descriptor, 12.
--
write(12, "GET / HTTP/1.1\r\nHost: orafun.inf"..., 56) = 56
 | 00000  47 45 54 20 2f 20 48 54  54 50 2f 31 2e 31 0d 0a  GET / HTTP/1.1.. |
 | 00010  48 6f 73 74 3a 20 6f 72  61 66 75 6e 2e 69 6e 66  Host: orafun.inf |
 | 00020  6f 0d 0a 43 6f 6e 6e 65  63 74 69 6f 6e 3a 20 63  o..Connection: c |
 | 00030  6c 6f 73 65 0d 0a 0d 0a                           lose....         |
--
-- And this is the wait event written by the process: TCP Socket (KGAS), p1=5
--
write(7, "WAIT #139864521752120: nam='TCP "..., 92) = 92
 | 00000  57 41 49 54 20 23 31 33  39 38 36 34 35 32 31 37  WAIT #1398645217 |
 | 00010  35 32 31 32 30 3a 20 6e  61 6d 3d 27 54 43 50 20  52120: nam='TCP  |
 | 00020  53 6f 63 6b 65 74 20 28  4b 47 41 53 29 27 20 65  Socket (KGAS)' e |
 | 00030  6c 61 3d 20 33 39 35 20  20 3d 35 20 20 3d 30 20  la= 395  =5  =0  |
 | 00040  20 3d 30 20 6f 62 6a 23  3d 36 36 32 20 74 69 6d   =0 obj#=662 tim |
 | 00050  3d 38 36 33 39 35 31 31  30 31 39 31              =86395110191     |

3. TCP Socket (KGAS) p1=6, earlier annotated as ‘receive’

--
-- Calling epoll_wait with timeout set to 0, so it doesn't block.
--
epoll_wait(9, [], 82, 0)                = 0
--
-- And this is the wait event written by the process: TCP Socket (KGAS), p1=6
-- 
write(7, "WAIT #139864521752120: nam='TCP "..., 92) = 92
 | 00000  57 41 49 54 20 23 31 33  39 38 36 34 35 32 31 37  WAIT #1398645217 |
 | 00010  35 32 31 32 30 3a 20 6e  61 6d 3d 27 54 43 50 20  52120: nam='TCP  |
 | 00020  53 6f 63 6b 65 74 20 28  4b 47 41 53 29 27 20 65  Socket (KGAS)' e |
 | 00030  6c 61 3d 20 31 35 30 20  20 3d 36 20 20 3d 30 20  la= 150  =6  =0  |
 | 00040  20 3d 30 20 6f 62 6a 23  3d 36 36 32 20 74 69 6d   =0 obj#=662 tim |
 | 00050  3d 38 36 33 39 35 31 31  31 31 31 35              =86395111115     |

I have been thinking a lot about this seemingly weird call. It calls epoll_wait, but indicates it doesn’t want to wait (timeout=0), and even if epol_wait would have returned anything, indicated by a return code > 0, the epoll_event pointer is not set (indicated by []). The epoll file descriptor is used, but the only working file descriptor in the epoll is file descriptor 12, which has just been sent a http GET command, so the functionality of epoll is used.

This doesn’t make sense, unless you think about the asynchronous IO implementation of Oracle (see a lot of my earlier investigations), for which (in the case of asynchronous IO) io_getevents was called in a similar matter, timeout set to 0, to be able to do more requests while earlier IO requests are executed by the kernel. So my current theory here is that if multiple requests are happening, this mechanism provides a way to handle them.

If you have a simple single request, like in my case, this systemcall seems redundant. And because it queries the epoll file descriptor right after the request, it returns zero events, because there hardly has been any time after sending the http GET request.

4. Second TCP Socket (KGAS) p1=6, earlier annotated as ‘receive’

--
-- Calling epoll_wait with timeout set to 30000 (milliseconds).
--
epoll_wait(9, [{EPOLLIN, {u32=3110998864, u64=139864426025808}}], 82, 30000) = 1
--
-- And this is the second wait event written by the process: TCP Socket (KGAS), p1=6
-- 
write(7, "WAIT #139864521752120: nam='TCP "..., 95) = 95
 | 00000  57 41 49 54 20 23 31 33  39 38 36 34 35 32 31 37  WAIT #1398645217 |
 | 00010  35 32 31 32 30 3a 20 6e  61 6d 3d 27 54 43 50 20  52120: nam='TCP  |
 | 00020  53 6f 63 6b 65 74 20 28  4b 47 41 53 29 27 20 65  Socket (KGAS)' e |
 | 00030  6c 61 3d 20 31 33 31 39  39 38 20 20 3d 36 20 20  la= 131998  =6   |
 | 00040  3d 30 20 20 3d 30 20 6f  62 6a 23 3d 36 36 32 20  =0  =0 obj#=662  |
 | 00050  74 69 6d 3d 38 36 33 39  35 32 34 33 37 36 34     tim=86395243764  |

This is the second time epoll_wait is called, and this one is blocking, because timeout has been set to 30000 milliseconds. If you look at the ela time, this took some time, and this now makes perfect sense: this system calls waits for an event to become available in the epoll, so it waits for the response of the remote http server. Please mind this call just notifies the userland process that the response is ready, the received data yet has to be read.

5. TCP Socket (KGAS) p1=4, earlier annotated as ‘receive’

--
-- At this point we know there is a response. First the original file descriptor is removed from the epoll:
--
epoll_ctl(9, EPOLL_CTL_DEL, 12, 0x7ffdea6b9710) = 0
--
-- The the response is read from file descriptor 12:
--
read(12, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 4096) = 2687
--
-- Then file descriptor 12 is added to the epoll again.
--
epoll_ctl(9, EPOLL_CTL_ADD, 12, {EPOLLIN, {u32=3110998864, u64=139864426025808}}) = 0
--
-- And a wait event written by the process: TCP Socket (KGAS), p1=4
--
write(7, "WAIT #139864521752120: nam='TCP "..., 92) = 92
 | 00000  57 41 49 54 20 23 31 33  39 38 36 34 35 32 31 37  WAIT #1398645217 |
 | 00010  35 32 31 32 30 3a 20 6e  61 6d 3d 27 54 43 50 20  52120: nam='TCP  |
 | 00020  53 6f 63 6b 65 74 20 28  4b 47 41 53 29 27 20 65  Socket (KGAS)' e |
 | 00030  6c 61 3d 20 32 36 39 20  20 3d 34 20 20 3d 30 20  la= 269  =4  =0  |
 | 00040  20 3d 30 20 6f 62 6a 23  3d 36 36 32 20 74 69 6d   =0 obj#=662 tim |
 | 00050  3d 38 36 33 39 35 32 34  35 31 38 32              =86395245182     |

So, what p1 set to 4 actually means, is that once the connection did return data, which is checked using epoll, and visible with p1 set to 6, it is read into the process. This is also the reason this takes very little time, this is the time to read data from kernelspace to user space, and to manage the connection’s file descriptor. It is taken off the epoll in order not to disturb it, and it is added again because there could be another request.

6. TCP Socket (KGAS) p1=3, earlier annotated as ‘close’

--
-- file descriptor 12 removed from the epoll
--
epoll_ctl(9, EPOLL_CTL_DEL, 12, 0x7ffdea6bac20) = 0
--
-- file descriptor 12 is closed, closing the network connection
--
close(12)                               = 0
--
-- And a wait event written by the process: TCP Socket (KGAS), p1=3
--
write(7, "WAIT #139864521752120: nam='TCP "..., 92) = 92
 | 00000  57 41 49 54 20 23 31 33  39 38 36 34 35 32 31 37  WAIT #1398645217 |
 | 00010  35 32 31 32 30 3a 20 6e  61 6d 3d 27 54 43 50 20  52120: nam='TCP  |
 | 00020  53 6f 63 6b 65 74 20 28  4b 47 41 53 29 27 20 65  Socket (KGAS)' e |
 | 00030  6c 61 3d 20 33 35 32 20  20 3d 33 20 20 3d 32 20  la= 352  =3  =2  |
 | 00040  20 3d 30 20 6f 62 6a 23  3d 36 36 32 20 74 69 6d   =0 obj#=662 tim |
 | 00050  3d 38 36 33 39 35 32 35  31 32 39 34              =86395251294     |

I don’t think this part holds any surprises. The network file descriptor is first removed from the epoll, and then it is closed, ending the TCP connection that was setup to perform a http request (in my case, I didn’t test, but I believe you will see the same with for example a SMTP connection, or any other type of TCP connection).

Summary

The basic message of this article is not surprising, nor does it conflict with current knowledge. Whenever you see a wait event ‘TCP Socket (KGAS)’, it means a foreground process is performing TCP networking via PLSQL. This wait event is a single event for creating, sending, receiving and closing a connection.

The true information of this article is how you can use the p1 value of the event to learn what actually the foreground is doing, and thus should give you more information to troubleshoot in the case of long waiting times.

TCP Socket (KGAS) p1 values:
1 - ?
2 - perform DNS lookup and create connection
3 - close connection
4 - copy TCP response into process space
5 - send request
6 - wait for TCP response to become available
7 - ?
8 - ? \
9 - ? | According to documentation, windows only in a 'special thread'.
10- ? /

Starting from Oracle 12, in a default configured database, there are more log writer processes than the well known ‘LGWR’ process itself, which are the ‘LGnn’ processes:

$ ps -ef | grep test | grep lg
oracle   18048     1  0 12:50 ?        00:00:13 ora_lgwr_test
oracle   18052     1  0 12:50 ?        00:00:06 ora_lg00_test
oracle   18056     1  0 12:50 ?        00:00:00 ora_lg01_test

These are the log writer worker processes, for which the minimal amount is equal to the amount public redo strands. Worker processes are assigned to a group, and the group is assigned to a public redo strand. The amount of worker processes in the group is dependent on the undocumented parameter “_max_log_write_parallelism”, which is one by default.

The actual usage of the worker processes is dependent in the first place on the value of the undocumented parameter “_use_single_log_writer”, for which the default value is ‘ADAPTIVE’, which means it’s switching automatically between ‘single log writer mode’, which is the traditional way of the LGWR process handling everything that the log writer functionality needs to do, and the ‘scalable log writer mode’, which means the log writer functionality is presumably using the log writer worker processes.

Other values for “_use_single_log_writer” are ‘TRUE’ to set ‘single log writer mode’, or ‘FALSE’ to set ‘scalable log writer mode’ fixed.

I assume most readers of this blog will know that the master log writer idle work cycle is sleeping on a semaphore (semtimedop()) under the wait event ‘rdbms ipc message’ for 3 seconds, then performs some “housekeeping”, after which it’ll sleep again repeating the small cycle of sleeping and housekeeping. For the log writer worker processes, this looks different if you look at the wait event information of the log writer worker processes:

135,59779,@1    14346                    DEDICATED oracle@memory-presentation.local (LGWR)	    time:1909.44ms,event:rdbms ipc message,seq#:292
48,34282,@1     14350                    DEDICATED oracle@memory-presentation.local (LG00)	    time:57561.85ms,event:LGWR worker group idle,seq#:150
136,24935,@1    14354                    DEDICATED oracle@memory-presentation.local (LG01)	    time:112785.66ms,event:LGWR worker group idle,seq#:74

The master log writer process (LGWR) has been sleeping for 1.9s when I queried the database, and it will sleep for 3 seconds, and then do some work and sleep again. However, the log writer worker processes have been sleeping for much longer: LG00 for 57.6s and LG01 for 112.8s, and the event is different: ‘LGWR worker group idle’. How is this implemented? Let’s look!

$ strace -p $(pgrep lg01)
strace: Process 14354 attached
semtimedop(360448, [{27, -1, 0}], 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)
semtimedop(360448, [{27, -1, 0}], 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)

I used strace on the LG01 process, and it’s still doing the same as most idle background processes are doing: sleeping on a semaphore for 3 seconds. But, it does not end its wait like LGWR does, the event the log writer worker processes are waiting in keeps on being timed.

Using a pin tools debugtrace shows the following:

 | | < semtimedop+0x000000000023 returns: 0xffffffffffffffff
 | | > __errno_location(0x38000, 0x7ffce278c328, ...)
 | | | > fthread_self(0x38000, 0x7ffce278c328, ...)
 | | | < fthread_self+0x000000000024 returns: 0
 | | < __errno_location+0x000000000010 returns: 0x7f7e930a26a0
 | < sskgpwwait+0x00000000014e returns: 0
 < skgpwwait+0x0000000000e0 returns: 0
 > ksuSdiInProgress(0x19e80, 0x19e80, ...)
 < ksuSdiInProgress+0x000000000035 returns: 0
 > sltrgftime64(0x19e80, 0x19e80, ...)
 | > clock_gettime@plt(0x1, 0x7ffce278c3a0, ...)
 | | > clock_gettime(0x1, 0x7ffce278c3a0, ...)
 | | < clock_gettime+0x000000000069 returns: 0
 | < clock_gettime+0x00000000003a returns: 0
 < sltrgftime64+0x00000000004c returns: 0x19c253f3ff
 > kslwo_getcbk(0xa2, 0xd80fa62, ...)
 < kslwo_getcbk+0x000000000017 returns: 0
 > kgslwait_last_waitctx_time_waited_usecs(0x7f7e930a29a0, 0x6dfd01c0, ...)
 < kgslwait_last_waitctx_time_waited_usecs+0x000000000045 returns: 0x25e5e80
 > kskiorm(0x6d1854a8, 0, ...)
 < kskiorm+0x00000000001e returns: 0
 > kfias_iswtgon_ksfd(0x6d1854a8, 0, ...)
 < kfias_iswtgon_ksfd+0x00000000002b returns: 0
 > kxdbio_has_work(0x7ffce278c3c4, 0x6003d010, ...)
 < kxdbio_has_work+0x000000000027 returns: 0
 > skgpwwait(0x7ffce278c630, 0x7f7e930a7ca0, ...)
 | > kslwait_conv_wait_time(0x2dc6c0, 0x7f7e930a7ca0, ...)
 | < kslwait_conv_wait_time+0x000000000027 returns: 0x2dc6c0
 | > sskgpwwait(0x7ffce278c630, 0x7f7e930a7ca0, ...)
 | | > semtimedop(0x38000, 0x7ffce278c328, ...)
 | | < semtimedop+0x000000000023 returns: 0xffffffffffffffff

And a full stack trace of a log writer worker look like this:

$ pstack $(pgrep lg01)
#0  0x00007feda8eaebda in semtimedop () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000000010f9cca6 in sskgpwwait ()
#2  0x0000000010f9a2e8 in skgpwwait ()
#3  0x0000000010a66995 in ksliwat ()
#4  0x0000000010a65d25 in kslwaitctx ()
#5  0x00000000031fb4d0 in kcrfw_slave_queue_remove ()
#6  0x00000000031fad2a in kcrfw_slave_group_main ()
#7  0x00000000012160fa in ksvrdp_int ()
#8  0x000000000370d99a in opirip ()
#9  0x0000000001eb034a in opidrv ()
#10 0x0000000002afedf1 in sou2o ()
#11 0x0000000000d0547a in opimai_real ()
#12 0x0000000002b09b31 in ssthrdmain ()
#13 0x0000000000d05386 in main ()

If you combine the pstack backtrace and the debugtrace information, you see that the idle cycle does not leave the ‘ksliwat’ function, so the wait event is not finished. Quickly looking at the other functions, it’s easy to spot it reads the system clock (sltrgftime64), updates some information (kgslwait_last_waitctx_time_waited_usecs) and then performs some proactive IO checks (kskiorm, kfias_iswtgon_ksfd, kxdbio_has_work) after which it calls the post/wait based functions to setup the semaphore again.

Conclusion so far is the log writer workers do perform a 3 second sleep just like the master log writer, however the wait event ‘LGWR worker group idle’ is not interrupted like ‘rdbms ipc message’ is for the master log writer. This means the wait time for the event for each worker process indicates the last time the worker process actually performed something. A next logical question then is: but what do the log writer worker processes perform? Do they entirely take over the master log writer functionality, or do they work together with the master log writer?

In order to fully understand the next part, it is very beneficial to read up on how the log writer works in ‘single log writer’ mode, where the master log writer handling the idle and work cycle itself:
https://fritshoogland.wordpress.com/2018/02/20/a-look-into-into-oracle-redo-part-4-the-log-writer-null-write/
https://fritshoogland.wordpress.com/2018/02/27/a-look-into-oracle-redo-part-5-the-log-writer-writing/

If you want to perform this investigation yourself, make sure the database is in ‘scalable log writer’ mode, by setting “_use_single_log_writer” to FALSE. This is exactly what I did in order to make sure a log write is done in ‘scalable log writer’ mode.

Now let’s first apply some logic. Above the idle cycle of a log writer worker process is shown. Based on the ‘log writer null write’ blog post, we know that the log writer does advance the LWN and On-disk SCN every 3 seconds. Clearly, the log writer worker process does not do that. So that must mean the master log writer is still performing that function. It would also make very much sense, because it doesn’t matter for scalability if the master log writer performs the function of advancing the LWN and On-disk SCN or a worker process, nothing is waiting on it. Plus, if the master log writer performs most of its functions just like in ‘single log writer’ mode, the change to scalable mode would mean no change for client processes, any committing process must semop() the log writer to start writing.

Let’s look at the relevant debugtrace output of the master log writer in scalable log writer mode:

 | > kcrfw_redo_write_driver(0, 0, ...)
 | | > kcrfw_handle_member_write_errors(0, 0, ...)
 | | < kcrfw_handle_member_write_errors+0x000000000020 returns: 0x600161a0
 | | > kcmgtsf(0, 0, ...)
 | | | > sltrgatime64(0, 0, ...)
 | | | | > sltrgftime64(0, 0, ...)
 | | | | | > clock_gettime@plt(0x1, 0x7fff1fe13010, ...)
 | | | | | | > clock_gettime(0x1, 0x7fff1fe13010, ...)
 | | | | | | < clock_gettime+0x000000000069 returns: 0
 | | | | | < clock_gettime+0x00000000003a returns: 0
 | | | | < sltrgftime64+0x00000000004c returns: 0x53747fe42
 | | | < sltrgatime64+0x00000000003e returns: 0x155d4fd
 | | < kcmgtsf+0x00000000032f returns: 0x3a182314
 | | > kcrfw_slave_adaptive_updatemode(0, 0x600161a0, ...)
 | | < kcrfw_slave_adaptive_updatemode+0x000000000080 returns: 0x7efe34d1f760
 | | > kcrfw_defer_write(0, 0x600161a0, ...)
 | | < kcrfw_defer_write+0x000000000038 returns: 0x7efe34d1f760
 | | > kcrfw_slave_queue_find(0, 0x600161a0, ...)
 | | < kcrfw_slave_queue_find+0x0000000000f1 returns: 0
 | | > kcrfw_slave_queue_setpreparing(0, 0x1, ...)
 | | < kcrfw_slave_queue_setpreparing+0x000000000021 returns: 0
 | | > kcrfw_slave_group_switchpic(0, 0x1, ...)
 | | < kcrfw_slave_group_switchpic+0x000000000050 returns: 0x699b4508
 | | > skgstmGetEpochTs(0, 0x1, ...)
 | | | > gettimeofday@plt(0x7fff1fe13070, 0, ...)
 | | | < __vdso_gettimeofday+0x0000000000fe returns: 0
 | | < skgstmGetEpochTs+0x000000000049 returns: 0x20debfd6192e5
 | | > kcsnew3(0x600113b8, 0x7fff1fe13228, ...)
 | | | > kcsnew8(0x600113b8, 0x7fff1fe13070, ...)
 | | | | > kslgetl(0x60049800, 0x1, ...)
 | | | | < kslgetl+0x00000000012f returns: 0x1
 | | | | > kslfre(0x60049800, 0x1, ...)
 | | | | < kslfre+0x0000000001e2 returns: 0
 | | | < kcsnew8+0x000000000117 returns: 0
 | | | > ub8_to_kscn_impl(0x66c3c7, 0x7fff1fe13228, ...)
 | | | < ub8_to_kscn_impl+0x000000000031 returns: 0
 | | < kcsnew3+0x00000000006f returns: 0x8000
 | | > ktfwtsm(0x3a182314, 0x7fff1fe13228, ...)
 | | | > kcmgtsf(0x2, 0x7fff1fe13228, ...)
 | | | | > sltrgatime64(0x2, 0x7fff1fe13228, ...)
 | | | | | > sltrgftime64(0x2, 0x7fff1fe13228, ...)
 | | | | | | > clock_gettime@plt(0x1, 0x7fff1fe12fe0, ...)
 | | | | | | | > clock_gettime(0x1, 0x7fff1fe12fe0, ...)
 | | | | | | | < clock_gettime+0x000000000069 returns: 0
 | | | | | | < clock_gettime+0x00000000003a returns: 0
 | | | | | < sltrgftime64+0x00000000004c returns: 0x537484a6d
 | | | | < sltrgatime64+0x00000000003e returns: 0x155d511
 | | | < kcmgtsf+0x0000000001b2 returns: 0x3a182314
 | | | > kcmtdif(0x3a182314, 0x3a182314, ...)
 | | | < kcmtdif+0x00000000001b returns: 0
 | | | > ksl_get_shared_latch_int(0x60050340, 0x6ddb1408, ...)
 | | | < ksl_get_shared_latch_int+0x00000000016b returns: 0x1
 | | <> kslfre(0x60050340, 0x66c3c7, ...)
 | | < kslfre+0x0000000001e2 returns: 0
 | | > kcn_stm_write(0x7fff1fe13228, 0x66c3c7, ...)
 | | | > kstmgetsectick(0x7fff1fe13228, 0x66c3c7, ...)
 | | | < kstmgetsectick+0x00000000003a returns: 0x5ae4c494
 | | | > ksl_get_shared_latch_int(0x6004ee40, 0x6ddb1408, ...)
 | | | < ksl_get_shared_latch_int+0x00000000016b returns: 0x1
 | | <> kslfre(0x6004ee40, 0x2244, ...)
 | | < kslfre+0x0000000001e2 returns: 0
 | | > kcrfw_redo_write_initpic(0x699b4508, 0x7fff1fe13228, ...)
 | | | > kscn_to_ub8_impl(0x7fff1fe13228, 0x7fff1fe13228, ...)
 | | | < kscn_to_ub8_impl+0x00000000003e returns: 0x66c3c7
 | | < kcrfw_redo_write_initpic+0x0000000000dc returns: 0x3a182314
 | | > kscn_to_ub8_impl(0x7fff1fe13228, 0, ...)
 | | < kscn_to_ub8_impl+0x00000000003e returns: 0x66c3c7
 | | > kcrfw_gather_lwn(0x7fff1fe13268, 0x699b4508, ...)
 | | | > kslgetl(0x6abe4538, 0x1, ...)
 | | | < kslgetl+0x00000000012f returns: 0x1
 | | | > kcrfw_gather_strand(0x7fff1fe13268, 0, ...)
 | | | < kcrfw_gather_strand+0x0000000000c2 returns: 0
 | | | > kslfre(0x6abe4538, 0x17d5f, ...)
 | | | < kslfre+0x0000000001e2 returns: 0
 | | | > kslgetl(0x6abe45d8, 0x1, ...)
 | | | < kslgetl+0x00000000012f returns: 0x1
 | | | > kcrfw_gather_strand(0x7fff1fe13268, 0x1, ...)
 | | | < kcrfw_gather_strand+0x0000000000c2 returns: 0
 | | | > kslfre(0x6abe45d8, 0x137, ...)
 | | | < kslfre+0x0000000001e2 returns: 0
 | | < kcrfw_gather_lwn+0x00000000065c returns: 0xffffffff
 | | > krsh_trace(0x1000, 0x200, ...)
 | | < krsh_trace+0x00000000005d returns: 0
 | | > kspgip(0x71e, 0x1, ...)
 | | < kspgip+0x00000000023f returns: 0
 | | > kcrfw_slave_queue_setpreparing(0, 0, ...)
 | | < kcrfw_slave_queue_setpreparing+0x000000000021 returns: 0
 | | > kcrfw_slave_queue_flush_internal(0x1, 0, ...)
 | | < kcrfw_slave_queue_flush_internal+0x0000000000d7 returns: 0x1
 | | > kcrfw_do_null_write(0, 0, ...)
 | | | > kcrfw_slave_phase_batchdo(0, 0, ...)
 | | | | > kcrfw_slave_phase_enter(0, 0x9b, ...)
 | | | | < kcrfw_slave_phase_enter+0x000000000449 returns: 0
 | | | <> kcrfw_slave_phase_exit(0, 0x9b, ...)
 | | | < kcrfw_slave_phase_exit+0x00000000035a returns: 0
 | | | > kcrfw_post(0, 0, ...)
 | | | | > kcrfw_slave_single_getactivegroup(0, 0, ...)
 | | | | < kcrfw_slave_single_getactivegroup+0x000000000047 returns: 0x6a9a0718
 | | | | > kspGetInstType(0x1, 0x1, ...)
 | | | | | > vsnffe_internal(0x19, 0x1, ...)
 | | | | | | > vsnfprd(0x19, 0x1, ...)
 | | | | | | < vsnfprd+0x00000000000f returns: 0x8
 | | | | | | > kfIsASMOn(0x19, 0x1, ...)
 | | | | | | <> kfOsmInstanceSafe(0x19, 0x1, ...)
 | | | | | | < kfOsmInstanceSafe+0x000000000031 returns: 0
 | | | | | < vsnffe_internal+0x0000000000a7 returns: 0
 | | | | | > kspges(0x115, 0x1, ...)
 | | | | | < kspges+0x00000000010f returns: 0
 | | | | < kspGetInstType+0x0000000000b1 returns: 0x1
 | | | | > kcrfw_slave_phase_enter(0x1, 0x9b, ...)
 | | | | < kcrfw_slave_phase_enter+0x00000000006f returns: 0x9b
 | | | | > kcscu8(0x60016290, 0x7fff1fe12f98, ...)
 | | | | < kcscu8+0x000000000047 returns: 0x1
 | | | | > kcsaj8(0x60016290, 0x7fff1fe12f38, ...)
 | | | | < kcsaj8+0x0000000000dc returns: 0x1
 | | | | > kcrfw_slave_phase_exit(0x1, 0x9b, ...)
 | | | | < kcrfw_slave_phase_exit+0x00000000008e returns: 0
 | | | | > kslpsemf(0x97, 0, ...)
 | | | | | > ksl_postm_init(0x7fff1fe0ac30, 0x7fff1fe12c50, ...)
 | | | | | < ksl_postm_init+0x00000000002b returns: 0
 | | | | < kslpsemf+0x0000000006b5 returns: 0x1f
 | | | | > kcrfw_slave_barrier_nonmasterwait(0x6a9a0720, 0x4, ...)
 | | | | < kcrfw_slave_barrier_nonmasterwait+0x000000000035 returns: 0x600161a0
 | | | < kcrfw_post+0x000000000c1c returns: 0xd3
 | | < kcrfw_do_null_write+0x0000000000b2 returns: 0xd3
 | < kcrfw_redo_write_driver+0x000000000535 returns: 0xd3

The highlighted functions are extra functions executed when the instance is set to scalable log writer mode, or when adaptive mode has set the instance to scalable log writer mode. This means that the changes between the modes is minimal when there’s no writes, and outside of a few extra functions, the log writer does exactly the same.

The absence of any spectacular changes in the behaviour of the log writer when in scalable log writer mode when there are no writes does hint what the actual changes will be of the scalable mode, which is how writing is handled. In single log writer mode, the most time the log writer is process is likely to spend on is writing the change vectors into the online redologfiles, and maybe, if you have a bad application (!) semop()-ing foreground sessions will be second, if there are a large number of processes committing, because every process needs to be semop()-ed individually. These two functions, along with some other functionality are exactly what the log writer worker processes are doing.

This means that foreground processes do nothing different in scalable log writer mode, they signal (semop) the master log writer, which will investigate the public redo strands, and if the master log writer finds change vectors to write, it will assign log writer worker processes to perform the write, and the log writer worker process will semop() the foreground sessions to indicate the redo has been written when the instance is in post/wait mode, or do not semop() when the instance is in polling mode.

This is the entire function flow of a write when the instance is in scalable log writer mode:

 | > kcrfw_slave_queue_insert(0, 0xd3, ...)
 | | > kcrfw_slave_group_setcurrsize(0, 0, ...)
 | | < kcrfw_slave_group_setcurrsize+0x0000000001d1 returns: 0x1
 | | > _intel_fast_memcpy(0x6a9a05f8, 0x7ffdae335fa0, ...)
 | | <> _intel_fast_memcpy.P(0x6a9a05f8, 0x7ffdae335fa0, ...)
 | | <> __intel_ssse3_rep_memcpy(0x6a9a05f8, 0x7ffdae335fa0, ...)
 | | < __intel_ssse3_rep_memcpy+0x000000002798 returns: 0x6a9a05f8
 | | > kcrfw_slave_group_postall(0, 0xf0, ...)
 | | | > ksvgcls(0, 0xf0, ...)
 | | | < ksvgcls+0x000000000021 returns: 0
 | | | > ksl_post_proc(0x6ddb32f0, 0, ...)
 | | | <> kskpthr(0x6ddb32f0, 0, ...)
 | | | <> kslpsprns(0x6ddb32f0, 0, ...)
 | | | | > ksl_update_post_stats(0x6ddb32f0, 0, ...)
 | | | | | > dbgtTrcData_int(0x7f464c0676c0, 0x2050031, ...)
 | | | | | | > dbgtBucketRedirect(0x7f464c0676c0, 0x7ffdae335338, ...)
 | | | | | | < dbgtBucketRedirect+0x000000000050 returns: 0x1
 | | | | | | > dbgtIncInMemTrcRedirect(0x7f464c0676c0, 0x6fa, ...)
 | | | | | | < dbgtIncInMemTrcRedirect+0x000000000035 returns: 0x1
 | | | | | | > skgstmGetEpochTs(0x7f464c0676c0, 0x6fa, ...)
 | | | | | | | > gettimeofday@plt(0x7ffdae334e40, 0, ...)
 | | | | | | | < __vdso_gettimeofday+0x0000000000fe returns: 0
 | | | | | | < skgstmGetEpochTs+0x000000000049 returns: 0x20e067375b55d
 | | | | | | > dbgtrRecAllocate(0x7f464c0676c0, 0x7ffdae3352e0, ...)
 | | | | | | | > dbgtrPrepareWrite(0x7f464c0676c0, 0x65accba0, ...)
 | | | | | | | < dbgtrPrepareWrite+0x00000000011c returns: 0x4
 | | | | | | < dbgtrRecAllocate+0x000000000144 returns: 0x1
 | | | | | | > _intel_fast_memcpy(0x65acda30, 0x7ffdae3353d8, ...)
 | | | | | | <> _intel_fast_memcpy.P(0x65acda30, 0x7ffdae3353d8, ...)
 | | | | | | <> __intel_ssse3_rep_memcpy(0x65acda30, 0x7ffdae3353d8, ...)
 | | | | | | < __intel_ssse3_rep_memcpy+0x000000002030 returns: 0x65acda30
 | | | | | | > dbgtrRecEndSegment(0x7f464c0676c0, 0x7ffdae3352e0, ...)
 | | | | | | < dbgtrRecEndSegment+0x00000000011c returns: 0x77c000a4
 | | | | | < dbgtTrcData_int+0x000000000323 returns: 0x77c000a4
 | | | | < ksl_update_post_stats+0x00000000024f returns: 0x77c000a4
 | | | | > skgpwpost(0x7ffdae335480, 0x7f464c0acca0, ...)
 | | | | <> sskgpwpost(0x7ffdae335480, 0x7f464c0acca0, ...)
 | | | | | > semop@plt(0xc0000, 0x7ffdae335410, ...)
 | | | | | < semop+0x00000000000f returns: 0
 | | | | < sskgpwpost+0x00000000009a returns: 0x1
 | | | < kslpsprns+0x0000000001c3 returns: 0
 | | < kcrfw_slave_group_postall+0x0000000000a8 returns: 0
 | < kcrfw_slave_queue_insert+0x0000000001b6 returns: 0x667bc540

After the instance has established there are change vectors in kcrfw_gather_lwn, in single log writer mode, the function kcrfw_redo_write is called, which will call kcrfw_do_write which handles the writing, and kslpslf to semop any waiting processes among other things. Now in scalable log writer mode, kcrfw_slave_queue_insert is called which assigns work to worker processes, and then kcrfw_slave_group_postall is called to semop one or more worker processes.

The worker processes are sleeping on a semaphore, and if a process gets signalled, it exits the kcrfw_slave_queue_remove function, ends the wait event, and calls kcrfw_redo_write, just like the master log writer process would call in single log writer mode, which includes doing the write (kcrfw_do_write) and posting the foregrounds (kslpslf), exactly all the functions.

Conclusion.
The adaptive scalable log writer processes function has been silently introduced with Oracle 12, although a lot of the used functionality has been available more or less in earlier versions. It is a fully automatic feature which will turn itself on and off based on heuristics. The purpose of this article is to explain how it works and what it is doing. Essentially, all the functionality that surrounds a log writer write has been moved to a worker process, which means the work can be done in parallel with multiple processes, whilst all the work outside of the work around the write, which is not performance critical, is left with the master log writer.

%d bloggers like this: