Archive

Tag Archives: oracle

There are a couple of underscore parameters changed from spare to named ones.
It’s interesting to see that in sysstat, ‘spare statistic 2’ changed to ‘cell XT granule IO bytes saved by HDFS tbs extent map scan’. This obviously has to do with big data access via cell servers. What is weird is that this is the only version where this had happened.

parameters unique in version 12.2.0.1.190416 versus 12.2.0.1.190716

NAME
--------------------------------------------------
_eleventh_spare_parameter
_one-hundred-and-forty-seventh_spare_parameter
_one-hundred-and-forty-sixth_spare_parameter
_tenth_spare_parameter

parameters unique in version 12.2.0.1.190716 versus 12.2.0.1.190416

NAME
--------------------------------------------------
_bug27693416_kgh_free_list_min_effort
_cell_offload_hybrid_processing
_dskm_single_instance
_seventh_spare_parameter

parameter values changed isdefault between 12.2.0.1.190416 versus 12.2.0.1.190716

parameter values unique to 12.2.0.1.190416 versus 12.2.0.1.190716

parameter values unique to 12.2.0.1.190716 versus 12.2.0.1.190416

waitevents unique in version 12.2.0.1.190416 versus 12.2.0.1.190716

waitevents unique in version 12.2.0.1.190716 versus 12.2.0.1.190416

waitevents changed parameter description between 12.2.0.1.190416 versus 12.2.0.1.190716

x$ tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

x$ tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

x$ tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

x$ tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

v$ tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

v$ tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

v$ tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

v$ tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

gv$ tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

gv$ tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

gv$ tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

gv$ tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

sysstat statistics unique to 12.2.0.1.190416 versus 12.2.0.1.190716

NAME
----------------------------------------------------------------------------------------------------
spare statistic 2

sysstat statistics unique to 12.2.0.1.190716 versus 12.2.0.1.190416

NAME
----------------------------------------------------------------------------------------------------
cell XT granule IO bytes saved by HDFS tbs extent map scan

sys_time_model statistics unique to 12.2.0.1.190416 versus 12.2.0.1.190716

sys_time_model statistics unique to 12.2.0.1.190716 versus 12.2.0.1.190416

dba tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

dba tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

dba tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

dba tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

cdb tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

cdb tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

cdb tables column unique to 12.2.0.1.190416 versus 12.2.0.1.190716

cdb tables column unique to 12.2.0.1.190716 versus 12.2.0.1.190416

And here are the differences in symbols (c functions).
Most notable:
– a lot of functions have been added that seem to be part of ‘little cms’, which seems to be a color management suite. I have no idea currently what these are doing in the oracle database executable.
– a significant amount of kola* (kernel objects lob) and prs* (parse) related functions have been added.

code symbol names unique in version 12.2.0.1.190416 versus 12.2.0.1.190716

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
free_ex_data                                                 free_ex_data                                                 ??
kcrfw_alfs_use_polling                                       (kcrfw_alfs)_use_polling                                     kernel cache recovery file write/broadcast SCN adaptive log file sync ??
kdmoBuildOperand                                             (kdmo)BuildOperand                                           kernel data in-memory data layer optimizer ??
kdmoDecompAvgCostPerSeg                                      (kdmo)DecompAvgCostPerSeg                                    kernel data in-memory data layer optimizer ??
kdmoDecompGetRate                                            (kdmo)DecompGetRate                                          kernel data in-memory data layer optimizer ??
kdmoGetStatHT                                                (kdmo)GetStatHT                                              kernel data in-memory data layer optimizer ??
kdmoIsValidTableCol                                          (kdmo)IsValidTableCol                                        kernel data in-memory data layer optimizer ??
kdmoMinMaxCountPrunableCUsSeg                                (kdmo)MinMaxCountPrunableCUsSeg                              kernel data in-memory data layer optimizer ??
kdmoMinMaxProcessPreds                                       (kdmo)MinMaxProcessPreds                                     kernel data in-memory data layer optimizer ??
kdmoPredEvalAvgCostPerSeg                                    (kdmo)PredEvalAvgCostPerSeg                                  kernel data in-memory data layer optimizer ??
kdmoPredEvalGetRate                                          (kdmo)PredEvalGetRate                                        kernel data in-memory data layer optimizer ??
kdpSizeOfGbyKdst                                             (kdp)SizeOfGbyKdst                                           kernel data archive compression: pcode ??
kdp_op_is_char                                               (kdp)_op_is_char                                             kernel data archive compression: pcode ??
kdp_pcode_calc_constant_area                                 (kdp)_pcode_calc_constant_area                               kernel data archive compression: pcode ??
kdp_pcode_setup_cbks                                         (kdp)_pcode_setup_cbks                                       kernel data archive compression: pcode ??
kdzdcol_agg_cols_imc_dict_dgk_update                         (kdzd)col_agg_cols_imc_dict_dgk_update                       kernel data archive compression decompression ??
kdzdcol_gby_dgk_create                                       (kdzd)col_gby_dgk_create                                     kernel data archive compression decompression ??
kdzdcol_gby_find_groups                                      (kdzd)col_gby_find_groups                                    kernel data archive compression decompression ??
kewmcomprmdif                                                (kewm)comprmdif                                              kernel event AWR metrics ??
kkesrc                                                       (kke)src                                                     kernel compile cost engine ??
kkfdGetQueReason                                             (kkfd)GetQueReason                                           kernel compile fast dataflow (PQ DFO) ??
kkqgCompareTTWAndWoGby                                       (kkq)gCompareTTWAndWoGby                                     kernel compile query  ??
kkqgMsqAggAppNampOrArgp                                      (kkq)gMsqAggAppNampOrArgp                                    kernel compile query  ??
kkqgMsqUnprs                                                 (kkq)gMsqUnprs                                               kernel compile query  ??
kkqjfCopyFroNoChn                                            (kkqjf)CopyFroNoChn                                          kernel compile query  join analysis join factorization ??
kkqvtOpnInView                                               (kkqvt)OpnInView                                             kernel compile query  vector transformation ??
krbr3sf                                                      (krbr)3sf                                                    kernel redo backup/restore restore and recovery ??
ksunetstat                                                   (ksu)netstat                                                 kernel service (VOS) user ??
ksws_map_fo_restore_to_text                                  (ksws)_map_fo_restore_to_text                                kernel service (VOS) workgroup services ??
kubsxiFetchOpenSetCpx                                        (ku)bsxiFetchOpenSetCpx                                      kernel utility ??
kzvdvechk_ownerid                                            (kzvd)vechk_ownerid                                          kernel security data vault ??
qerxaSetXVMCtx                                               (qer)xaSetXVMCtx                                             query execute rowsource ??
qkexrXC_SetOptTyp                                            (qke)xrXC_SetOptTyp                                          query kernel expressions ??
qksbgUnderOFE                                                (qksbg)UnderOFE                                              query kernel sql bind (variable) management(?) ??
qmxqcSetUDFVar2Parm                                          (qmxq)cSetUDFVar2Parm                                        query XDB XML Objects XML ??
qmxqcTreeAplyOp                                              (qmxq)cTreeAplyOp                                            query XDB XML Objects XML ??
qmxqcXPathNeedInputRetNodeRef                                (qmxq)cXPathNeedInputRetNodeRef                              query XDB XML Objects XML ??
qmxqtcPathHasPred                                            (qmxq)tcPathHasPred                                          query XDB XML Objects XML ??
qmxtr2IsFroRefdInFUP                                         (qmxt)r2IsFroRefdInFUP                                       query XDB XML Objects XML ??
qmxtr2MrkGrpChains                                           (qmxt)r2MrkGrpChains                                         query XDB XML Objects XML ??
set_dh                                                       set_dh                                                       ??

code symbol names unique in version 12.2.0.1.190716 versus 12.2.0.1.190416

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
AddConversion                                                AddConversion                                                ??
AllocArray                                                   AllocArray                                                   ??
AllocEmptyTransform                                          AllocEmptyTransform                                          ??
AllocateToneCurveStruct                                      AllocateToneCurveStruct                                      ??
BilinearInterp16                                             BilinearInterp16                                             ??
BilinearInterpFloat                                          BilinearInterpFloat                                          ??
BlackPreservingGrayOnlySampler                               BlackPreservingGrayOnlySampler                               ??
BlackPreservingKOnlyIntents                                  BlackPreservingKOnlyIntents                                  ??
BlackPreservingKPlaneIntents                                 BlackPreservingKPlaneIntents                                 ??
BlackPreservingSampler                                       BlackPreservingSampler                                       ??
CLUTElemDup                                                  CLUTElemDup                                                  ??
CLutElemTypeFree                                             CLutElemTypeFree                                             ??
CachedXFORM                                                  CachedXFORM                                                  ??
CachedXFORMGamutCheck                                        CachedXFORMGamutCheck                                        ??
Clipper                                                      Clipper                                                      ??
ComputeAbsoluteIntent                                        ComputeAbsoluteIntent                                        ??
CurveSetDup                                                  CurveSetDup                                                  ??
CurveSetElemTypeFree                                         CurveSetElemTypeFree                                         ??
CurvesDup                                                    CurvesDup                                                    ??
CurvesFree                                                   CurvesFree                                                   ??
DecideCurveType                                              DecideCurveType                                              ??
DecideLUTtypeA2B                                             DecideLUTtypeA2B                                             ??
DecideLUTtypeB2A                                             DecideLUTtypeB2A                                             ??
DecideTextDescType                                           DecideTextDescType                                           ??
DecideTextType                                               DecideTextType                                               ??
DecideXYZtype                                                DecideXYZtype                                                ??
DefaultEvalParametricFn                                      DefaultEvalParametricFn                                      ??
DefaultICCintents                                            DefaultICCintents                                            ??
DefaultLogErrorHandlerFunction                               DefaultLogErrorHandlerFunction                               ??
DupMatShaper                                                 DupMatShaper                                                 ??
DupNamedColorList                                            DupNamedColorList                                            ??
EstimateTAC                                                  EstimateTAC                                                  ??
Eval16nop1D                                                  Eval16nop1D                                                  ??
Eval1Input                                                   Eval1Input                                                   ??
Eval1InputFloat                                              Eval1InputFloat                                              ??
Eval4Inputs                                                  Eval4Inputs                                                  ??
Eval4InputsFloat                                             Eval4InputsFloat                                             ??
Eval5Inputs                                                  Eval5Inputs                                                  ??
Eval5InputsFloat                                             Eval5InputsFloat                                             ??
Eval6Inputs                                                  Eval6Inputs                                                  ??
Eval6InputsFloat                                             Eval6InputsFloat                                             ??
Eval7Inputs                                                  Eval7Inputs                                                  ??
Eval7InputsFloat                                             Eval7InputsFloat                                             ??
Eval8Inputs                                                  Eval8Inputs                                                  ??
Eval8InputsFloat                                             Eval8InputsFloat                                             ??
EvalNamedColor                                               EvalNamedColor                                               ??
EvalNamedColorPCS                                            EvalNamedColorPCS                                            ??
EvaluateCLUTfloat                                            EvaluateCLUTfloat                                            ??
EvaluateCLUTfloatIn16                                        EvaluateCLUTfloatIn16                                        ??
EvaluateCurves                                               EvaluateCurves                                               ??
EvaluateIdentity                                             EvaluateIdentity                                             ??
EvaluateLab2XYZ                                              EvaluateLab2XYZ                                              ??
EvaluateMatrix                                               EvaluateMatrix                                               ??
EvaluateXYZ2Lab                                              EvaluateXYZ2Lab                                              ??
FastEvaluateCurves16                                         FastEvaluateCurves16                                         ??
FastEvaluateCurves8                                          FastEvaluateCurves8                                          ??
FastIdentity16                                               FastIdentity16                                               ??
FileClose                                                    FileClose                                                    ??
FileRead                                                     FileRead                                                     ??
FileSeek                                                     FileSeek                                                     ??
FileTell                                                     FileTell                                                     ??
FileWrite                                                    FileWrite                                                    ??
FixWhiteMisalignment                                         FixWhiteMisalignment                                         ??
FloatXFORM                                                   FloatXFORM                                                   ??
FreeMatShaper                                                FreeMatShaper                                                ??
FreeNamedColorList                                           FreeNamedColorList                                           ??
GamutSampler                                                 GamutSampler                                                 ??
GenericMPEdup                                                GenericMPEdup                                                ??
GenericMPEfree                                               GenericMPEfree                                               ??
IdentitySampler                                              IdentitySampler                                              ??
InkLimitingSampler                                           InkLimitingSampler                                           ??
Java_sun_java2d_cmm_lcms_LCMS_colorConvert                   Java_sun_java2d_cmm_lcms_LCMS_colorConvert                   ??
Java_sun_java2d_cmm_lcms_LCMS_createNativeTransform          Java_sun_java2d_cmm_lcms_LCMS_createNativeTransform          ??
Java_sun_java2d_cmm_lcms_LCMS_getProfileDataNative           Java_sun_java2d_cmm_lcms_LCMS_getProfileDataNative           ??
Java_sun_java2d_cmm_lcms_LCMS_getProfileID                   Java_sun_java2d_cmm_lcms_LCMS_getProfileID                   ??
Java_sun_java2d_cmm_lcms_LCMS_getProfileSizeNative           Java_sun_java2d_cmm_lcms_LCMS_getProfileSizeNative           ??
Java_sun_java2d_cmm_lcms_LCMS_getTagNative                   Java_sun_java2d_cmm_lcms_LCMS_getTagNative                   ??
Java_sun_java2d_cmm_lcms_LCMS_initLCMS                       Java_sun_java2d_cmm_lcms_LCMS_initLCMS                       ??
Java_sun_java2d_cmm_lcms_LCMS_loadProfileNative              Java_sun_java2d_cmm_lcms_LCMS_loadProfileNative              ??
Java_sun_java2d_cmm_lcms_LCMS_setTagDataNative               Java_sun_java2d_cmm_lcms_LCMS_setTagDataNative               ??
LCMS_freeProfile                                             LCMS_freeProfile                                             ??
LCMS_freeTransform                                           LCMS_freeTransform                                           ??
LinLerp1D                                                    LinLerp1D                                                    ??
LinLerp1Dfloat                                               LinLerp1Dfloat                                               ??
MatShaperEval16                                              MatShaperEval16                                              ??
MatrixElemDup                                                MatrixElemDup                                                ??
MatrixElemTypeFree                                           MatrixElemTypeFree                                           ??
MemoryClose                                                  MemoryClose                                                  ??
MemoryRead                                                   MemoryRead                                                   ??
MemorySeek                                                   MemorySeek                                                   ??
MemoryTell                                                   MemoryTell                                                   ??
MemoryWrite                                                  MemoryWrite                                                  ??
NULLClose                                                    NULLClose                                                    ??
NULLRead                                                     NULLRead                                                     ??
NULLSeek                                                     NULLSeek                                                     ??
NULLTell                                                     NULLTell                                                     ??
NULLWrite                                                    NULLWrite                                                    ??
NullFloatXFORM                                               NullFloatXFORM                                               ??
NullXFORM                                                    NullXFORM                                                    ??
OptimizeByComputingLinearization                             OptimizeByComputingLinearization                             ??
OptimizeByJoiningCurves                                      OptimizeByJoiningCurves                                      ??
OptimizeByResampling                                         OptimizeByResampling                                         ??
OptimizeMatrixShaper                                         OptimizeMatrixShaper                                         ??
Pack1Byte                                                    Pack1Byte                                                    ??
Pack1ByteReversed                                            Pack1ByteReversed                                            ??
Pack1ByteSkip1                                               Pack1ByteSkip1                                               ??
Pack1ByteSkip1SwapFirst                                      Pack1ByteSkip1SwapFirst                                      ??
Pack1Word                                                    Pack1Word                                                    ??
Pack1WordBigEndian                                           Pack1WordBigEndian                                           ??
Pack1WordReversed                                            Pack1WordReversed                                            ??
Pack1WordSkip1                                               Pack1WordSkip1                                               ??
Pack1WordSkip1SwapFirst                                      Pack1WordSkip1SwapFirst                                      ??
Pack3Bytes                                                   Pack3Bytes                                                   ??
Pack3BytesAndSkip1                                           Pack3BytesAndSkip1                                           ??
Pack3BytesAndSkip1Optimized                                  Pack3BytesAndSkip1Optimized                                  ??
Pack3BytesAndSkip1Swap                                       Pack3BytesAndSkip1Swap                                       ??
Pack3BytesAndSkip1SwapFirst                                  Pack3BytesAndSkip1SwapFirst                                  ??
Pack3BytesAndSkip1SwapFirstOptimized                         Pack3BytesAndSkip1SwapFirstOptimized                         ??
Pack3BytesAndSkip1SwapOptimized                              Pack3BytesAndSkip1SwapOptimized                              ??
Pack3BytesAndSkip1SwapSwapFirst                              Pack3BytesAndSkip1SwapSwapFirst                              ??
Pack3BytesAndSkip1SwapSwapFirstOptimized                     Pack3BytesAndSkip1SwapSwapFirstOptimized                     ??
Pack3BytesOptimized                                          Pack3BytesOptimized                                          ??
Pack3BytesSwap                                               Pack3BytesSwap                                               ??
Pack3BytesSwapOptimized                                      Pack3BytesSwapOptimized                                      ??
Pack3Words                                                   Pack3Words                                                   ??
Pack3WordsAndSkip1                                           Pack3WordsAndSkip1                                           ??
Pack3WordsAndSkip1Swap                                       Pack3WordsAndSkip1Swap                                       ??
Pack3WordsAndSkip1SwapFirst                                  Pack3WordsAndSkip1SwapFirst                                  ??
Pack3WordsAndSkip1SwapSwapFirst                              Pack3WordsAndSkip1SwapSwapFirst                              ??
Pack3WordsBigEndian                                          Pack3WordsBigEndian                                          ??
Pack3WordsSwap                                               Pack3WordsSwap                                               ??
Pack4Bytes                                                   Pack4Bytes                                                   ??
Pack4BytesReverse                                            Pack4BytesReverse                                            ??
Pack4BytesSwap                                               Pack4BytesSwap                                               ??
Pack4BytesSwapFirst                                          Pack4BytesSwapFirst                                          ??
Pack4BytesSwapSwapFirst                                      Pack4BytesSwapSwapFirst                                      ??
Pack4Words                                                   Pack4Words                                                   ??
Pack4WordsBigEndian                                          Pack4WordsBigEndian                                          ??
Pack4WordsReverse                                            Pack4WordsReverse                                            ??
Pack4WordsSwap                                               Pack4WordsSwap                                               ??
Pack6Bytes                                                   Pack6Bytes                                                   ??
Pack6BytesSwap                                               Pack6BytesSwap                                               ??
Pack6Words                                                   Pack6Words                                                   ??
Pack6WordsSwap                                               Pack6WordsSwap                                               ??
PackALabV2_8                                                 PackALabV2_8                                                 ??
PackAnyBytes                                                 PackAnyBytes                                                 ??
PackAnyWords                                                 PackAnyWords                                                 ??
PackDoubleFrom16                                             PackDoubleFrom16                                             ??
PackDoublesFromFloat                                         PackDoublesFromFloat                                         ??
PackFloatFrom16                                              PackFloatFrom16                                              ??
PackFloatsFromFloat                                          PackFloatsFromFloat                                          ??
PackHalfFrom16                                               PackHalfFrom16                                               ??
PackHalfFromFloat                                            PackHalfFromFloat                                            ??
PackLabDoubleFrom16                                          PackLabDoubleFrom16                                          ??
PackLabDoubleFromFloat                                       PackLabDoubleFromFloat                                       ??
PackLabFloatFrom16                                           PackLabFloatFrom16                                           ??
PackLabFloatFromFloat                                        PackLabFloatFromFloat                                        ??
PackLabV2_16                                                 PackLabV2_16                                                 ??
PackLabV2_8                                                  PackLabV2_8                                                  ??
PackPlanarBytes                                              PackPlanarBytes                                              ??
PackPlanarWords                                              PackPlanarWords                                              ??
PackXYZDoubleFrom16                                          PackXYZDoubleFrom16                                          ??
PackXYZDoubleFromFloat                                       PackXYZDoubleFromFloat                                       ??
PackXYZFloatFrom16                                           PackXYZFloatFrom16                                           ??
PackXYZFloatFromFloat                                        PackXYZFloatFromFloat                                        ??
PreOptimize                                                  PreOptimize                                                  ??
PrecalculatedXFORM                                           PrecalculatedXFORM                                           ??
PrecalculatedXFORMGamutCheck                                 PrecalculatedXFORMGamutCheck                                 ??
Prelin16dup                                                  Prelin16dup                                                  ??
Prelin8dup                                                   Prelin8dup                                                   ??
Prelin8free                                                  Prelin8free                                                  ??
PrelinEval16                                                 PrelinEval16                                                 ??
PrelinEval8                                                  PrelinEval8                                                  ??
PrelinOpt16free                                              PrelinOpt16free                                              ??
R_SSL_clear_options                                          R_SSL_clear_options                                          ??
R_SSL_clear_options_by_type                                  R_SSL_clear_options_by_type                                  ??
ReadMPECurve                                                 ReadMPECurve                                                 ??
ReadMPEElem                                                  ReadMPEElem                                                  ??
ReadSeqID                                                    ReadSeqID                                                    ??
ReadSetOfCurves                                              ReadSetOfCurves                                              ??
SaveTags                                                     SaveTags                                                     ??
TetrahedralInterp16                                          TetrahedralInterp16                                          ??
TetrahedralInterpFloat                                       TetrahedralInterpFloat                                       ??
TrilinearInterp16                                            TrilinearInterp16                                            ??
TrilinearInterpFloat                                         TrilinearInterpFloat                                         ??
Type_Chromaticity_Dup                                        Type_Chromaticity_Dup                                        ??
Type_Chromaticity_Free                                       Type_Chromaticity_Free                                       ??
Type_Chromaticity_Read                                       Type_Chromaticity_Read                                       ??
Type_Chromaticity_Write                                      Type_Chromaticity_Write                                      ??
Type_ColorantOrderType_Dup                                   Type_ColorantOrderType_Dup                                   ??
Type_ColorantOrderType_Free                                  Type_ColorantOrderType_Free                                  ??
Type_ColorantOrderType_Read                                  Type_ColorantOrderType_Read                                  ??
Type_ColorantOrderType_Write                                 Type_ColorantOrderType_Write                                 ??
Type_ColorantTable_Dup                                       Type_ColorantTable_Dup                                       ??
Type_ColorantTable_Free                                      Type_ColorantTable_Free                                      ??
Type_ColorantTable_Read                                      Type_ColorantTable_Read                                      ??
Type_ColorantTable_Write                                     Type_ColorantTable_Write                                     ??
Type_CrdInfo_Dup                                             Type_CrdInfo_Dup                                             ??
Type_CrdInfo_Free                                            Type_CrdInfo_Free                                            ??
Type_CrdInfo_Read                                            Type_CrdInfo_Read                                            ??
Type_CrdInfo_Write                                           Type_CrdInfo_Write                                           ??
Type_Curve_Dup                                               Type_Curve_Dup                                               ??
Type_Curve_Free                                              Type_Curve_Free                                              ??
Type_Curve_Read                                              Type_Curve_Read                                              ??
Type_Curve_Write                                             Type_Curve_Write                                             ??
Type_Data_Dup                                                Type_Data_Dup                                                ??
Type_Data_Free                                               Type_Data_Free                                               ??
Type_Data_Read                                               Type_Data_Read                                               ??
Type_Data_Write                                              Type_Data_Write                                              ??
Type_DateTime_Dup                                            Type_DateTime_Dup                                            ??
Type_DateTime_Free                                           Type_DateTime_Free                                           ??
Type_DateTime_Read                                           Type_DateTime_Read                                           ??
Type_DateTime_Write                                          Type_DateTime_Write                                          ??
Type_Dictionary_Dup                                          Type_Dictionary_Dup                                          ??
Type_Dictionary_Free                                         Type_Dictionary_Free                                         ??
Type_Dictionary_Read                                         Type_Dictionary_Read                                         ??
Type_Dictionary_Write                                        Type_Dictionary_Write                                        ??
Type_LUT16_Dup                                               Type_LUT16_Dup                                               ??
Type_LUT16_Free                                              Type_LUT16_Free                                              ??
Type_LUT16_Read                                              Type_LUT16_Read                                              ??
Type_LUT16_Write                                             Type_LUT16_Write                                             ??
Type_LUT8_Dup                                                Type_LUT8_Dup                                                ??
Type_LUT8_Free                                               Type_LUT8_Free                                               ??
Type_LUT8_Read                                               Type_LUT8_Read                                               ??
Type_LUT8_Write                                              Type_LUT8_Write                                              ??
Type_LUTA2B_Dup                                              Type_LUTA2B_Dup                                              ??
Type_LUTA2B_Free                                             Type_LUTA2B_Free                                             ??
Type_LUTA2B_Read                                             Type_LUTA2B_Read                                             ??
Type_LUTA2B_Write                                            Type_LUTA2B_Write                                            ??
Type_LUTB2A_Dup                                              Type_LUTB2A_Dup                                              ??
Type_LUTB2A_Free                                             Type_LUTB2A_Free                                             ??
Type_LUTB2A_Read                                             Type_LUTB2A_Read                                             ??
Type_LUTB2A_Write                                            Type_LUTB2A_Write                                            ??
Type_MLU_Dup                                                 Type_MLU_Dup                                                 ??
Type_MLU_Free                                                Type_MLU_Free                                                ??
Type_MLU_Read                                                Type_MLU_Read                                                ??
Type_MLU_Write                                               Type_MLU_Write                                               ??
Type_MPE_Dup                                                 Type_MPE_Dup                                                 ??
Type_MPE_Free                                                Type_MPE_Free                                                ??
Type_MPE_Read                                                Type_MPE_Read                                                ??
Type_MPE_Write                                               Type_MPE_Write                                               ??
Type_MPEclut_Read                                            Type_MPEclut_Read                                            ??
Type_MPEclut_Write                                           Type_MPEclut_Write                                           ??
Type_MPEcurve_Read                                           Type_MPEcurve_Read                                           ??
Type_MPEcurve_Write                                          Type_MPEcurve_Write                                          ??
Type_MPEmatrix_Read                                          Type_MPEmatrix_Read                                          ??
Type_MPEmatrix_Write                                         Type_MPEmatrix_Write                                         ??
Type_Measurement_Dup                                         Type_Measurement_Dup                                         ??
Type_Measurement_Free                                        Type_Measurement_Free                                        ??
Type_Measurement_Read                                        Type_Measurement_Read                                        ??
Type_Measurement_Write                                       Type_Measurement_Write                                       ??
Type_NamedColor_Dup                                          Type_NamedColor_Dup                                          ??
Type_NamedColor_Free                                         Type_NamedColor_Free                                         ??
Type_NamedColor_Read                                         Type_NamedColor_Read                                         ??
Type_NamedColor_Write                                        Type_NamedColor_Write                                        ??
Type_ParametricCurve_Dup                                     Type_ParametricCurve_Dup                                     ??
Type_ParametricCurve_Free                                    Type_ParametricCurve_Free                                    ??
Type_ParametricCurve_Read                                    Type_ParametricCurve_Read                                    ??
Type_ParametricCurve_Write                                   Type_ParametricCurve_Write                                   ??
Type_ProfileSequenceDesc_Dup                                 Type_ProfileSequenceDesc_Dup                                 ??
Type_ProfileSequenceDesc_Free                                Type_ProfileSequenceDesc_Free                                ??
Type_ProfileSequenceDesc_Read                                Type_ProfileSequenceDesc_Read                                ??
Type_ProfileSequenceDesc_Write                               Type_ProfileSequenceDesc_Write                               ??
Type_ProfileSequenceId_Dup                                   Type_ProfileSequenceId_Dup                                   ??
Type_ProfileSequenceId_Free                                  Type_ProfileSequenceId_Free                                  ??
Type_ProfileSequenceId_Read                                  Type_ProfileSequenceId_Read                                  ??
Type_ProfileSequenceId_Write                                 Type_ProfileSequenceId_Write                                 ??
Type_S15Fixed16_Dup                                          Type_S15Fixed16_Dup                                          ??
Type_S15Fixed16_Free                                         Type_S15Fixed16_Free                                         ??
Type_S15Fixed16_Read                                         Type_S15Fixed16_Read                                         ??
Type_S15Fixed16_Write                                        Type_S15Fixed16_Write                                        ??
Type_Screening_Dup                                           Type_Screening_Dup                                           ??
Type_Screening_Free                                          Type_Screening_Free                                          ??
Type_Screening_Read                                          Type_Screening_Read                                          ??
Type_Screening_Write                                         Type_Screening_Write                                         ??
Type_Signature_Dup                                           Type_Signature_Dup                                           ??
Type_Signature_Free                                          Type_Signature_Free                                          ??
Type_Signature_Read                                          Type_Signature_Read                                          ??
Type_Signature_Write                                         Type_Signature_Write                                         ??
Type_Text_Description_Dup                                    Type_Text_Description_Dup                                    ??
Type_Text_Description_Free                                   Type_Text_Description_Free                                   ??
Type_Text_Description_Read                                   Type_Text_Description_Read                                   ??
Type_Text_Description_Write                                  Type_Text_Description_Write                                  ??
Type_Text_Dup                                                Type_Text_Dup                                                ??
Type_Text_Free                                               Type_Text_Free                                               ??
Type_Text_Read                                               Type_Text_Read                                               ??
Type_Text_Write                                              Type_Text_Write                                              ??
Type_U16Fixed16_Dup                                          Type_U16Fixed16_Dup                                          ??
Type_U16Fixed16_Free                                         Type_U16Fixed16_Free                                         ??
Type_U16Fixed16_Read                                         Type_U16Fixed16_Read                                         ??
Type_U16Fixed16_Write                                        Type_U16Fixed16_Write                                        ??
Type_UcrBg_Dup                                               Type_UcrBg_Dup                                               ??
Type_UcrBg_Free                                              Type_UcrBg_Free                                              ??
Type_UcrBg_Read                                              Type_UcrBg_Read                                              ??
Type_UcrBg_Write                                             Type_UcrBg_Write                                             ??
Type_ViewingConditions_Dup                                   Type_ViewingConditions_Dup                                   ??
Type_ViewingConditions_Free                                  Type_ViewingConditions_Free                                  ??
Type_ViewingConditions_Read                                  Type_ViewingConditions_Read                                  ??
Type_ViewingConditions_Write                                 Type_ViewingConditions_Write                                 ??
Type_XYZ_Dup                                                 Type_XYZ_Dup                                                 ??
Type_XYZ_Free                                                Type_XYZ_Free                                                ??
Type_XYZ_Read                                                Type_XYZ_Read                                                ??
Type_XYZ_Write                                               Type_XYZ_Write                                               ??
Type_vcgt_Dup                                                Type_vcgt_Dup                                                ??
Type_vcgt_Free                                               Type_vcgt_Free                                               ??
Type_vcgt_Read                                               Type_vcgt_Read                                               ??
Type_vcgt_Write                                              Type_vcgt_Write                                              ??
Unroll1Byte                                                  Unroll1Byte                                                  ??
Unroll1ByteReversed                                          Unroll1ByteReversed                                          ??
Unroll1ByteSkip1                                             Unroll1ByteSkip1                                             ??
Unroll1ByteSkip2                                             Unroll1ByteSkip2                                             ??
Unroll1Word                                                  Unroll1Word                                                  ??
Unroll1WordReversed                                          Unroll1WordReversed                                          ??
Unroll1WordSkip3                                             Unroll1WordSkip3                                             ??
Unroll2Bytes                                                 Unroll2Bytes                                                 ??
Unroll2Words                                                 Unroll2Words                                                 ??
Unroll3Bytes                                                 Unroll3Bytes                                                 ??
Unroll3BytesSkip1Swap                                        Unroll3BytesSkip1Swap                                        ??
Unroll3BytesSkip1SwapFirst                                   Unroll3BytesSkip1SwapFirst                                   ??
Unroll3BytesSkip1SwapSwapFirst                               Unroll3BytesSkip1SwapSwapFirst                               ??
Unroll3BytesSwap                                             Unroll3BytesSwap                                             ??
Unroll3Words                                                 Unroll3Words                                                 ??
Unroll3WordsSkip1Swap                                        Unroll3WordsSkip1Swap                                        ??
Unroll3WordsSkip1SwapFirst                                   Unroll3WordsSkip1SwapFirst                                   ??
Unroll3WordsSwap                                             Unroll3WordsSwap                                             ??
Unroll4Bytes                                                 Unroll4Bytes                                                 ??
Unroll4BytesReverse                                          Unroll4BytesReverse                                          ??
Unroll4BytesSwap                                             Unroll4BytesSwap                                             ??
Unroll4BytesSwapFirst                                        Unroll4BytesSwapFirst                                        ??
Unroll4BytesSwapSwapFirst                                    Unroll4BytesSwapSwapFirst                                    ??
Unroll4Words                                                 Unroll4Words                                                 ??
Unroll4WordsReverse                                          Unroll4WordsReverse                                          ??
Unroll4WordsSwap                                             Unroll4WordsSwap                                             ??
Unroll4WordsSwapFirst                                        Unroll4WordsSwapFirst                                        ??
Unroll4WordsSwapSwapFirst                                    Unroll4WordsSwapSwapFirst                                    ??
UnrollALabV2_8                                               UnrollALabV2_8                                               ??
UnrollAnyWords                                               UnrollAnyWords                                               ??
UnrollChunkyBytes                                            UnrollChunkyBytes                                            ??
UnrollDouble1Chan                                            UnrollDouble1Chan                                            ??
UnrollDoubleTo16                                             UnrollDoubleTo16                                             ??
UnrollDoublesToFloat                                         UnrollDoublesToFloat                                         ??
UnrollFloatTo16                                              UnrollFloatTo16                                              ??
UnrollFloatsToFloat                                          UnrollFloatsToFloat                                          ??
UnrollHalfTo16                                               UnrollHalfTo16                                               ??
UnrollHalfToFloat                                            UnrollHalfToFloat                                            ??
UnrollLabDoubleTo16                                          UnrollLabDoubleTo16                                          ??
UnrollLabDoubleToFloat                                       UnrollLabDoubleToFloat                                       ??
UnrollLabFloatTo16                                           UnrollLabFloatTo16                                           ??
UnrollLabFloatToFloat                                        UnrollLabFloatToFloat                                        ??
UnrollLabV2_16                                               UnrollLabV2_16                                               ??
UnrollLabV2_8                                                UnrollLabV2_8                                                ??
UnrollPlanarBytes                                            UnrollPlanarBytes                                            ??
UnrollPlanarWords                                            UnrollPlanarWords                                            ??
UnrollXYZDoubleTo16                                          UnrollXYZDoubleTo16                                          ??
UnrollXYZDoubleToFloat                                       UnrollXYZDoubleToFloat                                       ??
UnrollXYZFloatTo16                                           UnrollXYZFloatTo16                                           ??
UnrollXYZFloatToFloat                                        UnrollXYZFloatToFloat                                        ??
WriteMPECurve                                                WriteMPECurve                                                ??
WriteSeqID                                                   WriteSeqID                                                   ??
WriteSetOfCurves                                             WriteSetOfCurves                                             ??
XFormSampler16                                               XFormSampler16                                               ??
_LUTeval16                                                   _LUTeval16                                                   ??
_LUTevalFloat                                                _LUTevalFloat                                                ??
_cms15Fixed16toDouble                                        (_cms)15Fixed16toDouble                                      color management system (little cms) ??
_cms8Fixed8toDouble                                          (_cms)8Fixed8toDouble                                        color management system (little cms) ??
_cmsAdaptationMatrix                                         (_cms)AdaptationMatrix                                       color management system (little cms) ??
_cmsAdjustEndianess16                                        (_cms)AdjustEndianess16                                      color management system (little cms) ??
_cmsAdjustEndianess32                                        (_cms)AdjustEndianess32                                      color management system (little cms) ??
_cmsAdjustEndianess64                                        (_cms)AdjustEndianess64                                      color management system (little cms) ??
_cmsAllocAdaptationStateChunk                                (_cms)AllocAdaptationStateChunk                              color management system (little cms) ??
_cmsAllocAlarmCodesChunk                                     (_cms)AllocAlarmCodesChunk                                   color management system (little cms) ??
_cmsAllocCurvesPluginChunk                                   (_cms)AllocCurvesPluginChunk                                 color management system (little cms) ??
_cmsAllocFormattersPluginChunk                               (_cms)AllocFormattersPluginChunk                             color management system (little cms) ??
_cmsAllocIntentsPluginChunk                                  (_cms)AllocIntentsPluginChunk                                color management system (little cms) ??
_cmsAllocInterpPluginChunk                                   (_cms)AllocInterpPluginChunk                                 color management system (little cms) ??
_cmsAllocLogErrorChunk                                       (_cms)AllocLogErrorChunk                                     color management system (little cms) ??
_cmsAllocMPETypePluginChunk                                  (_cms)AllocMPETypePluginChunk                                color management system (little cms) ??
_cmsAllocMemPluginChunk                                      (_cms)AllocMemPluginChunk                                    color management system (little cms) ??
_cmsAllocMutexPluginChunk                                    (_cms)AllocMutexPluginChunk                                  color management system (little cms) ??
_cmsAllocOptimizationPluginChunk                             (_cms)AllocOptimizationPluginChunk                           color management system (little cms) ??
_cmsAllocTagPluginChunk                                      (_cms)AllocTagPluginChunk                                    color management system (little cms) ??
_cmsAllocTagTypePluginChunk                                  (_cms)AllocTagTypePluginChunk                                color management system (little cms) ??
_cmsAllocTransformPluginChunk                                (_cms)AllocTransformPluginChunk                              color management system (little cms) ??
_cmsBuildKToneCurve                                          (_cms)BuildKToneCurve                                        color management system (little cms) ??
_cmsBuildRGB2XYZtransferMatrix                               (_cms)BuildRGB2XYZtransferMatrix                             color management system (little cms) ??
_cmsCalloc                                                   (_cms)Calloc                                                 color management system (little cms) ??
_cmsCallocDefaultFn                                          (_cms)CallocDefaultFn                                        color management system (little cms) ??
_cmsChain2Lab                                                (_cms)Chain2Lab                                              color management system (little cms) ??
_cmsCompileProfileSequence                                   (_cms)CompileProfileSequence                                 color management system (little cms) ??
_cmsComputeInterpParams                                      (_cms)ComputeInterpParams                                    color management system (little cms) ??
_cmsComputeInterpParamsEx                                    (_cms)ComputeInterpParamsEx                                  color management system (little cms) ??
_cmsContextGetClientChunk                                    (_cms)ContextGetClientChunk                                  color management system (little cms) ??
_cmsCreateGamutCheckPipeline                                 (_cms)CreateGamutCheckPipeline                               color management system (little cms) ??
_cmsCreateMutex                                              (_cms)CreateMutex                                            color management system (little cms) ??
_cmsCreateSubAlloc                                           (_cms)CreateSubAlloc                                         color management system (little cms) ??
_cmsDecodeDateTimeNumber                                     (_cms)DecodeDateTimeNumber                                   color management system (little cms) ??
_cmsDefaultICCintents                                        (_cms)DefaultICCintents                                      color management system (little cms) ??
_cmsDestroyMutex                                             (_cms)DestroyMutex                                           color management system (little cms) ??
_cmsDoubleTo15Fixed16                                        (_cms)DoubleTo15Fixed16                                      color management system (little cms) ??
_cmsDoubleTo8Fixed8                                          (_cms)DoubleTo8Fixed8                                        color management system (little cms) ??
_cmsDupDefaultFn                                             (_cms)DupDefaultFn                                           color management system (little cms) ??
_cmsDupMem                                                   (_cms)DupMem                                                 color management system (little cms) ??
_cmsEncodeDateTimeNumber                                     (_cms)EncodeDateTimeNumber                                   color management system (little cms) ??
_cmsEndPointsBySpace                                         (_cms)EndPointsBySpace                                       color management system (little cms) ??
_cmsFloat2Half                                               (_cms)Float2Half                                             color management system (little cms) ??
_cmsFormatterIs8bit                                          (_cms)FormatterIs8bit                                        color management system (little cms) ??
_cmsFormatterIsFloat                                         (_cms)FormatterIsFloat                                       color management system (little cms) ??
_cmsFree                                                     (_cms)Free                                                   color management system (little cms) ??
_cmsFreeDefaultFn                                            (_cms)FreeDefaultFn                                          color management system (little cms) ??
_cmsFreeInterpParams                                         (_cms)FreeInterpParams                                       color management system (little cms) ??
_cmsGetContext                                               (_cms)GetContext                                             color management system (little cms) ??
_cmsGetFormatter                                             (_cms)GetFormatter                                           color management system (little cms) ??
_cmsGetTagDescriptor                                         (_cms)GetTagDescriptor                                       color management system (little cms) ??
_cmsGetTagTrueType                                           (_cms)GetTagTrueType                                         color management system (little cms) ??
_cmsGetTagTypeHandler                                        (_cms)GetTagTypeHandler                                      color management system (little cms) ??
_cmsGetTransformFormatters16                                 (_cms)GetTransformFormatters16                               color management system (little cms) ??
_cmsGetTransformFormattersFloat                              (_cms)GetTransformFormattersFloat                            color management system (little cms) ??
_cmsGetTransformUserData                                     (_cms)GetTransformUserData                                   color management system (little cms) ??
_cmsHalf2Float                                               (_cms)Half2Float                                             color management system (little cms) ??
_cmsHandleExtraChannels                                      (_cms)HandleExtraChannels                                    color management system (little cms) ??
_cmsICCcolorSpace                                            (_cms)ICCcolorSpace                                          color management system (little cms) ??
_cmsIOPrintf                                                 (_cms)IOPrintf                                               color management system (little cms) ??
_cmsInstallAllocFunctions                                    (_cms)InstallAllocFunctions                                  color management system (little cms) ??
_cmsLCMScolorSpace                                           (_cms)LCMScolorSpace                                         color management system (little cms) ??
_cmsLinkProfiles                                             (_cms)LinkProfiles                                           color management system (little cms) ??
_cmsLockMutex                                                (_cms)LockMutex                                              color management system (little cms) ??
_cmsMAT3eval                                                 (_cms)MAT3eval                                               color management system (little cms) ??
_cmsMAT3identity                                             (_cms)MAT3identity                                           color management system (little cms) ??
_cmsMAT3inverse                                              (_cms)MAT3inverse                                            color management system (little cms) ??
_cmsMAT3isIdentity                                           (_cms)MAT3isIdentity                                         color management system (little cms) ??
_cmsMAT3per                                                  (_cms)MAT3per                                                color management system (little cms) ??
_cmsMAT3solve                                                (_cms)MAT3solve                                              color management system (little cms) ??
_cmsMalloc                                                   (_cms)Malloc                                                 color management system (little cms) ??
_cmsMallocDefaultFn                                          (_cms)MallocDefaultFn                                        color management system (little cms) ??
_cmsMallocZero                                               (_cms)MallocZero                                             color management system (little cms) ??
_cmsMallocZeroDefaultFn                                      (_cms)MallocZeroDefaultFn                                    color management system (little cms) ??
_cmsOptimizePipeline                                         (_cms)OptimizePipeline                                       color management system (little cms) ??
_cmsPipelineSetOptimizationParameters                        (_cms)PipelineSetOptimizationParameters                      color management system (little cms) ??
_cmsPluginMalloc                                             (_cms)PluginMalloc                                           color management system (little cms) ??
_cmsQuantizeVal                                              (_cms)QuantizeVal                                            color management system (little cms) ??
_cmsRead15Fixed16Number                                      (_cms)Read15Fixed16Number                                    color management system (little cms) ??
_cmsReadAlignment                                            (_cms)ReadAlignment                                          color management system (little cms) ??
_cmsReadCHAD                                                 (_cms)ReadCHAD                                               color management system (little cms) ??
_cmsReadDevicelinkLUT                                        (_cms)ReadDevicelinkLUT                                      color management system (little cms) ??
_cmsReadFloat32Number                                        (_cms)ReadFloat32Number                                      color management system (little cms) ??
_cmsReadHeader                                               (_cms)ReadHeader                                             color management system (little cms) ??
_cmsReadInputLUT                                             (_cms)ReadInputLUT                                           color management system (little cms) ??
_cmsReadMediaWhitePoint                                      (_cms)ReadMediaWhitePoint                                    color management system (little cms) ??
_cmsReadOutputLUT                                            (_cms)ReadOutputLUT                                          color management system (little cms) ??
_cmsReadProfileSequence                                      (_cms)ReadProfileSequence                                    color management system (little cms) ??
_cmsReadTypeBase                                             (_cms)ReadTypeBase                                           color management system (little cms) ??
_cmsReadUInt16Array                                          (_cms)ReadUInt16Array                                        color management system (little cms) ??
_cmsReadUInt16Number                                         (_cms)ReadUInt16Number                                       color management system (little cms) ??
_cmsReadUInt32Number                                         (_cms)ReadUInt32Number                                       color management system (little cms) ??
_cmsReadUInt64Number                                         (_cms)ReadUInt64Number                                       color management system (little cms) ??
_cmsReadUInt8Number                                          (_cms)ReadUInt8Number                                        color management system (little cms) ??
_cmsReadXYZNumber                                            (_cms)ReadXYZNumber                                          color management system (little cms) ??
_cmsRealloc                                                  (_cms)Realloc                                                color management system (little cms) ??
_cmsReallocDefaultFn                                         (_cms)ReallocDefaultFn                                       color management system (little cms) ??
_cmsReasonableGridpointsByColorspace                         (_cms)ReasonableGridpointsByColorspace                       color management system (little cms) ??
_cmsRegisterFormattersPlugin                                 (_cms)RegisterFormattersPlugin                               color management system (little cms) ??
_cmsRegisterInterpPlugin                                     (_cms)RegisterInterpPlugin                                   color management system (little cms) ??
_cmsRegisterMemHandlerPlugin                                 (_cms)RegisterMemHandlerPlugin                               color management system (little cms) ??
_cmsRegisterMultiProcessElementPlugin                        (_cms)RegisterMultiProcessElementPlugin                      color management system (little cms) ??
_cmsRegisterMutexPlugin                                      (_cms)RegisterMutexPlugin                                    color management system (little cms) ??
_cmsRegisterOptimizationPlugin                               (_cms)RegisterOptimizationPlugin                             color management system (little cms) ??
_cmsRegisterParametricCurvesPlugin                           (_cms)RegisterParametricCurvesPlugin                         color management system (little cms) ??
_cmsRegisterRenderingIntentPlugin                            (_cms)RegisterRenderingIntentPlugin                          color management system (little cms) ??
_cmsRegisterTagPlugin                                        (_cms)RegisterTagPlugin                                      color management system (little cms) ??
_cmsRegisterTagTypePlugin                                    (_cms)RegisterTagTypePlugin                                  color management system (little cms) ??
_cmsRegisterTransformPlugin                                  (_cms)RegisterTransformPlugin                                color management system (little cms) ??
_cmsSearchTag                                                (_cms)SearchTag                                              color management system (little cms) ??
_cmsSetInterpolationRoutine                                  (_cms)SetInterpolationRoutine                                color management system (little cms) ??
_cmsSetTransformUserData                                     (_cms)SetTransformUserData                                   color management system (little cms) ??
_cmsStageAllocIdentityCLut                                   (_cms)StageAllocIdentityCLut                                 color management system (little cms) ??
_cmsStageAllocIdentityCurves                                 (_cms)StageAllocIdentityCurves                               color management system (little cms) ??
_cmsStageAllocLab2XYZ                                        (_cms)StageAllocLab2XYZ                                      color management system (little cms) ??
_cmsStageAllocLabPrelin                                      (_cms)StageAllocLabPrelin                                    color management system (little cms) ??
_cmsStageAllocLabV2ToV4                                      (_cms)StageAllocLabV2ToV4                                    color management system (little cms) ??
_cmsStageAllocLabV2ToV4curves                                (_cms)StageAllocLabV2ToV4curves                              color management system (little cms) ??
_cmsStageAllocLabV4ToV2                                      (_cms)StageAllocLabV4ToV2                                    color management system (little cms) ??
_cmsStageAllocNamedColor                                     (_cms)StageAllocNamedColor                                   color management system (little cms) ??
_cmsStageAllocPlaceholder                                    (_cms)StageAllocPlaceholder                                  color management system (little cms) ??
_cmsStageAllocXYZ2Lab                                        (_cms)StageAllocXYZ2Lab                                      color management system (little cms) ??
_cmsStageClipNegatives                                       (_cms)StageClipNegatives                                     color management system (little cms) ??
_cmsStageGetPtrToCurveSet                                    (_cms)StageGetPtrToCurveSet                                  color management system (little cms) ??
_cmsStageNormalizeFromLabFloat                               (_cms)StageNormalizeFromLabFloat                             color management system (little cms) ??
_cmsStageNormalizeFromXyzFloat                               (_cms)StageNormalizeFromXyzFloat                             color management system (little cms) ??
_cmsStageNormalizeToLabFloat                                 (_cms)StageNormalizeToLabFloat                               color management system (little cms) ??
_cmsStageNormalizeToXyzFloat                                 (_cms)StageNormalizeToXyzFloat                               color management system (little cms) ??
_cmsSubAlloc                                                 (_cms)SubAlloc                                               color management system (little cms) ??
_cmsSubAllocDestroy                                          (_cms)SubAllocDestroy                                        color management system (little cms) ??
_cmsSubAllocDup                                              (_cms)SubAllocDup                                            color management system (little cms) ??
_cmsTagSignature2String                                      (_cms)TagSignature2String                                    color management system (little cms) ??
_cmsTransform2toTransformAdaptor                             (_cms)Transform2toTransformAdaptor                           color management system (little cms) ??
_cmsUnlockMutex                                              (_cms)UnlockMutex                                            color management system (little cms) ??
_cmsVEC3cross                                                (_cms)VEC3cross                                              color management system (little cms) ??
_cmsVEC3distance                                             (_cms)VEC3distance                                           color management system (little cms) ??
_cmsVEC3dot                                                  (_cms)VEC3dot                                                color management system (little cms) ??
_cmsVEC3init                                                 (_cms)VEC3init                                               color management system (little cms) ??
_cmsVEC3length                                               (_cms)VEC3length                                             color management system (little cms) ??
_cmsVEC3minus                                                (_cms)VEC3minus                                              color management system (little cms) ??
_cmsWrite15Fixed16Number                                     (_cms)Write15Fixed16Number                                   color management system (little cms) ??
_cmsWriteAlignment                                           (_cms)WriteAlignment                                         color management system (little cms) ??
_cmsWriteFloat32Number                                       (_cms)WriteFloat32Number                                     color management system (little cms) ??
_cmsWriteHeader                                              (_cms)WriteHeader                                            color management system (little cms) ??
_cmsWriteProfileSequence                                     (_cms)WriteProfileSequence                                   color management system (little cms) ??
_cmsWriteTypeBase                                            (_cms)WriteTypeBase                                          color management system (little cms) ??
_cmsWriteUInt16Array                                         (_cms)WriteUInt16Array                                       color management system (little cms) ??
_cmsWriteUInt16Number                                        (_cms)WriteUInt16Number                                      color management system (little cms) ??
_cmsWriteUInt32Number                                        (_cms)WriteUInt32Number                                      color management system (little cms) ??
_cmsWriteUInt64Number                                        (_cms)WriteUInt64Number                                      color management system (little cms) ??
_cmsWriteUInt8Number                                         (_cms)WriteUInt8Number                                       color management system (little cms) ??
_cmsWriteXYZNumber                                           (_cms)WriteXYZNumber                                         color management system (little cms) ??
bchswSampler                                                 bchswSampler                                                 ??
cmsAdaptToIlluminant                                         (cms)AdaptToIlluminant                                       color management system (little cms) ??
cmsAllocNamedColorList                                       (cms)AllocNamedColorList                                     color management system (little cms) ??
cmsAllocProfileSequenceDescription                           (cms)AllocProfileSequenceDescription                         color management system (little cms) ??
cmsAppendNamedColor                                          (cms)AppendNamedColor                                        color management system (little cms) ??
cmsBFDdeltaE                                                 (cms)BFDdeltaE                                               color management system (little cms) ??
cmsBuildGamma                                                (cms)BuildGamma                                              color management system (little cms) ??
cmsBuildParametricToneCurve                                  (cms)BuildParametricToneCurve                                color management system (little cms) ??
cmsBuildSegmentedToneCurve                                   (cms)BuildSegmentedToneCurve                                 color management system (little cms) ??
cmsBuildTabulatedToneCurve16                                 (cms)BuildTabulatedToneCurve16                               color management system (little cms) ??
cmsBuildTabulatedToneCurveFloat                              (cms)BuildTabulatedToneCurveFloat                            color management system (little cms) ??
cmsCIE2000DeltaE                                             (cms)CIE2000DeltaE                                           color management system (little cms) ??
cmsCIE94DeltaE                                               (cms)CIE94DeltaE                                             color management system (little cms) ??
cmsCMCdeltaE                                                 (cms)CMCdeltaE                                               color management system (little cms) ??
cmsChangeBuffersFormat                                       (cms)ChangeBuffersFormat                                     color management system (little cms) ??
cmsChannelsOf                                                (cms)ChannelsOf                                              color management system (little cms) ??
cmsCloseIOhandler                                            (cms)CloseIOhandler                                          color management system (little cms) ??
cmsCloseProfile                                              (cms)CloseProfile                                            color management system (little cms) ??
cmsCreateBCHSWabstractProfile                                (cms)CreateBCHSWabstractProfile                              color management system (little cms) ??
cmsCreateBCHSWabstractProfileTHR                             (cms)CreateBCHSWabstractProfileTHR                           color management system (little cms) ??
cmsCreateContext                                             (cms)CreateContext                                           color management system (little cms) ??
cmsCreateExtendedTransform                                   (cms)CreateExtendedTransform                                 color management system (little cms) ??
cmsCreateGrayProfile                                         (cms)CreateGrayProfile                                       color management system (little cms) ??
cmsCreateGrayProfileTHR                                      (cms)CreateGrayProfileTHR                                    color management system (little cms) ??
cmsCreateInkLimitingDeviceLink                               (cms)CreateInkLimitingDeviceLink                             color management system (little cms) ??
cmsCreateInkLimitingDeviceLinkTHR                            (cms)CreateInkLimitingDeviceLinkTHR                          color management system (little cms) ??
cmsCreateLab2Profile                                         (cms)CreateLab2Profile                                       color management system (little cms) ??
cmsCreateLab2ProfileTHR                                      (cms)CreateLab2ProfileTHR                                    color management system (little cms) ??
cmsCreateLab4Profile                                         (cms)CreateLab4Profile                                       color management system (little cms) ??
cmsCreateLab4ProfileTHR                                      (cms)CreateLab4ProfileTHR                                    color management system (little cms) ??
cmsCreateLinearizationDeviceLink                             (cms)CreateLinearizationDeviceLink                           color management system (little cms) ??
cmsCreateLinearizationDeviceLinkTHR                          (cms)CreateLinearizationDeviceLinkTHR                        color management system (little cms) ??
cmsCreateMultiprofileTransform                               (cms)CreateMultiprofileTransform                             color management system (little cms) ??
cmsCreateMultiprofileTransformTHR                            (cms)CreateMultiprofileTransformTHR                          color management system (little cms) ??
cmsCreateNULLProfile                                         (cms)CreateNULLProfile                                       color management system (little cms) ??
cmsCreateNULLProfileTHR                                      (cms)CreateNULLProfileTHR                                    color management system (little cms) ??
cmsCreateProfilePlaceholder                                  (cms)CreateProfilePlaceholder                                color management system (little cms) ??
cmsCreateProofingTransform                                   (cms)CreateProofingTransform                                 color management system (little cms) ??
cmsCreateProofingTransformTHR                                (cms)CreateProofingTransformTHR                              color management system (little cms) ??
cmsCreateRGBProfile                                          (cms)CreateRGBProfile                                        color management system (little cms) ??
cmsCreateRGBProfileTHR                                       (cms)CreateRGBProfileTHR                                     color management system (little cms) ??
cmsCreateTransform                                           (cms)CreateTransform                                         color management system (little cms) ??
cmsCreateTransformTHR                                        (cms)CreateTransformTHR                                      color management system (little cms) ??
cmsCreateXYZProfile                                          (cms)CreateXYZProfile                                        color management system (little cms) ??
cmsCreateXYZProfileTHR                                       (cms)CreateXYZProfileTHR                                     color management system (little cms) ??
cmsCreate_sRGBProfile                                        (cms)Create_sRGBProfile                                      color management system (little cms) ??
cmsCreate_sRGBProfileTHR                                     (cms)Create_sRGBProfileTHR                                   color management system (little cms) ??
cmsD50_XYZ                                                   (cms)D50_XYZ                                                 color management system (little cms) ??
cmsD50_xyY                                                   (cms)D50_xyY                                                 color management system (little cms) ??
cmsDeleteContext                                             (cms)DeleteContext                                           color management system (little cms) ??
cmsDeleteTransform                                           (cms)DeleteTransform                                         color management system (little cms) ??
cmsDeltaE                                                    (cms)DeltaE                                                  color management system (little cms) ??
cmsDesaturateLab                                             (cms)DesaturateLab                                           color management system (little cms) ??
cmsDetectBlackPoint                                          (cms)DetectBlackPoint                                        color management system (little cms) ??
cmsDetectDestinationBlackPoint                               (cms)DetectDestinationBlackPoint                             color management system (little cms) ??
cmsDetectTAC                                                 (cms)DetectTAC                                               color management system (little cms) ??
cmsDictAddEntry                                              (cms)DictAddEntry                                            color management system (little cms) ??
cmsDictAlloc                                                 (cms)DictAlloc                                               color management system (little cms) ??
cmsDictDup                                                   (cms)DictDup                                                 color management system (little cms) ??
cmsDictFree                                                  (cms)DictFree                                                color management system (little cms) ??
cmsDictGetEntryList                                          (cms)DictGetEntryList                                        color management system (little cms) ??
cmsDictNextEntry                                             (cms)DictNextEntry                                           color management system (little cms) ??
cmsDoTransform                                               (cms)DoTransform                                             color management system (little cms) ??
cmsDoTransformLineStride                                     (cms)DoTransformLineStride                                   color management system (little cms) ??
cmsDoTransformStride                                         (cms)DoTransformStride                                       color management system (little cms) ??
cmsDupContext                                                (cms)DupContext                                              color management system (little cms) ??
cmsDupNamedColorList                                         (cms)DupNamedColorList                                       color management system (little cms) ??
cmsDupProfileSequenceDescription                             (cms)DupProfileSequenceDescription                           color management system (little cms) ??
cmsDupToneCurve                                              (cms)DupToneCurve                                            color management system (little cms) ??
cmsEstimateGamma                                             (cms)EstimateGamma                                           color management system (little cms) ??
cmsEvalToneCurve16                                           (cms)EvalToneCurve16                                         color management system (little cms) ??
cmsEvalToneCurveFloat                                        (cms)EvalToneCurveFloat                                      color management system (little cms) ??
cmsFloat2LabEncoded                                          (cms)Float2LabEncoded                                        color management system (little cms) ??
cmsFloat2LabEncodedV2                                        (cms)Float2LabEncodedV2                                      color management system (little cms) ??
cmsFloat2XYZEncoded                                          (cms)Float2XYZEncoded                                        color management system (little cms) ??
cmsFormatterForColorspaceOfProfile                           (cms)FormatterForColorspaceOfProfile                         color management system (little cms) ??
cmsFormatterForPCSOfProfile                                  (cms)FormatterForPCSOfProfile                                color management system (little cms) ??
cmsFreeNamedColorList                                        (cms)FreeNamedColorList                                      color management system (little cms) ??
cmsFreeProfileSequenceDescription                            (cms)FreeProfileSequenceDescription                          color management system (little cms) ??
cmsFreeToneCurve                                             (cms)FreeToneCurve                                           color management system (little cms) ??
cmsFreeToneCurveTriple                                       (cms)FreeToneCurveTriple                                     color management system (little cms) ??
cmsGetAlarmCodes                                             (cms)GetAlarmCodes                                           color management system (little cms) ??
cmsGetAlarmCodesTHR                                          (cms)GetAlarmCodesTHR                                        color management system (little cms) ??
cmsGetColorSpace                                             (cms)GetColorSpace                                           color management system (little cms) ??
cmsGetContextUserData                                        (cms)GetContextUserData                                      color management system (little cms) ??
cmsGetDeviceClass                                            (cms)GetDeviceClass                                          color management system (little cms) ??
cmsGetEncodedCMMversion                                      (cms)GetEncodedCMMversion                                    color management system (little cms) ??
cmsGetEncodedICCversion                                      (cms)GetEncodedICCversion                                    color management system (little cms) ??
cmsGetHeaderAttributes                                       (cms)GetHeaderAttributes                                     color management system (little cms) ??
cmsGetHeaderCreationDateTime                                 (cms)GetHeaderCreationDateTime                               color management system (little cms) ??
cmsGetHeaderCreator                                          (cms)GetHeaderCreator                                        color management system (little cms) ??
cmsGetHeaderFlags                                            (cms)GetHeaderFlags                                          color management system (little cms) ??
cmsGetHeaderManufacturer                                     (cms)GetHeaderManufacturer                                   color management system (little cms) ??
cmsGetHeaderModel                                            (cms)GetHeaderModel                                          color management system (little cms) ??
cmsGetHeaderProfileID                                        (cms)GetHeaderProfileID                                      color management system (little cms) ??
cmsGetHeaderRenderingIntent                                  (cms)GetHeaderRenderingIntent                                color management system (little cms) ??
cmsGetNamedColorList                                         (cms)GetNamedColorList                                       color management system (little cms) ??
cmsGetPCS                                                    (cms)GetPCS                                                  color management system (little cms) ??
cmsGetPipelineContextID                                      (cms)GetPipelineContextID                                    color management system (little cms) ??
cmsGetProfileContextID                                       (cms)GetProfileContextID                                     color management system (little cms) ??
cmsGetProfileIOhandler                                       (cms)GetProfileIOhandler                                     color management system (little cms) ??
cmsGetProfileInfo                                            (cms)GetProfileInfo                                          color management system (little cms) ??
cmsGetProfileInfoASCII                                       (cms)GetProfileInfoASCII                                     color management system (little cms) ??
cmsGetProfileVersion                                         (cms)GetProfileVersion                                       color management system (little cms) ??
cmsGetSupportedIntents                                       (cms)GetSupportedIntents                                     color management system (little cms) ??
cmsGetSupportedIntentsTHR                                    (cms)GetSupportedIntentsTHR                                  color management system (little cms) ??
cmsGetTagCount                                               (cms)GetTagCount                                             color management system (little cms) ??
cmsGetTagSignature                                           (cms)GetTagSignature                                         color management system (little cms) ??
cmsGetToneCurveEstimatedTable                                (cms)GetToneCurveEstimatedTable                              color management system (little cms) ??
cmsGetToneCurveEstimatedTableEntries                         (cms)GetToneCurveEstimatedTableEntries                       color management system (little cms) ??
cmsGetToneCurveParametricType                                (cms)GetToneCurveParametricType                              color management system (little cms) ??
cmsGetTransformContextID                                     (cms)GetTransformContextID                                   color management system (little cms) ??
cmsGetTransformInputFormat                                   (cms)GetTransformInputFormat                                 color management system (little cms) ??
cmsGetTransformOutputFormat                                  (cms)GetTransformOutputFormat                                color management system (little cms) ??
cmsIsCLUT                                                    (cms)IsCLUT                                                  color management system (little cms) ??
cmsIsIntentSupported                                         (cms)IsIntentSupported                                       color management system (little cms) ??
cmsIsMatrixShaper                                            (cms)IsMatrixShaper                                          color management system (little cms) ??
cmsIsTag                                                     (cms)IsTag                                                   color management system (little cms) ??
cmsIsToneCurveDescending                                     (cms)IsToneCurveDescending                                   color management system (little cms) ??
cmsIsToneCurveLinear                                         (cms)IsToneCurveLinear                                       color management system (little cms) ??
cmsIsToneCurveMonotonic                                      (cms)IsToneCurveMonotonic                                    color management system (little cms) ??
cmsIsToneCurveMultisegment                                   (cms)IsToneCurveMultisegment                                 color management system (little cms) ??
cmsJoinToneCurve                                             (cms)JoinToneCurve                                           color management system (little cms) ??
cmsLCh2Lab                                                   (cms)LCh2Lab                                                 color management system (little cms) ??
cmsLab2LCh                                                   (cms)Lab2LCh                                                 color management system (little cms) ??
cmsLab2XYZ                                                   (cms)Lab2XYZ                                                 color management system (little cms) ??
cmsLabEncoded2Float                                          (cms)LabEncoded2Float                                        color management system (little cms) ??
cmsLabEncoded2FloatV2                                        (cms)LabEncoded2FloatV2                                      color management system (little cms) ??
cmsLinkTag                                                   (cms)LinkTag                                                 color management system (little cms) ??
cmsMLUalloc                                                  (cms)MLUalloc                                                color management system (little cms) ??
cmsMLUdup                                                    (cms)MLUdup                                                  color management system (little cms) ??
cmsMLUfree                                                   (cms)MLUfree                                                 color management system (little cms) ??
cmsMLUgetASCII                                               (cms)MLUgetASCII                                             color management system (little cms) ??
cmsMLUgetTranslation                                         (cms)MLUgetTranslation                                       color management system (little cms) ??
cmsMLUgetWide                                                (cms)MLUgetWide                                              color management system (little cms) ??
cmsMLUsetASCII                                               (cms)MLUsetASCII                                             color management system (little cms) ??
cmsMLUsetWide                                                (cms)MLUsetWide                                              color management system (little cms) ??
cmsMLUtranslationsCodes                                      (cms)MLUtranslationsCodes                                    color management system (little cms) ??
cmsMLUtranslationsCount                                      (cms)MLUtranslationsCount                                    color management system (little cms) ??
cmsNamedColorCount                                           (cms)NamedColorCount                                         color management system (little cms) ??
cmsNamedColorIndex                                           (cms)NamedColorIndex                                         color management system (little cms) ??
cmsNamedColorInfo                                            (cms)NamedColorInfo                                          color management system (little cms) ??
cmsOpenIOhandlerFromFile                                     (cms)OpenIOhandlerFromFile                                   color management system (little cms) ??
cmsOpenIOhandlerFromMem                                      (cms)OpenIOhandlerFromMem                                    color management system (little cms) ??
cmsOpenIOhandlerFromNULL                                     (cms)OpenIOhandlerFromNULL                                   color management system (little cms) ??
cmsOpenIOhandlerFromStream                                   (cms)OpenIOhandlerFromStream                                 color management system (little cms) ??
cmsOpenProfileFromFile                                       (cms)OpenProfileFromFile                                     color management system (little cms) ??
cmsOpenProfileFromFileTHR                                    (cms)OpenProfileFromFileTHR                                  color management system (little cms) ??
cmsOpenProfileFromIOhandler2THR                              (cms)OpenProfileFromIOhandler2THR                            color management system (little cms) ??
cmsOpenProfileFromIOhandlerTHR                               (cms)OpenProfileFromIOhandlerTHR                             color management system (little cms) ??
cmsOpenProfileFromMem                                        (cms)OpenProfileFromMem                                      color management system (little cms) ??
cmsOpenProfileFromMemTHR                                     (cms)OpenProfileFromMemTHR                                   color management system (little cms) ??
cmsOpenProfileFromStream                                     (cms)OpenProfileFromStream                                   color management system (little cms) ??
cmsOpenProfileFromStreamTHR                                  (cms)OpenProfileFromStreamTHR                                color management system (little cms) ??
cmsPipelineAlloc                                             (cms)PipelineAlloc                                           color management system (little cms) ??
cmsPipelineCat                                               (cms)PipelineCat                                             color management system (little cms) ??
cmsPipelineCheckAndRetreiveStages                            (cms)PipelineCheckAndRetreiveStages                          color management system (little cms) ??
cmsPipelineDup                                               (cms)PipelineDup                                             color management system (little cms) ??
cmsPipelineEval16                                            (cms)PipelineEval16                                          color management system (little cms) ??
cmsPipelineEvalFloat                                         (cms)PipelineEvalFloat                                       color management system (little cms) ??
cmsPipelineEvalReverseFloat                                  (cms)PipelineEvalReverseFloat                                color management system (little cms) ??
cmsPipelineFree                                              (cms)PipelineFree                                            color management system (little cms) ??
cmsPipelineGetPtrToFirstStage                                (cms)PipelineGetPtrToFirstStage                              color management system (little cms) ??
cmsPipelineGetPtrToLastStage                                 (cms)PipelineGetPtrToLastStage                               color management system (little cms) ??
cmsPipelineInputChannels                                     (cms)PipelineInputChannels                                   color management system (little cms) ??
cmsPipelineInsertStage                                       (cms)PipelineInsertStage                                     color management system (little cms) ??
cmsPipelineOutputChannels                                    (cms)PipelineOutputChannels                                  color management system (little cms) ??
cmsPipelineSetSaveAs8bitsFlag                                (cms)PipelineSetSaveAs8bitsFlag                              color management system (little cms) ??
cmsPipelineStageCount                                        (cms)PipelineStageCount                                      color management system (little cms) ??
cmsPipelineUnlinkStage                                       (cms)PipelineUnlinkStage                                     color management system (little cms) ??
cmsPlugin                                                    (cms)Plugin                                                  color management system (little cms) ??
cmsPluginTHR                                                 (cms)PluginTHR                                               color management system (little cms) ??
cmsReadRawTag                                                (cms)ReadRawTag                                              color management system (little cms) ??
cmsReadTag                                                   (cms)ReadTag                                                 color management system (little cms) ??
cmsReverseToneCurve                                          (cms)ReverseToneCurve                                        color management system (little cms) ??
cmsReverseToneCurveEx                                        (cms)ReverseToneCurveEx                                      color management system (little cms) ??
cmsSaveProfileToFile                                         (cms)SaveProfileToFile                                       color management system (little cms) ??
cmsSaveProfileToIOhandler                                    (cms)SaveProfileToIOhandler                                  color management system (little cms) ??
cmsSaveProfileToMem                                          (cms)SaveProfileToMem                                        color management system (little cms) ??
cmsSaveProfileToStream                                       (cms)SaveProfileToStream                                     color management system (little cms) ??
cmsSetAdaptationState                                        (cms)SetAdaptationState                                      color management system (little cms) ??
cmsSetAdaptationStateTHR                                     (cms)SetAdaptationStateTHR                                   color management system (little cms) ??
cmsSetAlarmCodes                                             (cms)SetAlarmCodes                                           color management system (little cms) ??
cmsSetAlarmCodesTHR                                          (cms)SetAlarmCodesTHR                                        color management system (little cms) ??
cmsSetColorSpace                                             (cms)SetColorSpace                                           color management system (little cms) ??
cmsSetDeviceClass                                            (cms)SetDeviceClass                                          color management system (little cms) ??
cmsSetEncodedICCversion                                      (cms)SetEncodedICCversion                                    color management system (little cms) ??
cmsSetHeaderAttributes                                       (cms)SetHeaderAttributes                                     color management system (little cms) ??
cmsSetHeaderFlags                                            (cms)SetHeaderFlags                                          color management system (little cms) ??
cmsSetHeaderManufacturer                                     (cms)SetHeaderManufacturer                                   color management system (little cms) ??
cmsSetHeaderModel                                            (cms)SetHeaderModel                                          color management system (little cms) ??
cmsSetHeaderProfileID                                        (cms)SetHeaderProfileID                                      color management system (little cms) ??
cmsSetHeaderRenderingIntent                                  (cms)SetHeaderRenderingIntent                                color management system (little cms) ??
cmsSetLogErrorHandler                                        (cms)SetLogErrorHandler                                      color management system (little cms) ??
cmsSetLogErrorHandlerTHR                                     (cms)SetLogErrorHandlerTHR                                   color management system (little cms) ??
cmsSetPCS                                                    (cms)SetPCS                                                  color management system (little cms) ??
cmsSetProfileVersion                                         (cms)SetProfileVersion                                       color management system (little cms) ??
cmsSignalError                                               (cms)SignalError                                             color management system (little cms) ??
cmsSliceSpace16                                              (cms)SliceSpace16                                            color management system (little cms) ??
cmsSliceSpaceFloat                                           (cms)SliceSpaceFloat                                         color management system (little cms) ??
cmsSmoothToneCurve                                           (cms)SmoothToneCurve                                         color management system (little cms) ??
cmsStageAllocCLut16bit                                       (cms)StageAllocCLut16bit                                     color management system (little cms) ??
cmsStageAllocCLut16bitGranular                               (cms)StageAllocCLut16bitGranular                             color management system (little cms) ??
cmsStageAllocCLutFloat                                       (cms)StageAllocCLutFloat                                     color management system (little cms) ??
cmsStageAllocCLutFloatGranular                               (cms)StageAllocCLutFloatGranular                             color management system (little cms) ??
cmsStageAllocIdentity                                        (cms)StageAllocIdentity                                      color management system (little cms) ??
cmsStageAllocMatrix                                          (cms)StageAllocMatrix                                        color management system (little cms) ??
cmsStageAllocToneCurves                                      (cms)StageAllocToneCurves                                    color management system (little cms) ??
cmsStageData                                                 (cms)StageData                                               color management system (little cms) ??
cmsStageDup                                                  (cms)StageDup                                                color management system (little cms) ??
cmsStageFree                                                 (cms)StageFree                                               color management system (little cms) ??
cmsStageInputChannels                                        (cms)StageInputChannels                                      color management system (little cms) ??
cmsStageNext                                                 (cms)StageNext                                               color management system (little cms) ??
cmsStageOutputChannels                                       (cms)StageOutputChannels                                     color management system (little cms) ??
cmsStageSampleCLut16bit                                      (cms)StageSampleCLut16bit                                    color management system (little cms) ??
cmsStageSampleCLutFloat                                      (cms)StageSampleCLutFloat                                    color management system (little cms) ??
cmsStageType                                                 (cms)StageType                                               color management system (little cms) ??
cmsTagLinkedTo                                               (cms)TagLinkedTo                                             color management system (little cms) ??
cmsTempFromWhitePoint                                        (cms)TempFromWhitePoint                                      color management system (little cms) ??
cmsTransform2DeviceLink                                      (cms)Transform2DeviceLink                                    color management system (little cms) ??
cmsUnregisterPlugins                                         (cms)UnregisterPlugins                                       color management system (little cms) ??
cmsUnregisterPluginsTHR                                      (cms)UnregisterPluginsTHR                                    color management system (little cms) ??
cmsWhitePointFromTemp                                        (cms)WhitePointFromTemp                                      color management system (little cms) ??
cmsWriteRawTag                                               (cms)WriteRawTag                                             color management system (little cms) ??
cmsWriteTag                                                  (cms)WriteTag                                                color management system (little cms) ??
cmsXYZ2Lab                                                   (cms)XYZ2Lab                                                 color management system (little cms) ??
cmsXYZ2xyY                                                   (cms)XYZ2xyY                                                 color management system (little cms) ??
cmsXYZEncoded2Float                                          (cms)XYZEncoded2Float                                        color management system (little cms) ??
cmsfilelength                                                (cms)filelength                                              color management system (little cms) ??
cmsstrcasecmp                                                (cms)strcasecmp                                              color management system (little cms) ??
cmsxyY2XYZ                                                   (cms)xyY2XYZ                                                 color management system (little cms) ??
copy16                                                       copy16                                                       ??
copy32                                                       copy32                                                       ??
copy64                                                       copy64                                                       ??
copy8                                                        copy8                                                        ??
defMtxCreate                                                 defMtxCreate                                                 ??
defMtxDestroy                                                defMtxDestroy                                                ??
defMtxLock                                                   defMtxLock                                                   ??
defMtxUnlock                                                 defMtxUnlock                                                 ??
errorHandler                                                 (err)orHandler                                               error recovery ??
free_ex_data_arg                                             free_ex_data_arg                                             ??
from16to8                                                    from16to8                                                    ??
from16toDBL                                                  from16toDBL                                                  ??
from16toFLT                                                  from16toFLT                                                  ??
from16toHLF                                                  from16toHLF                                                  ??
from8to16                                                    from8to16                                                    ??
from8toDBL                                                   from8toDBL                                                   ??
from8toFLT                                                   from8toFLT                                                   ??
from8toHLF                                                   from8toHLF                                                   ??
fromDBLto16                                                  fromDBLto16                                                  ??
fromDBLto8                                                   fromDBLto8                                                   ??
fromDBLtoFLT                                                 fromDBLtoFLT                                                 ??
fromDBLtoHLF                                                 fromDBLtoHLF                                                 ??
fromFLTto16                                                  fromFLTto16                                                  ??
fromFLTto8                                                   fromFLTto8                                                   ??
fromFLTtoDBL                                                 fromFLTtoDBL                                                 ??
fromFLTtoHLF                                                 fromFLTtoHLF                                                 ??
fromHLFto16                                                  fromHLFto16                                                  ??
fromHLFto8                                                   fromHLFto8                                                   ??
fromHLFtoDBL                                                 fromHLFtoDBL                                                 ??
fromHLFtoFLT                                                 fromHLFtoFLT                                                 ??
getILData                                                    getILData                                                    ??
kcfis_oss_wait_per_ctx                                       (kcfis)_oss_wait_per_ctx                                     kernel cache file management intelligent storage ??
kcfis_set_appliance_ctx                                      (kcfis)_set_appliance_ctx                                    kernel cache file management intelligent storage ??
kcfis_tablespace_is_smart_scannable                          (kcfis)_tablespace_is_smart_scannable                        kernel cache file management intelligent storage ??
kdmoCheckGbyAgg                                              (kdmo)CheckGbyAgg                                            kernel data in-memory data layer optimizer ??
kdmoCheckGbyOpn                                              (kdmo)CheckGbyOpn                                            kernel data in-memory data layer optimizer ??
kdmoColsUlvlIsDict                                           (kdmo)ColsUlvlIsDict                                         kernel data in-memory data layer optimizer ??
kdmoGByPushdownCardEst                                       (kdmo)GByPushdownCardEst                                     kernel data in-memory data layer optimizer ??
kdmoGByPushdownValid                                         (kdmo)GByPushdownValid                                       kernel data in-memory data layer optimizer ??
kdmoGenericPcodeCheck                                        (kdmo)GenericPcodeCheck                                      kernel data in-memory data layer optimizer ??
kdmoStorageColCheck                                          (kdmo)StorageColCheck                                        kernel data in-memory data layer optimizer ??
kdmoStorageConstantCheck                                     (kdmo)StorageConstantCheck                                   kernel data in-memory data layer optimizer ??
kdmoStorageOpnCheckAux                                       (kdmo)StorageOpnCheckAux                                     kernel data in-memory data layer optimizer ??
kdmoTabUlvlIsDict                                            (kdmo)TabUlvlIsDict                                          kernel data in-memory data layer optimizer ??
kdmoValidGbyPushdownPredCB                                   (kdmo)ValidGbyPushdownPredCB                                 kernel data in-memory data layer optimizer ??
kdzfCheckDbaMatch                                            (kdz)fCheckDbaMatch                                          kernel data archive compression ??
kdzsDumpKaf                                                  (kdzs)DumpKaf                                                kernel data archive compression decompression ??
kfgbCheckInterrupt                                           (kfgb)CheckInterrupt                                         kernel automatic storage management diskgroups background ??
kgh_check_simple_free_canary                                 (kgh)_check_simple_free_canary                               kernel generic heap manager ??
kgh_get_max_size_on_list                                     (kgh)_get_max_size_on_list                                   kernel generic heap manager ??
kgh_get_visit_limit                                          (kgh)_get_visit_limit                                        kernel generic heap manager ??
kgh_set_fl_effort                                            (kgh)_set_fl_effort                                          kernel generic heap manager ??
kghsrch_best_fit                                             (kghsrch)_best_fit                                           kernel generic heap manager search freelists for a memory chunk ??
kghunalo                                                     (kgh)unalo                                                   kernel generic heap manager ??
kgzm_encode_identify_ctype                                   (kg)zm_encode_identify_ctype                                 kernel generic ??
kgzm_encode_version_with_reid                                (kg)zm_encode_version_with_reid                              kernel generic ??
kjbmisgdbabast                                               (kjb)misgdbabast                                             kernel lock management global cache service ??
kjbmrejectbast                                               (kjb)mrejectbast                                             kernel lock management global cache service ??
kkesrcCard                                                   (kke)srcCard                                                 kernel compile cost engine ??
kkqjpdrace                                                   (kkqjpd)race                                                 kernel compile query  join analysis predicate push down ??
kkqoreIsUnsupported                                          (kkqore)IsUnsupported                                        kernel compile query  or-expansion ??
klxGetBuffer                                                 (kl)xGetBuffer                                               kernel loader ??
klxGetLength                                                 (kl)xGetLength                                               kernel loader ??
koklc_screate                                                (kokl)c_screate                                              kernel objects kernel side lob access ??
kolaCreateFromExternalSrc                                    (kola)CreateFromExternalSrc                                  kernel objects lob ??
kolaetAssign                                                 (kola)etAssign                                               kernel objects lob ??
kolaetChkSize                                                (kola)etChkSize                                              kernel objects lob ??
kolaetCreate                                                 (kola)etCreate                                               kernel objects lob ??
kolaetCreateCtx                                              (kola)etCreateCtx                                            kernel objects lob ??
kolaetCreateExternalInline                                   (kola)etCreateExternalInline                                 kernel objects lob ??
kolaetCreateFromExternalInline                               (kola)etCreateFromExternalInline                             kernel objects lob ??
kolaetDmpData                                                (kola)etDmpData                                              kernel objects lob ??
kolaetFree                                                   (kola)etFree                                                 kernel objects lob ??
kolaetGetLength                                              (kola)etGetLength                                            kernel objects lob ??
kolaetGetRawData                                             (kola)etGetRawData                                           kernel objects lob ??
kolaetIsExternalInlineLob                                    (kola)etIsExternalInlineLob                                  kernel objects lob ??
kolaetRead                                                   (kola)etRead                                                 kernel objects lob ??
kolaetWrite                                                  (kola)etWrite                                                kernel objects lob ??
kolamalSage                                                  (kola)malSage                                                kernel objects lob ??
kolamfrSage                                                  (kola)mfrSage                                                kernel objects lob ??
kolasugc                                                     (kola)sugc                                                   kernel objects lob ??
kolasugi                                                     (kola)sugi                                                   kernel objects lob ??
kolrsdesht                                                   (kolr)sdesht                                                 kernel objects lob refcount ??
kolrsugi                                                     (kolr)sugi                                                   kernel objects lob refcount ??
kqltprom                                                     (kql)tprom                                                   kernel query library cache ??
krbCreateGuidStr                                             (krb)CreateGuidStr                                           kernel redo backup/restore ??
krbrCheckLogical                                             (krbr)CheckLogical                                           kernel redo backup/restore restore and recovery ??
krvxdsr                                                      (krvx)dsr                                                    kernel redo recovery extract ??
ksmsq_memlock_limit_alert_error                              (ksmsq)_memlock_limit_alert_error                            kernel service (VOS) memory sga heap message queue services ??
ksmsq_transport_require_memlock                              (ksmsq)_transport_require_memlock                            kernel service (VOS) memory sga heap message queue services ??
ksz_oss_get_ip_info                                          (ksz)_oss_get_ip_info                                        kernel service (VOS) oracle storage server (OSS) server layer ??
ksz_skgxp_check_default_query                                (ksz)_skgxp_check_default_query                              kernel service (VOS) oracle storage server (OSS) server layer ??
ktmaLgwrHeartbeat                                            (ktma)LgwrHeartbeat                                          kernel transaction transaction monitor (smon) IM transaction ADG ??
ktsttsn_to_tsnpdb                                            (ktst)tsn_to_tsnpdb                                          kernel transaction segment management sort management ??
ktsttsnpdb_to_pdb                                            (ktst)tsnpdb_to_pdb                                          kernel transaction segment management sort management ??
ktsttsnpdb_to_tsn                                            (ktst)tsnpdb_to_tsn                                          kernel transaction segment management sort management ??
kubsjniSkip                                                  (ku)bsjniSkip                                                kernel utility ??
kubsxiMapCluUser                                             (ku)bsxiMapCluUser                                           kernel utility ??
kubsxiSetCpx                                                 (ku)bsxiSetCpx                                               kernel utility ??
kxfrGetNextSplit                                             (kxfr)GetNextSplit                                           kernel execution parallel query granules ??
kzvdvechk_ownerid_1                                          (kzvd)vechk_ownerid_1                                        kernel security data vault ??
oracle_storidx_recompute_ridx_summary                        oracle_storidx_recompute_ridx_summary                        ??
prsUnusable                                                  (prs)Unusable                                                parse ??
prscViewColAlias                                             (prs)cViewColAlias                                           parse ??
prsc_edition                                                 (prs)c_edition                                               parse ??
prscal                                                       (prs)cal                                                     parse ??
prscap                                                       (prs)cap                                                     parse ??
prscbl                                                       (prs)cbl                                                     parse ??
prsccc                                                       (prs)ccc                                                     parse ??
prscct                                                       (prs)cct                                                     parse ??
prscdg                                                       (prs)cdg                                                     parse ??
prscdr                                                       (prs)cdr                                                     parse ??
prscet_parse_extab                                           (prs)cet_parse_extab                                         parse ??
prscipc_index_placement_clause                               (prs)cipc_index_placement_clause                             parse ??
prscisfbdrp                                                  (prs)cisfbdrp                                                parse ??
prscisfbtbl                                                  (prs)cisfbtbl                                                parse ??
prscnd                                                       (prs)cnd                                                     parse ??
prscpfil                                                     (prs)cpfil                                                   parse ??
prscrefv                                                     (prs)crefv                                                   parse ??
prscrely                                                     (prs)crely                                                   parse ??
prscspfi                                                     (prs)cspfi                                                   parse ??
prscvac                                                      (prs)cvac                                                    parse ??
prsdfl                                                       (prs)dfl                                                     parse ??
prsfbdrp                                                     (prs)fbdrp                                                   parse ??
prsfbdrptab                                                  (prs)fbdrptab                                                parse ??
prsfla                                                       (prs)fla                                                     parse ??
prshve_high_val_expr                                         (prs)hve_high_val_expr                                       parse ??
prsicsl                                                      (prs)icsl                                                    parse ??
prsident                                                     (prs)ident                                                   parse ??
prsini_iot_init                                              (prs)ini_iot_init                                            parse ??
prsref                                                       (prs)ref                                                     parse ??
prssoasn                                                     (prs)soasn                                                   parse ??
prssptoa                                                     (prs)sptoa                                                   parse ??
qcpiSetFnInOpn                                               (qcpi)SetFnInOpn                                             query compile parse interim ??
qcpiSetFnInSel                                               (qcpi)SetFnInSel                                             query compile parse interim ??
qertbFilter_qertbs                                           (qertb)Filter_qertbs                                         query execute rowsource table access ??
qertbInitXTFilter                                            (qertb)InitXTFilter                                          query execute rowsource table access ??
qertbQueryHXTFilter                                          (qertb)QueryHXTFilter                                        query execute rowsource table access ??
qertbQueryLXTFilter                                          (qertb)QueryLXTFilter                                        query execute rowsource table access ??
qertbUpdateXTFilter                                          (qertb)UpdateXTFilter                                        query execute rowsource table access ??
qertbXTCbk                                                   (qertb)XTCbk                                                 query execute rowsource table access ??
qesSageIsHDFSTsn                                             (qes)SageIsHDFSTsn                                           query execute services ??
qesxtcCloseScan                                              (qes)xtcCloseScan                                            query execute services ??
qesxtcFetchCurrScan                                          (qes)xtcFetchCurrScan                                        query execute services ??
qesxtcFetchNextScan                                          (qes)xtcFetchNextScan                                        query execute services ??
qesxtcOpenScan                                               (qes)xtcOpenScan                                             query execute services ??
qjsnGenerateCapabilitySet                                    (qjsn)GenerateCapabilitySet                                  query json ??
qjsngCloseSageUga                                            (qjsn)gCloseSageUga                                          query json ??
qjsngCloseUga                                                (qjsn)gCloseUga                                              query json ??
qjsngInitSageUga                                             (qjsn)gInitSageUga                                           query json ??
qjsngStreamFromLobInSage                                     (qjsn)gStreamFromLobInSage                                   query json ??
qjsngStreamFromLob_h                                         (qjsn)gStreamFromLob_h                                       query json ??
qkaQknSTPruneKaf                                             (qka)QknSTPruneKaf                                           query kernel allocation ??
qksSageFlushSgaCIC                                           (qksSage)FlushSgaCIC                                         query kernel sql exadata ??
qksSageFlushSgaCICDump                                       (qksSage)FlushSgaCICDump                                     query kernel sql exadata ??
qksSageFlushSgaImpl                                          (qksSage)FlushSgaImpl                                        query kernel sql exadata ??
qksSageLoadInternalExternalMD                                (qksSage)LoadInternalExternalMD                              query kernel sql exadata ??
qksSageMakeTbsHDFS                                           (qksSage)MakeTbsHDFS                                         query kernel sql exadata ??
qkssage_kkxbct                                               (qkssa)ge_kkxbct                                             query kernel sql sampling ??
r_ex_data_clear                                              r_ex_data_clear                                              ??
r_ex_data_update                                             r_ex_data_update                                             ??
r_ssl_ctx_ex_data_clear                                      r_ssl_ctx_ex_data_clear                                      ??
r_ssl_ctx_get_dh_uses                                        r_ssl_ctx_get_dh_uses                                        ??
r_ssl_ctx_set_dh_uses                                        r_ssl_ctx_set_dh_uses                                        ??
r_ssl_ec_cert_algs_are_equal                                 r_ssl_ec_cert_algs_are_equal                                 ??
r_ssl_get_dh_uses                                            r_ssl_get_dh_uses                                            ??
r_ssl_set_dh_uses                                            r_ssl_set_dh_uses                                            ??
releaseILData                                                releaseILData                                                ??
ri_ssl3_base                                                 ri_ssl3_base                                                 ??
ri_ssl3_ctx_dh_tmp                                           ri_ssl3_ctx_dh_tmp                                           ??
ri_ssl3_dh_tmp                                               ri_ssl3_dh_tmp                                               ??
ri_ssl_cert_dup_params                                       ri_ssl_cert_dup_params                                       ??
ri_ssl_cipher_ctx_cipher_size                                ri_ssl_cipher_ctx_cipher_size                                ??
ri_ssl_cipher_ctx_is_aead                                    ri_ssl_cipher_ctx_is_aead                                    ??
sageDataReInitCtx                                            (sageData)ReInitCtx                                          exadata specific data layer ??
sageetProcessInternalTable                                   (sage)etProcessInternalTable                                 exadata specific ??
sageet_fp_buf_corrupt                                        (sage)et_fp_buf_corrupt                                      exadata specific ??
sageet_fp_buf_noalloc                                        (sage)et_fp_buf_noalloc                                      exadata specific ??
sageet_fp_buf_noresize                                       (sage)et_fp_buf_noresize                                     exadata specific ??
sagesqlFindCidInStoridx                                      (sagesql)FindCidInStoridx                                    exadata specific sql ??
seqSetStartWith                                              (seq)SetStartWith                                            sequence numbers ??
seqclp                                                       (seq)clp                                                     sequence numbers ??
seqmmv                                                       (seq)mmv                                                     sequence numbers ??
seqsetmin                                                    (seq)setmin                                                  sequence numbers ??
Advertisements

This blogpost is about oracle’s patching tool for the database and grid infrastructure: opatch. I personally have a love/hate relationship with opatch. In essence, opatch automates a lot of things that would be very error prone if it were to be done by hand, which is a good thing. However, there are a lot of things surrounding opatch that I despise. An example of that is the version numbering of opatch itself.

Versions and more versions

To jump into the matter directly: versions. I don’t understand why this has to be this complicated. I would even go as far as saying that somebody needs to step in and clean this up.

All opatch versions are identified by patch number 6880880. Anyone working with patching oracle probably knows this patch number by heart. You can go to this patch very quickly in MOS if you go to the search box which sits at the right side of the screen on the same height as the tabs, and type ‘patch 6880880’. A green rectangle will say ‘You have been directed to this PATCH based on an ID match’… So far the easy part.

On the right side below that, there is a rectangle, for which the first dropdown box is ‘release’. The release here means the release for which you want to download opatch. There are a great number of releases here:
– OPatch 12.2.0.1.0
– OPatch 13.9.0.0.0                           – EMCC
– OPatch 18.0.0.0.0
– OPatch for FMW 12c (OUI 13.9.x)   – Obsolete FMW(?)
– OUI NextGen 13.1                           – FMW
– OUI NextGen 13.2                           – FMW
– Oracle 10.1.0.0.0
– Oracle 10.1.0.3                                – Obsolete
– Oracle 10.2.0.0.0
– Oracle 11.1.0.0.0
– Oracle 11.2.0.0.0
– Oracle 12.1.0.1.0
– Oracle 12.1.0.1.1                             – Obsolete
– Oracle Database 12.2.0.1                – Obsolete

The red entries describe the patch being obsolete, and not downloadable. IMO, they should not be there.
The blue entries are non-database opatch releases. I would beg oracle to publish non-database opatch releases under their own patch number.

This eliminates 7 of the 14 choices. The left over entries show a list of versions that seems to make sense, these seem to show the database versions for which you can download a version of opatch. Let’s look at the patch descriptions of the leftover versions:

– OPatch 12.2.0.1.0    – Patch 6880880: OPatch 12.2.0.1.14 for DB 12.2 releases (JUL 2018)
– OPatch 18.0.0.0.0    – Patch 6880880: OPatch 12.2.0.1.14 for DB 18.x releases (JUL 2018)
– Oracle 10.1.0.0.0     – Patch 6880880: OPatch 9i, 10.1
– Oracle 10.2.0.0.0     – Patch 6880880: OPatch 10.2
– Oracle 11.1.0.0.0     – Patch 6880880: OPatch patch of version 11.1.0.12.9 for Oracle software releases 11.1.0.x (OCT 2015)
– Oracle 11.2.0.0.0     – Patch 6880880: OPatch patch of version 11.2.0.3.19 for Oracle software releases 11.2.0.x and 18.x (APR 2018)
– Oracle 12.1.0.1.0     – Patch 6880880: OPatch 12.2.0.1.14 for DB 12.x and 18.x releases (JUL 2018)

If we follow the versions from the lowest to the highest versions of Opatch:
– 10.1.0.0.0     -> ‘OPatch 9i, 10.1’ This is opatch for database versions 9i and 10.1.
– 10.2.0.0.0     -> ‘OPatch 10.2’ This is opatch for database version 10.2.
– 11.1.0.0.0     -> ‘OPatch patch of version 11.1.0.12.9 for Oracle software releases 11.1.0.x (OCT 2015)’ This is opatch for database version 11.1.0.x.
– 11.2.0.0.0     -> ‘OPatch patch of version 11.2.0.3.19 for Oracle software releases 11.2.0.x and 18.x (APR 2018)’ This is opatch for database version 11.2.0.x. I have no idea what ‘and 18.x’ means.
– 12.1.0.1.0     -> ‘OPatch patch 12.2.0.1.14 for DB 12.x and 18.x releases (JUL 2018)’ This is opatch for database version 12.1.x. NOT 12.x as the patch text indicates, it’s 12.1.x, and again I have no idea what ‘and 18.x’ means. I also don’t understand why the the 4th number is getting used all of a sudden.
– 12.2.0.1.0    -> ‘OPatch 12.2.0.1.14 for DB 12.2 releases (JUL 2018)’ This is opatch for database version 12.2. The description now aptly describes the database version and does not confuse.
– 18.0.0.0.0    -> ‘OPatch 12.2.0.1.14 for DB 18.x releases (JUL 2018)’ This is opatch for database version 18. The description again now aptly describes the version.

We are not there yet. Now that the patch release that should be used for every version of the database has been identified, let me throw in another version. This is the actual version that opatch reports when it is queried with ‘opatch version’. Let me list the opatch versions:
– 11.1.0.0.0 -> opatch version 11.1.0.12.9
– 11.2.0.0.0 -> opatch version 11.2.0.3.19
– 12.1.0.1.0 -> opatch version 12.2.0.1.14
– 12.2.0.1.0 -> opatch version 12.2.0.1.14
– 18.0.0.0.0 -> opatch version 12.2.0.1.14
Yes, this is again some weirdness. The opatch version seemed to have followed the database version, and -apparently- starting from version 12.1 up to version 18 there is one opatch version: 12.2.

That begs the question whether the 12.1.0.1.0, 12.2.0.1.0 and 18.0.0.0.0 opatch versions is actually the same opatch version. I got these versions downloaded, with the opatch version included in the name:

$ md5sum p6880880*
1ee44d25f5e858eb67424b69b89b8a25  p6880880_121010_Linux-x86-64-12.2.0.1.14.zip
1ee44d25f5e858eb67424b69b89b8a25  p6880880_122010_Linux-x86-64-12.2.0.1.14.zip
1ee44d25f5e858eb67424b69b89b8a25  p6880880_180000_Linux-x86-64-12.2.0.1.14.zip

Okay…so these are actually 100% identical copies, which just have a different name. I have no idea why the exact same file is offered with a different name. This means that the addition of ‘and 18.x’ in the patch description of the release means that that opatch version can also be used with version 18 of the database.

To be honest, the original intention of this blog article was to describe the some specific usage of opatch, and I now already got enough content for a post, and will retain the original intended content for a next blogpost.

The question you might be having at this point is: but what version should I download now? For databases up to version 11.2 it is simple, there is actually only one choice for for each version. For database versions starting from version 12.1 you -currently- have a choice. I would advise simply downloading the “correct” opatch version for every database version, which means the exact oracle version as indicated in the release pulldown menu at the download page (‘Oracle 12.1.0.1.0’, ‘OPatch 12.2.0.1.0’ or ‘OPatch 18.0.0.0.0’) despite the fact that these are the same opatch versions *currently*.

I suspect that once 12.1 goes into extended support, the opatch version will freeze, whilst the opatch version for version 12.2 and 18 will be improved and increase in version. So the only way to download the correct opatch version is still by choosing the actual database version it is intended for.

This brings me to another important thing to realise: the actual opatch version that is downloaded. At the time of writing the current and only available opatch release for Oracle database versions 12.1 to 18 is 12.2.0.1.14. Once oracle brings out a newer opatch version, the previous version will not be available anywhere anymore (as far as I know). To me this means that if you patch databases per tier (from test, development and so on up to production), you have to stage opatch in order to be able to use the same opatch version consistently in the future. Of course Oracle advises to use the latest opatch version, but the patch will check for a minimal opatch version, and if you tested your database software version, opatch version and patch to be working correctly together, in my opinion the most important thing is consistency.

This post was triggered upon rereading a blogpost by Mike Dietrich called databases need patched minimum april 2019. Mike’s blogpost makes it clear this is about databases that are connected using database links, and that:
– Newer databases do not need additional patching for this issue (11.2.0.4, 12.1.0.2, 12.2 and newer).
– Recent PSU patches contain a fix for certain older versions (11.1.0.7, 11.2.0.3 and 12.1.0.1).
– This means versions 11.2.0.2 and earlier 11.2 versions, 11.1.0.6 and earlier and anything at version 10 or earlier can not be fixed and thus are affected.

But what is the actual issue?

The first link in the article: Recommended patches and actions for Oracle databases versions 12.1.0.1, 11.2.0.3 and earlier – before June 2019 (Doc ID 2361478.1) provides essentially the same information as Mike’s blogpost, however it additionaly mentions that the interoperability of database clients with database servers is not affected.

Mike’s article mentions the following:
The alert refers to an SCN issue which came up a while ago. The system change number (SCN) is a logical, internal timestamp used by the Oracle Database. SCNs order events that occur within the database, which is necessary to satisfy the ACID properties of a transaction. The database uses SCNs to query and track changes.

So I guess it has something to do with SCNs. Most of the links are about SCNs. The MOS article that is most descriptive is: System Change Number (SCN), Headroom, Security and Patch Information (Doc ID 1376995.1).

This article describes a lot of details:
– SCNs are necessary for the database to keep changes organised. I got an article that explains that SCNs are not unique to a transaction, but are “just” granular enough to keep changes organised.
– SCNs are an ever increasing number. SCNs are never decreased!
– SCNs have a hard limit, which version specific, and is based on the number of bits for the number:
– – The general limit is 2^48.
– – From version 12.2 on, with compatibility set to 12.2 or higher, the limit is 2^63.
– SCNs have a per second increasing soft limit, called ‘RSL’ or ‘reasonable SCN limit’, which is version specific:
– – The general soft limit is 16384 (16k) SCNs per second calculated by the number of second from Januari 1st, 1988 times 16384.
– – From version 12.2 on, with compatibility set to 12.2 or higher, the limit is 98304 (96k) SCNs per second calculated by the number of seconds from Januari 1st, 2008 times 98304.
– The RSL can not be exceeded, if a database session tries to go beyond the soft limit, an ORA 600 [2252] is raised and the transaction is rolled back. This will not corrupt data (but obviously the data in the transaction is not applied).
– The difference between the current SCN and the RSL SCN is known as ‘SCN headroom’.
– There have been bugs that can increase SCNs at a higher rate, decreasing the SCN headroom or even reaching the soft limit, but these have all been solved in the Januari 2012 CPU/PSU/patch bundles.
– When databases communicate which each other via a database link, the SCNs of both databases are synchronised by picking the highest of the two.

But it doesn’t really explain why patches must be applied before June 2019. However, another article is more concrete on that: Recommended patching and actions for Oracle database versions 12.1.0.1, 11.2.0.3 and earlier – before June 2019 (Doc ID 2335265.1). The first interesting thing mentioned is:

3. What is the change introduced by the patches listed above?
These patches increase the database’s current maximum SCN (system change number) limit.
At any point in time, the Oracle Database calculates a “not to exceed” limit for the number of SCNs a database can have used, based on the number of seconds elapsed since 1988.

So, this means the patched database have a change (increase) in the RSL.

These recommended patches enable the databases to allow for a higher current maximum SCN limit. The rate at which this limit is calculated can be referred to as the “SCN rate” and these patches help allow higher SCN rates to enable databases to support many times higher transaction rates than earlier releases.

And this means the RSL is increased from the 16k per second since Januari 1988 to the 96k per second since Januari 2008.

Please note that the patches only increase the max limit but the current SCN is not impacted. So, if all your databases don’t have any major change in transaction rate, the current SCN would still remain below the current maximum SCN limit and database links between newer (or patched) and unpatched databases would continue to work. The patches provide the safety measure to ensure that you don’t have any issue with dblinks independent of any possible future change in your transaction rate.

With the patches applied, this change in current maximum SCN limit will happen automatically starting 23rd June 2019.

That is important information! So with the patch applied to some databases and not applied to some other databases and patched and non-patched databases have database links between them, everything should remain working. This is true for any database at this moment, because the change of the limit will happen on the 23rd of June 2019, which at the time of writing is in the future.

Once the change of limit has happened at the 23rd of June 2019, database links between older, non-patched versions of the database and newer or patched versions of the database could be affected if after the 23rd of June 2019 the SCN rate is increased in a newer or patched database and it runs beyond the headroom available in databases with the 16k rate, because the earlier mentioned SCN synchronisation then will fail because it’s beyond the 16k rate database headroom meaning it can not synchronise with the newer dataebase.

So the problem we are talking about here is two databases talking with each other over a database link, which have a different RSL, which could lead to a situation that one database is at an SCN number which is too high for the other older or non-patched database, meaning the communication will fail, which will persist until the older or non-patched databases is able to reach the SCN number of the newer database over time, if that is possible at all.

A thing that is not clear to me at this point: the documentation to me seems to suggest that Oracle version 12.2 with compatibility set to 12.2 or higher versions already allows the higher number of SCNs per second. (the above point: – – From version 12.2 on, with compatibility set to 12.2 or higher, the limit is 98304 (96k) SCNs per second calculated by the number of seconds from Januari 1st, 2008) If that is true, the issue that is warned for could potentially already happen!

Luckily, there is are ways to investigate this:

The reported “newer” versions and the versions that are patched for the rate change have an SGA variable “kcmscnc_” that lists the current SCN rate of the database. There are 3 values that I have seen for “kcmscnc_”:
1: This is the rate of 16k per second since Januari 1st 1988.
2: ?
3: This is the rate of 96k per second since Januari 1st 2008.

SQL> select ksmfsnam, ksmfsadr, ksmfssiz from x$ksmfsv where ksmfsnam like 'kcmscnc_';
KSMFSNAM							 KSMFSADR	    KSMFSSIZ
---------------------------------------------------------------- ---------------- ----------
kcmscnc_							 000000006001579C	   4
SQL> oradebug peek 0x6001579C 4
[06001579C, 0600157A0) = 00000001

So, this databases is capable of switching (because kcmscnc_ exists), and is currently running with the 16k per second threshold.

In fact, I tested this on 11.2.0.4, 12.1.0.2 and 18.3.0.0, all these version report currently (currently is before June 2019) “1” or “scheme 1”. So the above mentioned rate of 96k for 12.2 and above does currently not apply for the soft limit or SCN headroom calculation for any database, including 12.2 and 18.3.

There is a less “hardcore” way to obtain this information, via the DBMS_SCN. This package equally to the “kcmscnc_” variable only exists if the database is of a version or patch version high enough to switch:

declare
  v_rsl number;
  v_headroom_in_scn number;
  v_headroom_in_sec number;
  v_cur_scn_compat number;
  v_max_scn_compat number;
begin
  dbms_scn.getcurrentscnparams(v_rsl, v_headroom_in_scn, v_headroom_in_sec, v_cur_scn_compat, v_max_scn_compat);
  dbms_output.put_line('reasonable scn limit (soft limit): '||to_char(v_rsl,'999,999,999,999,999,999'));
  dbms_output.put_line('headroom in scn                  : '||to_char(v_headroom_in_scn,'999,999,999,999,999,999'));
  dbms_output.put_line('headroom in sec                  : '||v_headroom_in_sec);
  dbms_output.put_line('current scn compatibility scheme : '||v_cur_scn_compat);
  dbms_output.put_line('max scn compatibility scheme     : '||v_max_scn_compat);
end;
/

SQL> /
reasonable scn limit (soft limit):	 16,104,861,483,008
headroom in scn 		 :	 16,104,860,802,471
headroom in sec 		 : 982962695
current scn compatibility scheme : 1
max scn compatibility scheme	 : 3

PL/SQL procedure successfully completed.

This is executed in a version 18.3.0.0 database. So the conclusion here is that currently all versions up to 18.3.0.0 are still compatible, because they all use the same SCN limit per second, which is referred to as ‘scheme 1’. However, on June 23, 2019 newer versions of the database will switch to a new scheme, which is referred to as ‘scheme 3’, which allows an more headroom.

First of all, I hope a lot of databases out there have enough headroom in the first place, and a modest rate of SCNs it is using per second, which means it doesn’t advance into the RSL. In such a case, when you got older versions that can not switch to scheme 3 and newer versions that will, I can see no reason to worry at all.

The second case is when your database is close to running out of headroom currently in scheme 1, and the SCN rate in the database is also close the limit, so you truly should worry when your database switches to scheme 3, it might actually run over the 16k per second limit, and when it does that long enough to run over RSL of scheme 1, communication over a database link between the current scheme 3 database with a scheme 1 database will be disrupted until the scheme 3 database SCN is low enough again to fit the scheme 1 RSL. If the SCN rate persists in the scheme 3 database, communication will be impossible between the scheme 1 and a scheme 3 database.

So, at this point it should be clear that the most important statistic for determining issues between scheme 1 and after June 2019 scheme 3 databases is the current headroom in your databases. For all of the databases involved that will have a database link between a scheme 1 and a scheme 3 database, you should investigate their SCN number and SCN rate. If some of these databases have run into the soft limit ora-600, ora 600 [2252], these are potential candidates for running over the soft limit when they switch to scheme 3.

However, the most important statistic is to see how close the current SCN is to the scheme 1 soft limit. This can be done using the following SQL (this SQL does not need a newer version of the database, and is tested with version 11.2.0.2 and higher):

col "RSL scheme 1" format 9,999,999,999,999,999
col "current value" format 9,999,999,999,999,999
select dbms_flashback.get_system_change_number "current value",
       ((((to_number(to_char(sysdate,'YYYY'))-1988)*12*31*24*60*60) +
       ((to_number(to_char(sysdate,'MM'))-1)*31*24*60*60) +
       (((to_number(to_char(sysdate,'DD'))-1))*24*60*60) +
       (to_number(to_char(sysdate,'HH24'))*60*60) +
       (to_number(to_char(sysdate,'MI'))*60) +
       (to_number(to_char(sysdate,'SS')))) * (16*1024)) "RSL scheme 1",
       round(dbms_flashback.get_system_change_number/((((to_number(to_char(sysdate,'YYYY'))-1988)*12*31*24*60*60) +
       ((to_number(to_char(sysdate,'MM'))-1)*31*24*60*60) +
       (((to_number(to_char(sysdate,'DD'))-1))*24*60*60) +
       (to_number(to_char(sysdate,'HH24'))*60*60) +
       (to_number(to_char(sysdate,'MI'))*60) +
       (to_number(to_char(sysdate,'SS')))) * (16*1024))*100,5) "% to RSL scheme 1"
from dual;

If a database is close the RSL (roughly speaking beyond 90%-95%), the next thing to do is measure if the database keeps on using SCNs and keeps on being close to the RSL. If that is true, an additional increase in SCN usage could in the current situation using scheme 1 lead to an ora-600 [2252], but if that database has switched to scheme 3 after June 2019, there will not be anything keeping that database from going beyond an SCN number that will exceed the RSL of scheme 1, which will then cause issues if that database has a database link with a scheme 1 database.

Is there anything you can do if you suspect or know a database will go over the scheme 1 RSL? Purely for this issue, the obvious solution would be to make sure you are on a version that will switch to scheme 3 on June 2019, so at least after June 2019 it will not run into ora-600 [2252].

However, if such a scheme 3 database needs to connect to an older scheme 1 database, you have two choices:
1. Potentially run over the the scheme 1 limit and disrupt the database link communication.
2. Stop a newer database from switching to scheme 3, potentially disrupt changes in the current database, but it guarantees the database link will always work.

To look into the switch to scheme 3, which oracle calls ‘auto rollover’, the following SQL can be used:

declare
  v_autorollover_date date;
  v_target_compat number;
  v_is_enabled boolean;
begin
  dbms_scn.getscnautorolloverparams(v_autorollover_date, v_target_compat, v_is_enabled);
  dbms_output.put_line('auto rollover date      : '||to_char(v_autorollover_date,'YYYY-MM-DD'));
  dbms_output.put_line('target scheme	        : '||v_target_compat);
  dbms_output.put_line('rollover enabled (1=yes): '||sys.diutil.bool_to_int(v_is_enabled));
end;
/
SQL> /
auto rollover date	: 2019-06-23
target scheme		: 3
rollover enabled (1=yes): 1

PL/SQL procedure successfully completed.

If you want to prevent a database from rolling over to scheme 3, the procedure dbms_scn.disableautorollover can be used:

exec dbms_scn.disableautorollover;

Obviously, the procedure enableautorollover does the opposite. Please mind to contact Oracle support before doing this with your production database, this is an undocumented procedure at this time.

Also mind that if you create a new database after June 23, 2019, with a new or patched version that can switch to scheme 3, it will probably be running scheme 3 by default. If you want to be absolutely sure it will not exceed the scheme 1 limit, you can revert it to scheme 1 manually using the alter database set scn compatibility N command in mount mode:

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1048574496 bytes
Fixed Size		    8665632 bytes
Variable Size		  281018368 bytes
Database Buffers	  616562688 bytes
Redo Buffers		  142327808 bytes
Database mounted.
SQL> alter database set scn compatibility 1;

Database altered.

SQL> alter database open;

Database altered.

For this too I would strongly advise to contact Oracle support first. The purpose of this blogpost is to define the problem, show all the technical details that have to do with it, and show all the tools that are part of it. There is in no way anything in this article to tell you what you should do, it just shows everything that surrounds the switch to scheme 3 in June 2019.

Another view that might be beneficial is x$kcmscn. This view seems to be created to help looking if a scheme 3 database can connect to a scheme 1 database:

col cur_max_scn format 999,999,999,999,999
col pre_11_2_0_2_cur_max_scn format 999,999,999,999,999
select * from x$ksmscn;
ADDR		       INDX    INST_ID	   CON_ID CUR_MAX_RATE	  CUR_SCN
---------------- ---------- ---------- ---------- ------------ ----------
	 CUR_MAX_SCN DIFF_IN_DAYS DIFF_IN_HOURS PRE_11_2_0_2_CUR_MAX_SCN
-------------------- ------------ ------------- ------------------------
PRE_11_2_0_2_DIFF_IN_DAYS
-------------------------
00007F773DEDAE10	  0	     1		0	     0	   800918
  16,108,830,064,640	    11379	 273112       16,108,830,064,640
		186444783

Because currently (before June 2019) every database by default will be in scheme 1, the cur_max_scn and pre_11_2_0_2_cur_max_scn are identical. I even believe the column naming is wrong, the first version that can switch if it is patched to a high enough PSU version is 11.2.0.3, I do believe the column name is suggesting scheme 1 databases are databases of a version lower than 11.2.0.2, not including 11.2.0.2.

Conclusion.
I think there’s been a lot of fuzz for something that in most cases is not an issue. This article is supposed to give you all the knowledge and the tools to determine how it looks like in your situation.

This might be an issue if you happen to have one or more databases that are high on SCN numbering, and continues to take a lot of SCN numbers, and will be converted to a scheme 3 database on June 29, 2019 and is suspected to increase on taking SCN numbers for whatever reason AND it has a database link to a scheme 1 database that remains scheme 1. That’s a lot of ifs.

On the other hand you only need one database to be high in SCN numbering which continues to take a lot of SCNs keeping it close to the soft limit, which will propagate its SCN to other databases if it is linked, or the required properties of the problem spread out over multiple linked databases.

Again, I do not advise anything in this article, the purpose here is to provide all the details that surround it so you can make the best decision for yourself.

Starting from Oracle 12, in a default configured database, there are more log writer processes than the well known ‘LGWR’ process itself, which are the ‘LGnn’ processes:

$ ps -ef | grep test | grep lg
oracle   18048     1  0 12:50 ?        00:00:13 ora_lgwr_test
oracle   18052     1  0 12:50 ?        00:00:06 ora_lg00_test
oracle   18056     1  0 12:50 ?        00:00:00 ora_lg01_test

These are the log writer worker processes, for which the minimal amount is equal to the amount public redo strands. Worker processes are assigned to a group, and the group is assigned to a public redo strand. The amount of worker processes in the group is dependent on the undocumented parameter “_max_log_write_parallelism”, which is one by default.

The actual usage of the worker processes is dependent in the first place on the value of the undocumented parameter “_use_single_log_writer”, for which the default value is ‘ADAPTIVE’, which means it’s switching automatically between ‘single log writer mode’, which is the traditional way of the LGWR process handling everything that the log writer functionality needs to do, and the ‘scalable log writer mode’, which means the log writer functionality is presumably using the log writer worker processes.

Other values for “_use_single_log_writer” are ‘TRUE’ to set ‘single log writer mode’, or ‘FALSE’ to set ‘scalable log writer mode’ fixed.

I assume most readers of this blog will know that the master log writer idle work cycle is sleeping on a semaphore (semtimedop()) under the wait event ‘rdbms ipc message’ for 3 seconds, then performs some “housekeeping”, after which it’ll sleep again repeating the small cycle of sleeping and housekeeping. For the log writer worker processes, this looks different if you look at the wait event information of the log writer worker processes:

135,59779,@1    14346                    DEDICATED oracle@memory-presentation.local (LGWR)	    time:1909.44ms,event:rdbms ipc message,seq#:292
48,34282,@1     14350                    DEDICATED oracle@memory-presentation.local (LG00)	    time:57561.85ms,event:LGWR worker group idle,seq#:150
136,24935,@1    14354                    DEDICATED oracle@memory-presentation.local (LG01)	    time:112785.66ms,event:LGWR worker group idle,seq#:74

The master log writer process (LGWR) has been sleeping for 1.9s when I queried the database, and it will sleep for 3 seconds, and then do some work and sleep again. However, the log writer worker processes have been sleeping for much longer: LG00 for 57.6s and LG01 for 112.8s, and the event is different: ‘LGWR worker group idle’. How is this implemented? Let’s look!

$ strace -p $(pgrep lg01)
strace: Process 14354 attached
semtimedop(360448, [{27, -1, 0}], 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)
semtimedop(360448, [{27, -1, 0}], 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)

I used strace on the LG01 process, and it’s still doing the same as most idle background processes are doing: sleeping on a semaphore for 3 seconds. But, it does not end its wait like LGWR does, the event the log writer worker processes are waiting in keeps on being timed.

Using a pin tools debugtrace shows the following:

 | | < semtimedop+0x000000000023 returns: 0xffffffffffffffff
 | | > __errno_location(0x38000, 0x7ffce278c328, ...)
 | | | > fthread_self(0x38000, 0x7ffce278c328, ...)
 | | | < fthread_self+0x000000000024 returns: 0
 | | < __errno_location+0x000000000010 returns: 0x7f7e930a26a0
 | < sskgpwwait+0x00000000014e returns: 0
 < skgpwwait+0x0000000000e0 returns: 0
 > ksuSdiInProgress(0x19e80, 0x19e80, ...)
 < ksuSdiInProgress+0x000000000035 returns: 0
 > sltrgftime64(0x19e80, 0x19e80, ...)
 | > clock_gettime@plt(0x1, 0x7ffce278c3a0, ...)
 | | > clock_gettime(0x1, 0x7ffce278c3a0, ...)
 | | < clock_gettime+0x000000000069 returns: 0
 | < clock_gettime+0x00000000003a returns: 0
 < sltrgftime64+0x00000000004c returns: 0x19c253f3ff
 > kslwo_getcbk(0xa2, 0xd80fa62, ...)
 < kslwo_getcbk+0x000000000017 returns: 0
 > kgslwait_last_waitctx_time_waited_usecs(0x7f7e930a29a0, 0x6dfd01c0, ...)
 < kgslwait_last_waitctx_time_waited_usecs+0x000000000045 returns: 0x25e5e80
 > kskiorm(0x6d1854a8, 0, ...)
 < kskiorm+0x00000000001e returns: 0
 > kfias_iswtgon_ksfd(0x6d1854a8, 0, ...)
 < kfias_iswtgon_ksfd+0x00000000002b returns: 0
 > kxdbio_has_work(0x7ffce278c3c4, 0x6003d010, ...)
 < kxdbio_has_work+0x000000000027 returns: 0
 > skgpwwait(0x7ffce278c630, 0x7f7e930a7ca0, ...)
 | > kslwait_conv_wait_time(0x2dc6c0, 0x7f7e930a7ca0, ...)
 | < kslwait_conv_wait_time+0x000000000027 returns: 0x2dc6c0
 | > sskgpwwait(0x7ffce278c630, 0x7f7e930a7ca0, ...)
 | | > semtimedop(0x38000, 0x7ffce278c328, ...)
 | | < semtimedop+0x000000000023 returns: 0xffffffffffffffff

And a full stack trace of a log writer worker look like this:

$ pstack $(pgrep lg01)
#0  0x00007feda8eaebda in semtimedop () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000000010f9cca6 in sskgpwwait ()
#2  0x0000000010f9a2e8 in skgpwwait ()
#3  0x0000000010a66995 in ksliwat ()
#4  0x0000000010a65d25 in kslwaitctx ()
#5  0x00000000031fb4d0 in kcrfw_slave_queue_remove ()
#6  0x00000000031fad2a in kcrfw_slave_group_main ()
#7  0x00000000012160fa in ksvrdp_int ()
#8  0x000000000370d99a in opirip ()
#9  0x0000000001eb034a in opidrv ()
#10 0x0000000002afedf1 in sou2o ()
#11 0x0000000000d0547a in opimai_real ()
#12 0x0000000002b09b31 in ssthrdmain ()
#13 0x0000000000d05386 in main ()

If you combine the pstack backtrace and the debugtrace information, you see that the idle cycle does not leave the ‘ksliwat’ function, so the wait event is not finished. Quickly looking at the other functions, it’s easy to spot it reads the system clock (sltrgftime64), updates some information (kgslwait_last_waitctx_time_waited_usecs) and then performs some proactive IO checks (kskiorm, kfias_iswtgon_ksfd, kxdbio_has_work) after which it calls the post/wait based functions to setup the semaphore again.

Conclusion so far is the log writer workers do perform a 3 second sleep just like the master log writer, however the wait event ‘LGWR worker group idle’ is not interrupted like ‘rdbms ipc message’ is for the master log writer. This means the wait time for the event for each worker process indicates the last time the worker process actually performed something. A next logical question then is: but what do the log writer worker processes perform? Do they entirely take over the master log writer functionality, or do they work together with the master log writer?

In order to fully understand the next part, it is very beneficial to read up on how the log writer works in ‘single log writer’ mode, where the master log writer handling the idle and work cycle itself:
https://fritshoogland.wordpress.com/2018/02/20/a-look-into-into-oracle-redo-part-4-the-log-writer-null-write/
https://fritshoogland.wordpress.com/2018/02/27/a-look-into-oracle-redo-part-5-the-log-writer-writing/

If you want to perform this investigation yourself, make sure the database is in ‘scalable log writer’ mode, by setting “_use_single_log_writer” to FALSE. This is exactly what I did in order to make sure a log write is done in ‘scalable log writer’ mode.

Now let’s first apply some logic. Above the idle cycle of a log writer worker process is shown. Based on the ‘log writer null write’ blog post, we know that the log writer does advance the LWN and On-disk SCN every 3 seconds. Clearly, the log writer worker process does not do that. So that must mean the master log writer is still performing that function. It would also make very much sense, because it doesn’t matter for scalability if the master log writer performs the function of advancing the LWN and On-disk SCN or a worker process, nothing is waiting on it. Plus, if the master log writer performs most of its functions just like in ‘single log writer’ mode, the change to scalable mode would mean no change for client processes, any committing process must semop() the log writer to start writing.

Let’s look at the relevant debugtrace output of the master log writer in scalable log writer mode:

 | > kcrfw_redo_write_driver(0, 0, ...)
 | | > kcrfw_handle_member_write_errors(0, 0, ...)
 | | < kcrfw_handle_member_write_errors+0x000000000020 returns: 0x600161a0
 | | > kcmgtsf(0, 0, ...)
 | | | > sltrgatime64(0, 0, ...)
 | | | | > sltrgftime64(0, 0, ...)
 | | | | | > clock_gettime@plt(0x1, 0x7fff1fe13010, ...)
 | | | | | | > clock_gettime(0x1, 0x7fff1fe13010, ...)
 | | | | | | < clock_gettime+0x000000000069 returns: 0
 | | | | | < clock_gettime+0x00000000003a returns: 0
 | | | | < sltrgftime64+0x00000000004c returns: 0x53747fe42
 | | | < sltrgatime64+0x00000000003e returns: 0x155d4fd
 | | < kcmgtsf+0x00000000032f returns: 0x3a182314
 | | > kcrfw_slave_adaptive_updatemode(0, 0x600161a0, ...)
 | | < kcrfw_slave_adaptive_updatemode+0x000000000080 returns: 0x7efe34d1f760
 | | > kcrfw_defer_write(0, 0x600161a0, ...)
 | | < kcrfw_defer_write+0x000000000038 returns: 0x7efe34d1f760
 | | > kcrfw_slave_queue_find(0, 0x600161a0, ...)
 | | < kcrfw_slave_queue_find+0x0000000000f1 returns: 0
 | | > kcrfw_slave_queue_setpreparing(0, 0x1, ...)
 | | < kcrfw_slave_queue_setpreparing+0x000000000021 returns: 0
 | | > kcrfw_slave_group_switchpic(0, 0x1, ...)
 | | < kcrfw_slave_group_switchpic+0x000000000050 returns: 0x699b4508
 | | > skgstmGetEpochTs(0, 0x1, ...)
 | | | > gettimeofday@plt(0x7fff1fe13070, 0, ...)
 | | | < __vdso_gettimeofday+0x0000000000fe returns: 0
 | | < skgstmGetEpochTs+0x000000000049 returns: 0x20debfd6192e5
 | | > kcsnew3(0x600113b8, 0x7fff1fe13228, ...)
 | | | > kcsnew8(0x600113b8, 0x7fff1fe13070, ...)
 | | | | > kslgetl(0x60049800, 0x1, ...)
 | | | | < kslgetl+0x00000000012f returns: 0x1
 | | | | > kslfre(0x60049800, 0x1, ...)
 | | | | < kslfre+0x0000000001e2 returns: 0
 | | | < kcsnew8+0x000000000117 returns: 0
 | | | > ub8_to_kscn_impl(0x66c3c7, 0x7fff1fe13228, ...)
 | | | < ub8_to_kscn_impl+0x000000000031 returns: 0
 | | < kcsnew3+0x00000000006f returns: 0x8000
 | | > ktfwtsm(0x3a182314, 0x7fff1fe13228, ...)
 | | | > kcmgtsf(0x2, 0x7fff1fe13228, ...)
 | | | | > sltrgatime64(0x2, 0x7fff1fe13228, ...)
 | | | | | > sltrgftime64(0x2, 0x7fff1fe13228, ...)
 | | | | | | > clock_gettime@plt(0x1, 0x7fff1fe12fe0, ...)
 | | | | | | | > clock_gettime(0x1, 0x7fff1fe12fe0, ...)
 | | | | | | | < clock_gettime+0x000000000069 returns: 0
 | | | | | | < clock_gettime+0x00000000003a returns: 0
 | | | | | < sltrgftime64+0x00000000004c returns: 0x537484a6d
 | | | | < sltrgatime64+0x00000000003e returns: 0x155d511
 | | | < kcmgtsf+0x0000000001b2 returns: 0x3a182314
 | | | > kcmtdif(0x3a182314, 0x3a182314, ...)
 | | | < kcmtdif+0x00000000001b returns: 0
 | | | > ksl_get_shared_latch_int(0x60050340, 0x6ddb1408, ...)
 | | | < ksl_get_shared_latch_int+0x00000000016b returns: 0x1
 | | <> kslfre(0x60050340, 0x66c3c7, ...)
 | | < kslfre+0x0000000001e2 returns: 0
 | | > kcn_stm_write(0x7fff1fe13228, 0x66c3c7, ...)
 | | | > kstmgetsectick(0x7fff1fe13228, 0x66c3c7, ...)
 | | | < kstmgetsectick+0x00000000003a returns: 0x5ae4c494
 | | | > ksl_get_shared_latch_int(0x6004ee40, 0x6ddb1408, ...)
 | | | < ksl_get_shared_latch_int+0x00000000016b returns: 0x1
 | | <> kslfre(0x6004ee40, 0x2244, ...)
 | | < kslfre+0x0000000001e2 returns: 0
 | | > kcrfw_redo_write_initpic(0x699b4508, 0x7fff1fe13228, ...)
 | | | > kscn_to_ub8_impl(0x7fff1fe13228, 0x7fff1fe13228, ...)
 | | | < kscn_to_ub8_impl+0x00000000003e returns: 0x66c3c7
 | | < kcrfw_redo_write_initpic+0x0000000000dc returns: 0x3a182314
 | | > kscn_to_ub8_impl(0x7fff1fe13228, 0, ...)
 | | < kscn_to_ub8_impl+0x00000000003e returns: 0x66c3c7
 | | > kcrfw_gather_lwn(0x7fff1fe13268, 0x699b4508, ...)
 | | | > kslgetl(0x6abe4538, 0x1, ...)
 | | | < kslgetl+0x00000000012f returns: 0x1
 | | | > kcrfw_gather_strand(0x7fff1fe13268, 0, ...)
 | | | < kcrfw_gather_strand+0x0000000000c2 returns: 0
 | | | > kslfre(0x6abe4538, 0x17d5f, ...)
 | | | < kslfre+0x0000000001e2 returns: 0
 | | | > kslgetl(0x6abe45d8, 0x1, ...)
 | | | < kslgetl+0x00000000012f returns: 0x1
 | | | > kcrfw_gather_strand(0x7fff1fe13268, 0x1, ...)
 | | | < kcrfw_gather_strand+0x0000000000c2 returns: 0
 | | | > kslfre(0x6abe45d8, 0x137, ...)
 | | | < kslfre+0x0000000001e2 returns: 0
 | | < kcrfw_gather_lwn+0x00000000065c returns: 0xffffffff
 | | > krsh_trace(0x1000, 0x200, ...)
 | | < krsh_trace+0x00000000005d returns: 0
 | | > kspgip(0x71e, 0x1, ...)
 | | < kspgip+0x00000000023f returns: 0
 | | > kcrfw_slave_queue_setpreparing(0, 0, ...)
 | | < kcrfw_slave_queue_setpreparing+0x000000000021 returns: 0
 | | > kcrfw_slave_queue_flush_internal(0x1, 0, ...)
 | | < kcrfw_slave_queue_flush_internal+0x0000000000d7 returns: 0x1
 | | > kcrfw_do_null_write(0, 0, ...)
 | | | > kcrfw_slave_phase_batchdo(0, 0, ...)
 | | | | > kcrfw_slave_phase_enter(0, 0x9b, ...)
 | | | | < kcrfw_slave_phase_enter+0x000000000449 returns: 0
 | | | <> kcrfw_slave_phase_exit(0, 0x9b, ...)
 | | | < kcrfw_slave_phase_exit+0x00000000035a returns: 0
 | | | > kcrfw_post(0, 0, ...)
 | | | | > kcrfw_slave_single_getactivegroup(0, 0, ...)
 | | | | < kcrfw_slave_single_getactivegroup+0x000000000047 returns: 0x6a9a0718
 | | | | > kspGetInstType(0x1, 0x1, ...)
 | | | | | > vsnffe_internal(0x19, 0x1, ...)
 | | | | | | > vsnfprd(0x19, 0x1, ...)
 | | | | | | < vsnfprd+0x00000000000f returns: 0x8
 | | | | | | > kfIsASMOn(0x19, 0x1, ...)
 | | | | | | <> kfOsmInstanceSafe(0x19, 0x1, ...)
 | | | | | | < kfOsmInstanceSafe+0x000000000031 returns: 0
 | | | | | < vsnffe_internal+0x0000000000a7 returns: 0
 | | | | | > kspges(0x115, 0x1, ...)
 | | | | | < kspges+0x00000000010f returns: 0
 | | | | < kspGetInstType+0x0000000000b1 returns: 0x1
 | | | | > kcrfw_slave_phase_enter(0x1, 0x9b, ...)
 | | | | < kcrfw_slave_phase_enter+0x00000000006f returns: 0x9b
 | | | | > kcscu8(0x60016290, 0x7fff1fe12f98, ...)
 | | | | < kcscu8+0x000000000047 returns: 0x1
 | | | | > kcsaj8(0x60016290, 0x7fff1fe12f38, ...)
 | | | | < kcsaj8+0x0000000000dc returns: 0x1
 | | | | > kcrfw_slave_phase_exit(0x1, 0x9b, ...)
 | | | | < kcrfw_slave_phase_exit+0x00000000008e returns: 0
 | | | | > kslpsemf(0x97, 0, ...)
 | | | | | > ksl_postm_init(0x7fff1fe0ac30, 0x7fff1fe12c50, ...)
 | | | | | < ksl_postm_init+0x00000000002b returns: 0
 | | | | < kslpsemf+0x0000000006b5 returns: 0x1f
 | | | | > kcrfw_slave_barrier_nonmasterwait(0x6a9a0720, 0x4, ...)
 | | | | < kcrfw_slave_barrier_nonmasterwait+0x000000000035 returns: 0x600161a0
 | | | < kcrfw_post+0x000000000c1c returns: 0xd3
 | | < kcrfw_do_null_write+0x0000000000b2 returns: 0xd3
 | < kcrfw_redo_write_driver+0x000000000535 returns: 0xd3

The highlighted functions are extra functions executed when the instance is set to scalable log writer mode, or when adaptive mode has set the instance to scalable log writer mode. This means that the changes between the modes is minimal when there’s no writes, and outside of a few extra functions, the log writer does exactly the same.

The absence of any spectacular changes in the behaviour of the log writer when in scalable log writer mode when there are no writes does hint what the actual changes will be of the scalable mode, which is how writing is handled. In single log writer mode, the most time the log writer is process is likely to spend on is writing the change vectors into the online redologfiles, and maybe, if you have a bad application (!) semop()-ing foreground sessions will be second, if there are a large number of processes committing, because every process needs to be semop()-ed individually. These two functions, along with some other functionality are exactly what the log writer worker processes are doing.

This means that foreground processes do nothing different in scalable log writer mode, they signal (semop) the master log writer, which will investigate the public redo strands, and if the master log writer finds change vectors to write, it will assign log writer worker processes to perform the write, and the log writer worker process will semop() the foreground sessions to indicate the redo has been written when the instance is in post/wait mode, or do not semop() when the instance is in polling mode.

This is the entire function flow of a write when the instance is in scalable log writer mode:

 | > kcrfw_slave_queue_insert(0, 0xd3, ...)
 | | > kcrfw_slave_group_setcurrsize(0, 0, ...)
 | | < kcrfw_slave_group_setcurrsize+0x0000000001d1 returns: 0x1
 | | > _intel_fast_memcpy(0x6a9a05f8, 0x7ffdae335fa0, ...)
 | | <> _intel_fast_memcpy.P(0x6a9a05f8, 0x7ffdae335fa0, ...)
 | | <> __intel_ssse3_rep_memcpy(0x6a9a05f8, 0x7ffdae335fa0, ...)
 | | < __intel_ssse3_rep_memcpy+0x000000002798 returns: 0x6a9a05f8
 | | > kcrfw_slave_group_postall(0, 0xf0, ...)
 | | | > ksvgcls(0, 0xf0, ...)
 | | | < ksvgcls+0x000000000021 returns: 0
 | | | > ksl_post_proc(0x6ddb32f0, 0, ...)
 | | | <> kskpthr(0x6ddb32f0, 0, ...)
 | | | <> kslpsprns(0x6ddb32f0, 0, ...)
 | | | | > ksl_update_post_stats(0x6ddb32f0, 0, ...)
 | | | | | > dbgtTrcData_int(0x7f464c0676c0, 0x2050031, ...)
 | | | | | | > dbgtBucketRedirect(0x7f464c0676c0, 0x7ffdae335338, ...)
 | | | | | | < dbgtBucketRedirect+0x000000000050 returns: 0x1
 | | | | | | > dbgtIncInMemTrcRedirect(0x7f464c0676c0, 0x6fa, ...)
 | | | | | | < dbgtIncInMemTrcRedirect+0x000000000035 returns: 0x1
 | | | | | | > skgstmGetEpochTs(0x7f464c0676c0, 0x6fa, ...)
 | | | | | | | > gettimeofday@plt(0x7ffdae334e40, 0, ...)
 | | | | | | | < __vdso_gettimeofday+0x0000000000fe returns: 0
 | | | | | | < skgstmGetEpochTs+0x000000000049 returns: 0x20e067375b55d
 | | | | | | > dbgtrRecAllocate(0x7f464c0676c0, 0x7ffdae3352e0, ...)
 | | | | | | | > dbgtrPrepareWrite(0x7f464c0676c0, 0x65accba0, ...)
 | | | | | | | < dbgtrPrepareWrite+0x00000000011c returns: 0x4
 | | | | | | < dbgtrRecAllocate+0x000000000144 returns: 0x1
 | | | | | | > _intel_fast_memcpy(0x65acda30, 0x7ffdae3353d8, ...)
 | | | | | | <> _intel_fast_memcpy.P(0x65acda30, 0x7ffdae3353d8, ...)
 | | | | | | <> __intel_ssse3_rep_memcpy(0x65acda30, 0x7ffdae3353d8, ...)
 | | | | | | < __intel_ssse3_rep_memcpy+0x000000002030 returns: 0x65acda30
 | | | | | | > dbgtrRecEndSegment(0x7f464c0676c0, 0x7ffdae3352e0, ...)
 | | | | | | < dbgtrRecEndSegment+0x00000000011c returns: 0x77c000a4
 | | | | | < dbgtTrcData_int+0x000000000323 returns: 0x77c000a4
 | | | | < ksl_update_post_stats+0x00000000024f returns: 0x77c000a4
 | | | | > skgpwpost(0x7ffdae335480, 0x7f464c0acca0, ...)
 | | | | <> sskgpwpost(0x7ffdae335480, 0x7f464c0acca0, ...)
 | | | | | > semop@plt(0xc0000, 0x7ffdae335410, ...)
 | | | | | < semop+0x00000000000f returns: 0
 | | | | < sskgpwpost+0x00000000009a returns: 0x1
 | | | < kslpsprns+0x0000000001c3 returns: 0
 | | < kcrfw_slave_group_postall+0x0000000000a8 returns: 0
 | < kcrfw_slave_queue_insert+0x0000000001b6 returns: 0x667bc540

After the instance has established there are change vectors in kcrfw_gather_lwn, in single log writer mode, the function kcrfw_redo_write is called, which will call kcrfw_do_write which handles the writing, and kslpslf to semop any waiting processes among other things. Now in scalable log writer mode, kcrfw_slave_queue_insert is called which assigns work to worker processes, and then kcrfw_slave_group_postall is called to semop one or more worker processes.

The worker processes are sleeping on a semaphore, and if a process gets signalled, it exits the kcrfw_slave_queue_remove function, ends the wait event, and calls kcrfw_redo_write, just like the master log writer process would call in single log writer mode, which includes doing the write (kcrfw_do_write) and posting the foregrounds (kslpslf), exactly all the functions.

Conclusion.
The adaptive scalable log writer processes function has been silently introduced with Oracle 12, although a lot of the used functionality has been available more or less in earlier versions. It is a fully automatic feature which will turn itself on and off based on heuristics. The purpose of this article is to explain how it works and what it is doing. Essentially, all the functionality that surrounds a log writer write has been moved to a worker process, which means the work can be done in parallel with multiple processes, whilst all the work outside of the work around the write, which is not performance critical, is left with the master log writer.

I gotten some requests to provide an overview of the redo series of blogposts I am currently running. Here it is:

https://fritshoogland.wordpress.com/2018/01/29/a-look-into-oracle-redo-part-1-redo-allocation-latches/
https://fritshoogland.wordpress.com/2018/02/05/a-look-into-oracle-redo-part-2-the-discovery-of-the-kcrfa-structure/
https://fritshoogland.wordpress.com/2018/02/12/a-look-into-oracle-redo-part-3-the-log-writer-work-cycle-overview/
https://fritshoogland.wordpress.com/2018/02/20/a-look-into-into-oracle-redo-part-4-the-log-writer-null-write/
https://fritshoogland.wordpress.com/2018/02/27/a-look-into-oracle-redo-part-5-the-log-writer-writing/
https://fritshoogland.wordpress.com/2018/03/05/a-look-into-oracle-redo-part-6-oracle-post-wait-commit-and-the-on-disk-scn/
https://fritshoogland.wordpress.com/2018/03/19/a-look-into-oracle-redo-part-7-adaptive-log-file-sync/
https://fritshoogland.wordpress.com/2018/03/26/a-look-into-oracle-redo-part-8-generate-redo/
https://fritshoogland.wordpress.com/2018/04/03/a-look-into-oracle-redo-part-9-commit/
https://fritshoogland.wordpress.com/2018/04/09/a-look-into-oracle-redo-part-9a-commit-concurrency-considerations/
https://fritshoogland.wordpress.com/2018/04/16/a-look-into-oracle-redo-part-10-commit_wait-and-commit_logging/

Private redo strands, In memory undo and throw away undo: https://fritshoogland.wordpress.com/2016/11/15/redo-a-blogpost/

The redo series would not be complete without writing about changing the behaviour of commit. There are two ways to change commit behaviour:

1. Changing waiting for the logwriter to get notified that the generated redo is persisted. The default is ‘wait’. This can be set to ‘nowait’.
2. Changing the way the logwriter handles generated redo. The default is ‘immediate’. This can be set to ‘batch’.

There are actually three ways these changes can be made:
1. As argument of the commit statement: ‘commit’ can be written as ‘commit write wait immediate’ (statement level).
2. As a system level setting. By omitting an explicit commit mode when executing the commit command, the setting as set with the parameters commit_wait (default: wait) and commit_logging (default: immediate).
3. As a session level setting. By omitting an explicit commit mode, but by setting either commit_wait or commit_logging it overrides the settings at the system level.

At this point I should say that in my personal opinion, if you need to change this, there is something very wrong with how the database is used in the first place. This can enhance performance a bit (totally depending on what you are doing and how your hardware looks like), but it does nothing magic, as you will see.

a) commit wait/nowait
I ran a pin tools debugtrace on a session that commits explicitly with the write mode explicitly set to wait (the default), and a session that commits explicitly with the write mode set to nowait. If you took the time to read the other redo related articles you know that a commit generates changes vectors that are written in the public redo strand, changes the transaction table in the undo segment header and then signals the logwriter to write in kcrf_commit_force_int, releases all transactional control on the rows in the transaction that are committed, after which kcrf_commit_force_int is called again in order to wait for the logwriter to get notified that the change vectors have been persisted.

When commit is set to nowait, actually what happens is very simple: everything that is executed in ‘wait mode’ commit is executed in ‘nowait mode’ too, except for calling the kcrf_commit_force_int a second time, which is the functionality to wait for the notification from the logwriter.

commit wait:

 | | < kpoal8+0x000000000f8c returns: 0x2
 | | > ksupop(0x1, 0x7a87a9a0, ...)
 | | | > ksugit_i(0x11526940, 0x7a87a9a0, ...)
 | | | < ksugit_i+0x00000000002a returns: 0
 | | | > _setjmp@plt(0x7ffda5959c50, 0x7a87a9a0, ...)
 | | | <> __sigsetjmp(0x7ffda5959c50, 0, ...)
 | | | <> __sigjmp_save(0x7ffda5959c50, 0, ...)
 | | | < __sigjmp_save+0x000000000025 returns: 0
 | | | > kcbdsy(0x7ffda5959c50, 0x7f3011cbc028, ...)
 | | | <> kcrf_commit_force_int(0x7f3011d75e10, 0x1, ...)
...
 | | | < kcrf_commit_force_int+0x000000000b9c returns: 0x1
 | | | > kslws_check_waitstack(0x3, 0x7f3011d82f40, ...)
 | | | < kslws_check_waitstack+0x000000000065 returns: 0
 | | | > kssdel(0x7a87a9a0, 0x1, ...)
 | | | | > kpdbUidToId(0, 0x1, ...)
 | | | | < kpdbUidToId+0x00000000014e returns: 0
 | | | | > kss_del_cb(0x7ffda5959b50, 0x7f3011d82f40, ...)
 | | | | | > kpdbUidToId(0, 0x7f3011d82f40, ...)
 | | | | | < kpdbUidToId+0x00000000014e returns: 0
 | | | | | > ksudlc(0x7a87a9a0, 0x1, ...)

commit nowait:

 | | < kpoal8+0x000000000f8c returns: 0x2
 | | > ksupop(0x1, 0x63c82a38, ...)
 | | | > ksugit_i(0x11526940, 0x63c82a38, ...)
 | | | < ksugit_i+0x00000000002a returns: 0
 | | | > _setjmp@plt(0x7fff43332a50, 0x63c82a38, ...)
 | | | <> __sigsetjmp(0x7fff43332a50, 0, ...)
 | | | <> __sigjmp_save(0x7fff43332a50, 0, ...)
 | | | < __sigjmp_save+0x000000000025 returns: 0
 | | | > kslws_check_waitstack(0x3, 0x7fd1cea22028, ...)
 | | | < kslws_check_waitstack+0x000000000065 returns: 0
 | | | > kssdel(0x63c82a38, 0x1, ...)
 | | | | > kpdbUidToId(0, 0x1, ...)
 | | | | < kpdbUidToId+0x00000000014e returns: 0
 | | | | > kss_del_cb(0x7fff43332950, 0x7fd1ceae8f40, ...)
 | | | | | > kpdbUidToId(0, 0x7fd1ceae8f40, ...)
 | | | | | < kpdbUidToId+0x00000000014e returns: 0
 | | | | | > ksudlc(0x63c82a38, 0x1, ...)

Yes, it’s that simple. In normal commit mode, commit wait, in ksupop (kernel service user pop (restore) user or recursive call) a call to kcbdsy is executed, which performs a tailcall to kcrf_commit_force_int. In nowait commit mode, kcbdsy is simply not called in ksupop, which actually exactly does what nowait means, the waiting for the logwriter notification is not done.

b) commit immediate/batch
I ran a pin tools debugtrace on a session that commits explicitly with the write mode explicitly set to immediate, and a session that commits explicitly with the write mode set to batch. If you read the other redo related articles you know that a commit generates changes vectors that are written in the public redo strand, changes the transaction table in the undo segment header and then signals the logwriter to write in kcrf_commit_force_int, then releases all transactional control on the rows in the transaction that are committed, after which kcrf_commit_force_int is called again in order to wait for the logwriter to get notified that the change vectors have been persisted.

When commit is set to batch, actually what happens is very simple: everything is done exactly the same in ‘immediate mode’ commit, except for calling the kcrf_commit_force_int the first time, which is the functionality that triggers the logwriter to write. So it looks like ‘batch mode’ is not explicitly batching writes for the logwriter, but rather the disablement of the signal to the logwriter to write right after the change vectors have been copied and the blocks are changed. But that is not all…

I noticed something weird when analysing the calls in the debugtrace of ‘commit write batch’: not only was the first invocation of kcrf_commit_force_int gone, the second invocation of kcrf_commit_force_int was also gone too! That is weird, because the Oracle documentation says:

WAIT | NOWAIT

Use these clauses to specify when control returns to the user.

The WAIT parameter ensures that the commit will return only after the corresponding redo is persistent in the online redo log. Whether in BATCH or IMMEDIATE mode, when the client receives a successful return from this COMMIT statement, the transaction has been committed to durable media. A crash occurring after a successful write to the log can prevent the success message from returning to the client. In this case the client cannot tell whether or not the transaction committed.

The NOWAIT parameter causes the commit to return to the client whether or not the write to the redo log has completed. This behavior can increase transaction throughput. With the WAIT parameter, if the commit message is received, then you can be sure that no data has been lost.

If you omit this clause, then the transaction commits with the WAIT behavior.

The important, and WRONG thing, is in the last line: ‘if you omit this clause, then the transaction commits with the WAIT behavior’. Actually, if the commit mode is set to batch, the commit wait mode flips to nowait with it. It does perform the ultimate batching, which is not sending a signal to the logwriter at all, so what happens is that change vectors in the public redo strands are written to disk by the logwriter only every 3 seconds, because that is the timeout for the logwriter sleeping on a semaphore, after which it obtains any potential redo to write via information in kcrfsg_ and KCRFA structures. This is important, because with NOWAIT behaviour, there is no guarantee changes have been persisted for the committing session.

I was surprised to find this, which for me it meant I was searching for ‘kcrf_commit_force_int’ in the debugtrace of a commit with the ‘write batch’ arguments, and did not find any of them. Actually, this has been reported by Marcin Przepiorowski in a comment on an article by Christian Antognini on this topic.

Can this commit batching be changed to include waiting for the logwriter? Yes, actually it can if you explicitly include ‘wait’ with the commit write batch. It is very interesting the kcrf_commit_force_int function then comes back at a totally different place:

 | | | | | | | | | | | | | < ktuulc+0x000000000119 returns: 0
 | | | | | | | | | | | | | > ktudnx(0x69fc8eb0, 0, ...)
 | | | | | | | | | | | | | | > ktuIMTabCacheCommittedTxn(0x69fc8eb0, 0x7ffe9eb79e74, ...)
 | | | | | | | | | | | | | | < ktuIMTabCacheCommittedTxn+0x000000000071 returns: 0
 | | | | | | | | | | | | | | > kslgetl(0x6ab9d6e8, 0x1, ...)
 | | | | | | | | | | | | | | < kslgetl+0x00000000012f returns: 0x1
 | | | | | | | | | | | | | | > kslfre(0x6ab9d6e8, 0x6ab9ce00, ...)
 | | | | | | | | | | | | | | < kslfre+0x0000000001e2 returns: 0
 | | | | | | | | | | | | | < ktudnx+0x0000000005e4 returns: 0
 | | | | | | | | | | | | | > ktuTempdnx(0x69fc8eb0, 0, ...)
 | | | | | | | | | | | | | < ktuTempdnx+0x000000000083 returns: 0
 | | | | | | | | | | | | | > kcb_sync_last_change(0x69fc8eb0, 0x6df64df8, ...)
 | | | | | | | | | | | | | <> kcrf_commit_force_int(0x7f525ba19c00, 0x1, ...)
...
 | | | | | | | | | | | | | < kcrf_commit_force_int+0x000000000b9c returns: 0x1
 | | | | | | | | | | | | | > kghstack_free(0x7f525bb359a0, 0x7f525690ead8, ...)
 | | | | | | | | | | | | | < kghstack_free+0x00000000005a returns: 0
 | | | | | | | | | | | | < ktucmt+0x000000000e0c returns: 0

Instead of simply keeping the separate call after the transaction in the ksupop function, described above with commit wait/nowait, which is kcrf_commit_force_int with second argument set to 1, which means it notifies the logwriter as well as waits for the logwriter notification of the write, it is now is called after the function to clear the TX enqueue (ktuulc) and the undo transaction count has been lowered (ktudnx) at the end of the ktucmt function as a tailcall of kcb_sync_last_change, which wasn’t called before. Of course this limits the IO batching opportunities.

Conclusion
Do not change your database or even your session to make your commit faster. If you must, read this article carefully and understand the trade offs. One trade off which hasn’t been highlighted is: this might change in a different version, and it requires some effort to investigate. And again: if you still are considering this: probably you have a different problem that you should look at. Do not take this option in desperation to hope for a magical restoration of performance.

The commit_write option nowait does trigger the logwriter to write (the first invocation of the kcrf_commit_force_int function), but it does not wait for write confirmation. The commit_logging option batch does something different than the documentation says it does, it does not issue a signal to the logwriter, nor wait for it. This way the logwriter can wait the full three seconds before it times out on its semaphore and write what is in the public redo strands. But there is no way to tell if the redo for your change has been persisted yet, because that wait is gone too (that wait is the infamous ‘log file sync’ wait). If you want batching but still want a write notification, you must set commit_write to wait explicitly. By doing that you do not get the optimal batching because then waiting for the logwriter, including sending a signal to write is executed, which I suspect to be in the same ballpark as regular committing, but I haven’t checked that.

During the investigations of my previous blogpost about what happens during a commit and when the data becomes available, I used breaks in gdb (GNU debugger) at various places of the execution of an insert and a commit to see what is visible for other sessions during the various stages of execution of the commit.

However, I did find something else, which is very logical, but is easily overlooked: at certain moments access to the table is blocked/serialised in order to let a session make changes to blocks belonging to the table, or peripheral blocks like undo, for the sake of consistency. These are changes made at the physical layer of an Oracle segment, the logical model of Oracle says that writers don’t block readers.

The first question is if this is a big issue. Well, for that you should simply look at any normal usage of an Oracle database. I don’t believe these serialisation moments are an issue, because the majority of my client’s does not experience serialisation problem.

However, this can become an issue if database processes can not run at will. What that means is that if processes are randomly stopped from execution by the operating system or hypervisor, it could be that the database process can be at an earlier mentioned ‘serialisation point’, which then means that access for all other processes remains blocked until the process is made running again, which then gives the process the opportunity to free exclusive access to a resource.

What are issues when a database process can not run at will?
– CPU oversubscription. If more processes are running than CPU’s are available (think about core’s and threads, and what it means for your CPU architecture), it means the operating system needs to make a decision on what it wants to be running, and what it wants to wait. No operating system scheduler understands or can know when an Oracle database process is running in a critical part of code, and therefore should not stop execution of it.
– Memory oversubscription. Whenever your system starts actively swapping in and out, your system is swapping, which means it’s moving active pages onto the swap device, and reads back active pages from the swap device for a process that demands them. This will activate additional tasks, and make processes get stuck waiting for memory to become available at all kinds of points during execution. Often, swapping is the beginning of the end, which means the services on the server stop functioning altogether for multiple reasons, like the Out Of Memory killer, or simply timing out for client’s of a process.
– Oversubscription of CPU or memory using virtualisation. As a troubleshooter, I would say this is actually worse than the ‘simple oversubscription’ mentioned in the two earlier points. Because typically, the visible part is the virtual machine. The virtual machine in this case itself is not oversubscribed, the oversubscription takes place at a layer invisible to the virtual machine, which unexplainably performs unpredictable. It is simply not okay to oversubscribe any machine that needs to run something that is latency sensitive like the Oracle database, unless you fully understand all the trade-offs that are coming with it, and you have the ability to understand when it does start to occur.

Whenever database processes can not run at will, you will see waits you didn’t see when processes could run at will. Typical waits in that case are (not an exhaustive list):
– any latch waits
– buffer busy waits
– any cursor: pin waits
Please mind these waits are not unique for this issue, any of these waits can occur for other reasons too.

Let me show a couple of examples for which the wait is artificially created by stopping execution at a point where this serialisation takes place, thereby mimicking getting put off cpu due to load, using the earlier insert and commit execution:

a) latch: cache buffers chains
Whenever a process needs to read or modify a database buffer, it needs to ‘pin’ the buffer. Such a pin is also known as a buffer handle or a buffer state object. The pin, which is a memory area, must be obtained from a freelist and attached to the buffer header in order to pin the buffer. A pin is obtained using the kcbzgs (kernel cache buffers helper functions get state object) function, which calls the kssadf_numa_intl (kernel service state objects add from freelist numa internal) function which initialises the state object.

The function that performs the actual pin of a block is not a single one, pinning a block is done in multiple functions. Two of them are kcbgtcr (kernel cache buffers get consistent read) and kcbgcur (kernel cache buffers get current). Under normal circumstances, concurrency for a consistent read of a buffer (kcbgtcr) does not lead to any blocking, because the latch (cache buffers chains) can be taken in shared mode, and the pinning is done in ‘fast mode’ (my interpretation):

> kcbgtcr(0x7f88a8b79db0, 0, ...)
| > kscn_to_ub8_impl(0x7f88a8b88d4c, 0x6baefdc0, ...)
| < kscn_to_ub8_impl+0x00000000003e returns: 0x287562
| > ktrexf(0x7fff998123f8, 0x7f88ae5d7f30, ...)
| | > ktrEvalBlockForCR(0x7f88a8b88d4c, 0x7f88ae5d7f30, ...)
| | < ktrEvalBlockForCR+0x0000000000f4 returns: 0x1
| < ktrexf+0x0000000000a4 returns: 0x9
| > kcbzgsf(0x1, 0x7fff998120b8, ...)
| | > kssadf_numa_intl(0x36, 0x6ddc0b50, ...)
| | | > kpdbIdToUid(0, 0x6ddc0b50, ...)
| | | < kpdbIdToUid+0x0000000000e7 returns: 0
| | < kssadf_numa_intl+0x000000000235 returns: 0x6bf84908
| < kcbzgsf+0x0000000001a0 returns: 0x6bf84908
| > kcbz_fp_cr(0x8dff9478, 0x6bf84998, ...)
| < kcbz_fp_cr+0x000000000089 returns: 0
| > kscn_to_ub8_impl(0x7f88a8b88d70, 0x1, ...)
| < kscn_to_ub8_impl+0x00000000003e returns: 0
< kcbgtcr+0x000000000b38 returns: 0x8df94014

Inside the kcbgtcr function, kcbzgsf (kernel cache buffers helper functions test state object fast) is called, and after that, the state object is pinned to the block in kcbz_fp_cr (kernel cache buffers helper functions fast pin for consistent read). Do you see the absence of ksl_get_shared_latch function to serialise access to add this pin? Actually the cache buffers chains latch is gotten, but it’s done inside the kcbgtcr function. The latch is gotten before the ktrexf (kernel transaction redo examine block fast) in shared mode, and the latch is released after the kcbz_fp_cr function. Multiple processes can pin the block for consistent read without serialisation or blocking each other because of the cache buffers chains latch being a shared latch, and this latch is taken in shared mode.

Why do I show this? Well, in the “old” days, which is Oracle 8.1, Oracle did not use shared latches (at least on the platforms I worked on, but if my memory serves me right, shared latches were introduced starting from Oracle 9), which meant for any logical read the cache buffers chains latch had to be obtained. This could get painful for very frequently visited blocks, like index root blocks, and high concurrency, and additional to that, because there was no post/wait messaging for latch waits but only spinning, this meant lots of processes waiting for latch number 98, and CPU usage going through the roof.

Back to today. Below is shown how the function calls look like when DML is done. When a buffer change is done (data is changed), a buffer has to be obtained in current mode, which is done using the kcbgcur function:

> kcbgcur(0x7ffda5957c70, 0x2, ...)
| > kcbzgs(0x1, 0x2, ...)
| | > kssadf_numa_intl(0x36, 0x7a6086c8, ...)
| | | > kpdbIdToUid(0, 0x7a6086c8, ...)
| | | < kpdbIdToUid+0x0000000000e7 returns: 0
| | < kssadf_numa_intl+0x000000000235 returns: 0x7867c9c0
| < kcbzgs+0x000000000178 returns: 0x7867c9c0
| > kti_is_imu(0x7867ca50, 0xb0f71018, ...)
| | > ktcgtx(0x76f29d50, 0xb0f71018, ...)
| | < ktcgtx+0x00000000001e returns: 0x76f29d50
| < kti_is_imu+0x000000000039 returns: 0
< kcbgcur+0x0000000009fb returns: 0xb0302014

This looks reasonably the same as the previous call overview which shown kcbgtcr, however the kcbzgsf function changed to kcbzgs, so minus the ‘f’ for fast, and a few other functions are missing. Another interesting thing (not visible) is the cache buffers chains latch is obtained after the kcbzgs function that obtains the buffer state object, and the latch is obtained with special bit 0x1000000000000000 set to indicate the shared latch is obtained in non-shared mode. After the kti_is_imu function the pin is attached to the buffer and the latch is freed.

To make things a little more complicated, a buffer can be gotten in current mode, but still get the cache buffers chains in shared mode. This is how that looks like:

> kcbgcur(0x7ffda5955620, 0x1, ...)
| > kcbzgs(0x1, 0x1, ...)
| | > kssadf_numa_intl(0x36, 0x7a6086c8, ...)
| | | > kpdbIdToUid(0, 0x7a6086c8, ...)
| | | < kpdbIdToUid+0x0000000000e7 returns: 0
| | < kssadf_numa_intl+0x000000000235 returns: 0x7867c9c0
| < kcbzgs+0x000000000178 returns: 0x7867c9c0
| > kcbz_fp_shr(0xb0fa3cb8, 0x7867ca50, ...)
| < kcbz_fp_shr+0x000000000085 returns: 0x7f3011d82f40
< kcbgcur+0x0000000009fb returns: 0xb07a0014

The function kcbzgs without ‘f’ is called, but the cache buffers chains latch is gotten in shared mode (not visible; done in the kcbgcur function after kcbzgs), and there’s the function kcbz_fp_shr (kernel cache buffers helper function fast pin shared) to pin the buffer in shared mode.

In order to be allowed to change a buffer, the cache buffers chains latch must be taken in non-shared mode to change the buffer header to indicate the buffer is busy and guarantee only the process itself can access it. Obtaining the cache buffers chains latch in non-shared mode means that access to the hash bucket (multiple hash buckets actually) to which the buffer is linked is blocked until the latch is freed. Please mind the period that the latch is obtained here is very short.

However, this blocking scenario can be easily replayed:
1. session 1: startup a database foreground session.
2. session 2: obtain the PID of the session 1’s process and attach to it with gdb.
3. session 2: break on the kti_is_imu function (break kti_is_imu) and continue.
4. session 1: insert a row into a table (insert into test values (‘a’);). the breakpoint will fire in gdb, stopping execution.
5. session 3: startup another database foreground session.
6. session 3: select the table on which the insert just broke (select * from test;). This will hang.
7. session 4: startup a SYSDBA session, and obtain the wait state of the foreground sessions:

SQL> @who active

SID_SERIAL_INST OSPID    USERNAME      C SERVER    PROGRAM                                          EXTRA
--------------- -------- ------------- - --------- ------------------------------------------------ -------------
59,53450,@1     28166    SYS           * DEDICATED sqlplus@memory-presentation.local (TNS V1-V3)    time:0ms,event:ON CPU:SQL,seq#:98
102,53150,@1    28083    TS              DEDICATED sqlplus@memory-presentation.local (TNS V1-V3)    time:66867.83ms,event:ON CPU:SQL,seq#:33
101,6637,@1     28186    TS              DEDICATED sqlplus@memory-presentation.local (TNS V1-V3)    time:344013.82ms,event:latch: cache buffers chainsp1:1811473056p2:289p3:24599> Blocked by (inst:sid): 1:102,seq#:39

SID 101 is waiting on the cache buffers chains latch, because SID 102 is holding that. SID 102 is showing ON CPU because we broke execution outside of a wait event, so Oracle thinks it’s busy executing, while it is actually stopped by gdb.

Again, this is a wait (latch: cache buffers chains) that is unlikely be an issue, and if it is, you either ran into a bug, or this is only the way a much bigger problem (oversubscription, bad application design) is showing itself.

b) buffer busy waits
The name of the wait event ‘buffer busy wait’ is self explanatory. The technical implementation is way lesser known. What technically happens during a buffer busy wait, is that a session needs to read a certain buffer, and in order to read it, it performs the usual actions of selecting the correct cache buffers chains bucket, following the double linked list to the buffer headers to find the best best buffer and then obtain a buffer state object. The critical part for running into the buffer busy wait is when the session checks the state of the buffer via the buffer header and finds the change state (X$BH.CSTATE) to be higher than 0, indicating the block is currently being changed. At this point the session calls kcbzwb (kernel cache buffers helper function wait for buffer), and must wait for the buffer state to be changed and therefore accessible, for which the wait event is the buffer busy wait event. When the change state is reverted back to 0, the buffer is accessible again.

Once a buffer is pinned in current mode using kcbgcur en kcbzgs, the buffer is still fully accessible for all sessions. Only once the database truly starts to change the buffer, which is done in kcbchg1_main, the cache buffers chains latch is taken non-shared, and the CSTATE field in the buffer header is changed to 1 and then the latch is freed. My current investigations show that the CSTATE is changed to 2 after the kcopcv and kcrfw_copy_cv functions, so when the redo is written to the public redo strand, and set to 4 after the changes have been written to the buffers, which is done in kcbapl. At the end of the kcbchg1_main function, when all the changes are done, the cache buffers chains latch is taken non-shared again, the CSTATE field is set to 0 and the latch is freed to enable access to the buffer again.

68801:kcbchg1_main+2536:0x000000006ba4e4d8(Variable Size(pgsz:2048k)|+182772952 shared pool|(child)latch#5383:cache buffers chains+0 shared pool|permanent memor,duration 1,cls perm+2204888 ):R:8:0/0()
68802:kcbchg1_main+2536:0x000000006ba4e4d8(Variable Size(pgsz:2048k)|+182772952 shared pool|(child)latch#5383:cache buffers chains+0 shared pool|permanent memor,duration 1,cls perm+2204888 ):W:8:0x1000000000000022/1152921504606847010()
..
68897:kcbchg1_main+3318:0x00000000967eddda(Variable Size(pgsz:2048k)|+901701082 buffer header|9668A000+66 ):W:1:0x1/1()
..
68939:kcbchg1_main+3715:0x000000006ba4e4d8(Variable Size(pgsz:2048k)|+182772952 shared pool|(child)latch#5383:cache buffers chains+0 shared pool|permanent memor,duration 1,cls perm+2204888 ):R:8:0x1000000000000022/1152921504606847010()
68940:kcbchg1_main+3715:0x000000006ba4e4d8(Variable Size(pgsz:2048k)|+182772952 shared pool|(child)latch#5383:cache buffers chains+0 shared pool|permanent memor,duration 1,cls perm+2204888 ):W:8:0/0()

Above you see a cache buffers chains latch being obtained non shared (special bit 0x1000000000000000 is set), a little later the buffer header is updated which sets the value 1 at offset 66. Once this happens, this is visible in the CSTATE column of X$BH. A little further the latch is freed.

Why am I showing all of this? Well, there is quite some work that is done while the buffer is in a changed state (CSTATE>0) and therefore inaccessible. In normal situations, this is not a problem because despite all the work this is done very quickly, and therefore it’s hardly noticeable/measurable, so there’s no or very little time spend in buffer busy waits in most databases

However… if processes do get stuck randomly, it’s possible that a process gets stuck while making changes to a buffer, which then means any additional access to the buffer results in buffer busy waits. Also mind that a change to a table buffer requires (at least; in the most simple case) three buffers to be changed: the buffer containing the table data, the buffer containing the undo segment transaction table and the buffer containing the actual undo. A commit requires one buffer: the undo segment transaction table.

Let me show you an example:
1. session 1: startup a database foreground session.
2. session 2: obtain the PID of the session 1’s process and attach to it with gdb.
3. session 2: break on ktugur and kcopcv and continue.
4. session 3: startup a database foreground session.
5. session 1: insert a row into a table: insert into test values (‘a’);
6. session 2: gdb breaks execution of session 1 because it encountered ktugur.
7. session 3: select * from test; this runs unblocked.
8. session 4: startup a database foreground session as sysdba.
9. session 4: select * from v$lock; this will hang.

This one wasn’t obvious. I found this upon doing the tests. When a foreground session is in ktugur, it is creating the change vectors (kernel transaction undo generate undo and redo), and it holds two buffers in current mode:

SQL> l
  1  select 	ts.name tablespace,
  2  	dbablk,
  3  	class,
  4  	state,
  5  	decode(state,0,'free',1,'xcur',2,'scur',3,'cr', 4,'read',5,'mrec',6,'irec',7,'write',8,'pi', 9,'memory',10,'mwrite',11,'donated', 12,'protected',  13,'securefile', 14,'siop',15,'recckpt', 16, 'flashfree',  17, 'flashcur', 18, 'flashna') state_decoded,
  6  	mode_held,
  7  	changes,
  8  	cstate,
  9  	flag,
 10  	flag2,
 11  	rflag,
 12  	sflag,
 13  	lru_flag,
 14  	dirty_queue
 15  from 	  x$bh b
 16  	, v$tablespace ts
 17  where 	1=1
 18  and	not (us_nxt = us_prv and wa_nxt = wa_prv and to_number(wa_nxt,'xxxxxxxxxxxxxxxx') = to_number(us_nxt,'xxxxxxxxxxxxxxxx') + 16)
 19* and	b.ts#=ts.ts#
SQL> /

TABLESPACE                         DBABLK      CLASS      STATE STATE_DECO  MODE_HELD    CHANGES     CSTATE       FLAG      FLAG2      RFLAG      SFLAG   LRU_FLAG DIRTY_QUEUE
------------------------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- -----------
UNDOTBS1                              128         17          1 xcur                2          1          0    2097152          0          0          0          0           0
TS                                  24597          1          1 xcur                2          1          0    2097153          0          0          0          0           0

The thing that was not obvious at first was that the blocks have CSTATE 0; so they can be accessed and read for a consistent read. Then I look at the wait and the p1/2/3 of the wait:

SID_SERIAL_INST OSPID    USERNAME      C SERVER    PROGRAM                                          EXTRA
--------------- -------- ------------- - --------- ------------------------------------------------ -------------------------------------------------------
100,262,@1      6274     SYS             DEDICATED sqlplus@memory-presentation.local (TNS V1-V3)    time:2519.48ms,event:buffer busy waitsp1:3p2:128p3:17> Blocked by (inst:sid): 1:59,seq#:476
142,15103,@1    6695     TS              DEDICATED sqlplus@memory-presentation.local (TNS V1-V3)    time:47115.23ms,event:SQL*Net message from client,seq#:226
12,25165,@1     7036     SYS           * DEDICATED sqlplus@memory-presentation.local (TNS V1-V3)    time:0ms,event:ON CPU:SQL,seq#:3165
59,31601,@1     5891     TS              DEDICATED sqlplus@memory-presentation.local (TNS V1-V3)    time:31292.51ms,event:ON CPU:SQL,seq#:111

Above SID 100 is session 4; it hangs on a buffer busy wait. The datafile is #2, which is the undo table space UNDOTBS1 and the buffer that it is waiting for is 128. Why does SID 100 needs to wait for the buffer, while another session (session 3 in the above runbook) can run without getting blocked for a buffer that is held exactly in the same state (xcur and CSTATE is zero)?

The answer can be gotten when executing pstack on the waiting process (or attach with gdb and obtain a backtrace):

(gdb) bt
#0  0x00007fd6a085bbda in semtimedop () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000000010f9cca6 in sskgpwwait ()
#2  0x0000000010f9a2e8 in skgpwwait ()
#3  0x0000000010a66995 in ksliwat ()
#4  0x0000000010a65d25 in kslwaitctx ()
#5  0x00000000015c5e5e in kcbzwb ()
#6  0x0000000010b3748e in kcbgcur ()
#7  0x0000000010ab9b17 in ktuGetUsegHdr ()
#8  0x00000000014075d3 in ktcxbcProcessInPDB ()
#9  0x0000000001406de2 in ktcxbc ()
#10 0x0000000010ea8781 in qerfxFetch ()
#11 0x0000000010bb919c in rwsfcd ()
#12 0x00000000035ae618 in qeruaFetch ()
#13 0x00000000035ad837 in qervwFetch ()
#14 0x0000000010bb919c in rwsfcd ()
#15 0x0000000010ea1528 in qerhnFetch ()
#16 0x0000000010bb919c in rwsfcd ()
#17 0x0000000010ea1528 in qerhnFetch ()
#18 0x0000000010c9eeb3 in opifch2 ()
#19 0x000000000268c3de in kpoal8 ()
#20 0x0000000010ca5b3d in opiodr ()
#21 0x0000000010f37a29 in ttcpip ()
#22 0x0000000001eb4674 in opitsk ()
#23 0x0000000001eb914d in opiino ()
#24 0x0000000010ca5b3d in opiodr ()
#25 0x0000000001eb04ed in opidrv ()
#26 0x0000000002afedf1 in sou2o ()
#27 0x0000000000d05577 in opimai_real ()
#28 0x0000000002b09b31 in ssthrdmain ()
#29 0x0000000000d05386 in main ()

The important bit is at #6: kcbgcur (kernel cache buffers get buffer in current mode)! In other words: the block is gotten in current mode while another session is holding the block in current mode, which means it has to wait. The function kcbzwb means kernel cache buffer helper function wait for buffer.

I really wondered why the undo segment buffer is gotten in current mode; this is a read, there is no intention to change something. The current educated guess is this a way to be sure this buffer contains the current state of the transactions across all nodes in a RAC cluster. So this might be a clever optimisation to be absolutely sure the current state of the undo segment transaction table is gotten.

10. session 2: continue. This will break on kcopcv.
11. session 3: select * from test; this will hang.

Now we have two hanging sessions waiting for the wait event buffer busy wait:

SQL> @who active

SID_SERIAL_INST OSPID    USERNAME      C SERVER    PROGRAM                                          EXTRA
--------------- -------- ------------- - --------- ------------------------------------------------ -----------------
100,262,@1      6274     SYS             DEDICATED sqlplus@memory-presentation.local (TNS V1-V3)    time:120790ms,event:buffer busy waitsp1:3p2:128p3:35> Blocked by (inst:sid): 1:59,seq#:501
142,15103,@1    6695     TS              DEDICATED sqlplus@memory-presentation.local (TNS V1-V3)    time:1175715.03ms,event:buffer busy waitsp1:4p2:24597p3:1> Blocked by (inst:sid): 1:59,seq#:250
12,25165,@1     7036     SYS           * DEDICATED sqlplus@memory-presentation.local (TNS V1-V3)    time:0ms,event:ON CPU:SQL,seq#:3309
59,31601,@1     5891     TS              DEDICATED sqlplus@memory-presentation.local (TNS V1-V3)    time:9544.97ms,event:ON CPU:SQL,seq#:116

SID 100 still hangs in wait event buffer busy wait for file 3, block 128, but now the session doing the full table scan also hangs in wait event buffer busy wait, but for file 4, block 24597. Let’s look at the buffers held in current mode again:

SQL> @pinned_blocks

TABLESPACE                         DBABLK      CLASS      STATE STATE_DECO  MODE_HELD    CHANGES     CSTATE       FLAG      FLAG2      RFLAG      SFLAG   LRU_FLAG DIRTY_QUEUE
------------------------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- -----------
UNDOTBS1                              128         35          1 xcur                2          0          1    2097156          0          0          0          0           0
UNDOTBS1                             3127         36          1 xcur                2          0          1    2097156          0          0          0          0           0
TS                                  24597          1          1 xcur                2          0          1    2097157          0          0          0          0           0

Because the insert progressed to kcopcv, the CSTATE of all three buffers involved in the insert, the undo segment transaction table, the undo segment undo entry and the buffer which has the inserted row are set to 1. This means that the two buffer busy waits are actually two different reasons: the select * from v$lock still waits to get a buffer in current mode, and the select * from test, which does a consistent read, now is waiting for the CSTATE to get set back to 0. In order to solve the hanging, go to session 2 and continue. This will break again on kcoapl, and again if you continue, because there are 3 blocks involved in this insert. Another way to make the insert finish is to disable all breakpoints (disable command) or to quit gdb.

Just for the fun of it, here is another example of how a stuck session can escalate: a process that gets stuck executing the kcbapl (kernel cache buffers apply a change to a buffer). This scenario can be replayed using the following steps:
1. session 1: startup a database foreground session.
2. session 2: obtain the PID of the session 1’s process and attach to it with gdb.
3. session 2: break on kcbapl and continue.
4. session 3: startup a database foreground session.
5. session 3: insert a row into a table: insert into test values (‘a’);
6. session 1: insert a row into a table: insert into test values (‘a’);
7. session 2: gdb broken execution of session 1 because it encountered kcbapl.
8. session 3: commit; this will hang

Can you reason why session 3 is hanging? Both the sessions are doing inserts to the same table…Give it a thought and replay it yourself.

Explanation: the breakpoint stopped execution before actually doing the changes. CSTATE for the buffer is set to 1, it will get set to 2 inside kcbapl to indicate the change vectors have been copied. But the important bit is that a redo copy latch was taken before executing kcbapl. The commit in session 3 sent a message to the logwriter asynchronously asking it to write (first invocation of kcrf_commit_force_int), and then freed the logical locking of the insert, and then invoked kcrf_commit_force again to wait for the logwriter to finish writing and get notified. However, the logwriter needs to obtain all the redo copy latches (it actually reads a metadata structure that indicates the redo copy latches are freed or not), at which it needs to wait because session 1 stuck holding a redo copy latch and therefore needs to wait. Not only does this show how dangerous it is when sessions can’t run, it also shows how dangerous it can be to run gdb in a production database, it can be quite hard to figure out what will be the implication of attaching with gdb to the process. So the actual outcome of above is that session 3 waits in the event ‘log file sync’, and the logwriter is waiting in ‘LGWR wait for redo copy’.

Summary
The Oracle database is carefully setup to be able to handle concurrency. In most normal situations you will see no to maybe very little time spend in wait events that indicate concurrency, like latches or buffer busy waits. If an Oracle database is on a machine that has more active processes than available CPU threads or cores (this depends on the hardware and CPU architecture; for Intel based architectures you should lean towards core’s, for SPARC you can lean more towards threads), the operating system needs to make choices, and will do that trying to give all processes an equal share. This means that processes will be put off CPU in order to give all processes a fair share of CPU run time. Also mind other oversubscription possibilities, like using more memory than physically available, and doing the same using virtualisation, which is handing out more VCPU’s than CPU threads, and handing out more memory than physically available. Because you can’t look outside your virtual machine, it’s often harder to troubleshoot virtualisation oversubscription issues.

Oracle really tries to minimise the amount of time spend while holding a non shared latch. Therefore with modern Oracle databases it’s unlikely that latch waits show up, even if a system is (not too much) oversubscribed. If you do have a latching problem, it’s most likely you have got another problem which causes this.

The most well documented reason for buffer busy waits is a buffer being written onto persistent media, and therefore the buffer is marked busy for the time of the write. Another reason that is documented, is getting a buffer in a mode that is not compatible with the mode it is held in. Having a buffer in current mode will block another request for the buffer in current mode. But, having a buffer pinned in current mode does not block consistent read access, it’s only when the buffer is being changed that requires the buffer to be marked busy. This state is externalised in the CSTATE column of the view X$BH, which shows fields of the buffer header struct that is used to manage the buffer cache buffers. Because there is quite some work to be done when a buffer is changed, and during making these changes the buffers are not usable (‘busy’) for other processes, a buffer busy wait is more likely to occur when oversubscription is taking place.

Thanks to Jonathan Lewis for explaining how to get the buffer header addresses.
Thanks to Abel Macias for brainstorming.

%d bloggers like this: