Archive

Tag Archives: database

There are a couple of underscore parameters changed from spare to named ones.
It’s interesting to see that in sysstat, ‘spare statistic 2’ changed to ‘cell XT granule IO bytes saved by HDFS tbs extent map scan’. This obviously has to do with big data access via cell servers. What is weird is that this is the only version where this had happened.

parameters unique in version 12.2.0.1.190416 versus 12.2.0.1.190716

NAME
--------------------------------------------------
_eleventh_spare_parameter
_one-hundred-and-forty-seventh_spare_parameter
_one-hundred-and-forty-sixth_spare_parameter
_tenth_spare_parameter

parameters unique in version 12.2.0.1.190716 versus 12.2.0.1.190416

NAME
--------------------------------------------------
_bug27693416_kgh_free_list_min_effort
_cell_offload_hybrid_processing
_dskm_single_instance
_seventh_spare_parameter

parameter values changed isdefault between 12.2.0.1.190416 versus 12.2.0.1.190716

parameter values unique to 12.2.0.1.190416 versus 12.2.0.1.190716

parameter values unique to 12.2.0.1.190716 versus 12.2.0.1.190416

waitevents unique in version 12.2.0.1.190416 versus 12.2.0.1.190716

waitevents unique in version 12.2.0.1.190716 versus 12.2.0.1.190416

waitevents changed parameter description between 12.2.0.1.190416 versus 12.2.0.1.190716

x$ tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

x$ tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

x$ tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

x$ tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

v$ tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

v$ tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

v$ tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

v$ tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

gv$ tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

gv$ tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

gv$ tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

gv$ tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

sysstat statistics unique to 12.2.0.1.190416 versus 12.2.0.1.190716

NAME
----------------------------------------------------------------------------------------------------
spare statistic 2

sysstat statistics unique to 12.2.0.1.190716 versus 12.2.0.1.190416

NAME
----------------------------------------------------------------------------------------------------
cell XT granule IO bytes saved by HDFS tbs extent map scan

sys_time_model statistics unique to 12.2.0.1.190416 versus 12.2.0.1.190716

sys_time_model statistics unique to 12.2.0.1.190716 versus 12.2.0.1.190416

dba tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

dba tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

dba tables columns unique to 12.2.0.1.190716 versus 12.2.0.1.190416

dba tables columns unique to 12.2.0.1.190416 versus 12.2.0.1.190716

cdb tables unique to 12.2.0.1.190416 versus 12.2.0.1.190716

cdb tables unique to 12.2.0.1.190716 versus 12.2.0.1.190416

cdb tables column unique to 12.2.0.1.190416 versus 12.2.0.1.190716

cdb tables column unique to 12.2.0.1.190716 versus 12.2.0.1.190416

And here are the differences in symbols (c functions).
Most notable:
– a lot of functions have been added that seem to be part of ‘little cms’, which seems to be a color management suite. I have no idea currently what these are doing in the oracle database executable.
– a significant amount of kola* (kernel objects lob) and prs* (parse) related functions have been added.

code symbol names unique in version 12.2.0.1.190416 versus 12.2.0.1.190716

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
free_ex_data                                                 free_ex_data                                                 ??
kcrfw_alfs_use_polling                                       (kcrfw_alfs)_use_polling                                     kernel cache recovery file write/broadcast SCN adaptive log file sync ??
kdmoBuildOperand                                             (kdmo)BuildOperand                                           kernel data in-memory data layer optimizer ??
kdmoDecompAvgCostPerSeg                                      (kdmo)DecompAvgCostPerSeg                                    kernel data in-memory data layer optimizer ??
kdmoDecompGetRate                                            (kdmo)DecompGetRate                                          kernel data in-memory data layer optimizer ??
kdmoGetStatHT                                                (kdmo)GetStatHT                                              kernel data in-memory data layer optimizer ??
kdmoIsValidTableCol                                          (kdmo)IsValidTableCol                                        kernel data in-memory data layer optimizer ??
kdmoMinMaxCountPrunableCUsSeg                                (kdmo)MinMaxCountPrunableCUsSeg                              kernel data in-memory data layer optimizer ??
kdmoMinMaxProcessPreds                                       (kdmo)MinMaxProcessPreds                                     kernel data in-memory data layer optimizer ??
kdmoPredEvalAvgCostPerSeg                                    (kdmo)PredEvalAvgCostPerSeg                                  kernel data in-memory data layer optimizer ??
kdmoPredEvalGetRate                                          (kdmo)PredEvalGetRate                                        kernel data in-memory data layer optimizer ??
kdpSizeOfGbyKdst                                             (kdp)SizeOfGbyKdst                                           kernel data archive compression: pcode ??
kdp_op_is_char                                               (kdp)_op_is_char                                             kernel data archive compression: pcode ??
kdp_pcode_calc_constant_area                                 (kdp)_pcode_calc_constant_area                               kernel data archive compression: pcode ??
kdp_pcode_setup_cbks                                         (kdp)_pcode_setup_cbks                                       kernel data archive compression: pcode ??
kdzdcol_agg_cols_imc_dict_dgk_update                         (kdzd)col_agg_cols_imc_dict_dgk_update                       kernel data archive compression decompression ??
kdzdcol_gby_dgk_create                                       (kdzd)col_gby_dgk_create                                     kernel data archive compression decompression ??
kdzdcol_gby_find_groups                                      (kdzd)col_gby_find_groups                                    kernel data archive compression decompression ??
kewmcomprmdif                                                (kewm)comprmdif                                              kernel event AWR metrics ??
kkesrc                                                       (kke)src                                                     kernel compile cost engine ??
kkfdGetQueReason                                             (kkfd)GetQueReason                                           kernel compile fast dataflow (PQ DFO) ??
kkqgCompareTTWAndWoGby                                       (kkq)gCompareTTWAndWoGby                                     kernel compile query  ??
kkqgMsqAggAppNampOrArgp                                      (kkq)gMsqAggAppNampOrArgp                                    kernel compile query  ??
kkqgMsqUnprs                                                 (kkq)gMsqUnprs                                               kernel compile query  ??
kkqjfCopyFroNoChn                                            (kkqjf)CopyFroNoChn                                          kernel compile query  join analysis join factorization ??
kkqvtOpnInView                                               (kkqvt)OpnInView                                             kernel compile query  vector transformation ??
krbr3sf                                                      (krbr)3sf                                                    kernel redo backup/restore restore and recovery ??
ksunetstat                                                   (ksu)netstat                                                 kernel service (VOS) user ??
ksws_map_fo_restore_to_text                                  (ksws)_map_fo_restore_to_text                                kernel service (VOS) workgroup services ??
kubsxiFetchOpenSetCpx                                        (ku)bsxiFetchOpenSetCpx                                      kernel utility ??
kzvdvechk_ownerid                                            (kzvd)vechk_ownerid                                          kernel security data vault ??
qerxaSetXVMCtx                                               (qer)xaSetXVMCtx                                             query execute rowsource ??
qkexrXC_SetOptTyp                                            (qke)xrXC_SetOptTyp                                          query kernel expressions ??
qksbgUnderOFE                                                (qksbg)UnderOFE                                              query kernel sql bind (variable) management(?) ??
qmxqcSetUDFVar2Parm                                          (qmxq)cSetUDFVar2Parm                                        query XDB XML Objects XML ??
qmxqcTreeAplyOp                                              (qmxq)cTreeAplyOp                                            query XDB XML Objects XML ??
qmxqcXPathNeedInputRetNodeRef                                (qmxq)cXPathNeedInputRetNodeRef                              query XDB XML Objects XML ??
qmxqtcPathHasPred                                            (qmxq)tcPathHasPred                                          query XDB XML Objects XML ??
qmxtr2IsFroRefdInFUP                                         (qmxt)r2IsFroRefdInFUP                                       query XDB XML Objects XML ??
qmxtr2MrkGrpChains                                           (qmxt)r2MrkGrpChains                                         query XDB XML Objects XML ??
set_dh                                                       set_dh                                                       ??

code symbol names unique in version 12.2.0.1.190716 versus 12.2.0.1.190416

NAME                                                         RESOLVE                                                      ANNOTATION
------------------------------------------------------------ ------------------------------------------------------------ --------------------------------------------------------------------------------
AddConversion                                                AddConversion                                                ??
AllocArray                                                   AllocArray                                                   ??
AllocEmptyTransform                                          AllocEmptyTransform                                          ??
AllocateToneCurveStruct                                      AllocateToneCurveStruct                                      ??
BilinearInterp16                                             BilinearInterp16                                             ??
BilinearInterpFloat                                          BilinearInterpFloat                                          ??
BlackPreservingGrayOnlySampler                               BlackPreservingGrayOnlySampler                               ??
BlackPreservingKOnlyIntents                                  BlackPreservingKOnlyIntents                                  ??
BlackPreservingKPlaneIntents                                 BlackPreservingKPlaneIntents                                 ??
BlackPreservingSampler                                       BlackPreservingSampler                                       ??
CLUTElemDup                                                  CLUTElemDup                                                  ??
CLutElemTypeFree                                             CLutElemTypeFree                                             ??
CachedXFORM                                                  CachedXFORM                                                  ??
CachedXFORMGamutCheck                                        CachedXFORMGamutCheck                                        ??
Clipper                                                      Clipper                                                      ??
ComputeAbsoluteIntent                                        ComputeAbsoluteIntent                                        ??
CurveSetDup                                                  CurveSetDup                                                  ??
CurveSetElemTypeFree                                         CurveSetElemTypeFree                                         ??
CurvesDup                                                    CurvesDup                                                    ??
CurvesFree                                                   CurvesFree                                                   ??
DecideCurveType                                              DecideCurveType                                              ??
DecideLUTtypeA2B                                             DecideLUTtypeA2B                                             ??
DecideLUTtypeB2A                                             DecideLUTtypeB2A                                             ??
DecideTextDescType                                           DecideTextDescType                                           ??
DecideTextType                                               DecideTextType                                               ??
DecideXYZtype                                                DecideXYZtype                                                ??
DefaultEvalParametricFn                                      DefaultEvalParametricFn                                      ??
DefaultICCintents                                            DefaultICCintents                                            ??
DefaultLogErrorHandlerFunction                               DefaultLogErrorHandlerFunction                               ??
DupMatShaper                                                 DupMatShaper                                                 ??
DupNamedColorList                                            DupNamedColorList                                            ??
EstimateTAC                                                  EstimateTAC                                                  ??
Eval16nop1D                                                  Eval16nop1D                                                  ??
Eval1Input                                                   Eval1Input                                                   ??
Eval1InputFloat                                              Eval1InputFloat                                              ??
Eval4Inputs                                                  Eval4Inputs                                                  ??
Eval4InputsFloat                                             Eval4InputsFloat                                             ??
Eval5Inputs                                                  Eval5Inputs                                                  ??
Eval5InputsFloat                                             Eval5InputsFloat                                             ??
Eval6Inputs                                                  Eval6Inputs                                                  ??
Eval6InputsFloat                                             Eval6InputsFloat                                             ??
Eval7Inputs                                                  Eval7Inputs                                                  ??
Eval7InputsFloat                                             Eval7InputsFloat                                             ??
Eval8Inputs                                                  Eval8Inputs                                                  ??
Eval8InputsFloat                                             Eval8InputsFloat                                             ??
EvalNamedColor                                               EvalNamedColor                                               ??
EvalNamedColorPCS                                            EvalNamedColorPCS                                            ??
EvaluateCLUTfloat                                            EvaluateCLUTfloat                                            ??
EvaluateCLUTfloatIn16                                        EvaluateCLUTfloatIn16                                        ??
EvaluateCurves                                               EvaluateCurves                                               ??
EvaluateIdentity                                             EvaluateIdentity                                             ??
EvaluateLab2XYZ                                              EvaluateLab2XYZ                                              ??
EvaluateMatrix                                               EvaluateMatrix                                               ??
EvaluateXYZ2Lab                                              EvaluateXYZ2Lab                                              ??
FastEvaluateCurves16                                         FastEvaluateCurves16                                         ??
FastEvaluateCurves8                                          FastEvaluateCurves8                                          ??
FastIdentity16                                               FastIdentity16                                               ??
FileClose                                                    FileClose                                                    ??
FileRead                                                     FileRead                                                     ??
FileSeek                                                     FileSeek                                                     ??
FileTell                                                     FileTell                                                     ??
FileWrite                                                    FileWrite                                                    ??
FixWhiteMisalignment                                         FixWhiteMisalignment                                         ??
FloatXFORM                                                   FloatXFORM                                                   ??
FreeMatShaper                                                FreeMatShaper                                                ??
FreeNamedColorList                                           FreeNamedColorList                                           ??
GamutSampler                                                 GamutSampler                                                 ??
GenericMPEdup                                                GenericMPEdup                                                ??
GenericMPEfree                                               GenericMPEfree                                               ??
IdentitySampler                                              IdentitySampler                                              ??
InkLimitingSampler                                           InkLimitingSampler                                           ??
Java_sun_java2d_cmm_lcms_LCMS_colorConvert                   Java_sun_java2d_cmm_lcms_LCMS_colorConvert                   ??
Java_sun_java2d_cmm_lcms_LCMS_createNativeTransform          Java_sun_java2d_cmm_lcms_LCMS_createNativeTransform          ??
Java_sun_java2d_cmm_lcms_LCMS_getProfileDataNative           Java_sun_java2d_cmm_lcms_LCMS_getProfileDataNative           ??
Java_sun_java2d_cmm_lcms_LCMS_getProfileID                   Java_sun_java2d_cmm_lcms_LCMS_getProfileID                   ??
Java_sun_java2d_cmm_lcms_LCMS_getProfileSizeNative           Java_sun_java2d_cmm_lcms_LCMS_getProfileSizeNative           ??
Java_sun_java2d_cmm_lcms_LCMS_getTagNative                   Java_sun_java2d_cmm_lcms_LCMS_getTagNative                   ??
Java_sun_java2d_cmm_lcms_LCMS_initLCMS                       Java_sun_java2d_cmm_lcms_LCMS_initLCMS                       ??
Java_sun_java2d_cmm_lcms_LCMS_loadProfileNative              Java_sun_java2d_cmm_lcms_LCMS_loadProfileNative              ??
Java_sun_java2d_cmm_lcms_LCMS_setTagDataNative               Java_sun_java2d_cmm_lcms_LCMS_setTagDataNative               ??
LCMS_freeProfile                                             LCMS_freeProfile                                             ??
LCMS_freeTransform                                           LCMS_freeTransform                                           ??
LinLerp1D                                                    LinLerp1D                                                    ??
LinLerp1Dfloat                                               LinLerp1Dfloat                                               ??
MatShaperEval16                                              MatShaperEval16                                              ??
MatrixElemDup                                                MatrixElemDup                                                ??
MatrixElemTypeFree                                           MatrixElemTypeFree                                           ??
MemoryClose                                                  MemoryClose                                                  ??
MemoryRead                                                   MemoryRead                                                   ??
MemorySeek                                                   MemorySeek                                                   ??
MemoryTell                                                   MemoryTell                                                   ??
MemoryWrite                                                  MemoryWrite                                                  ??
NULLClose                                                    NULLClose                                                    ??
NULLRead                                                     NULLRead                                                     ??
NULLSeek                                                     NULLSeek                                                     ??
NULLTell                                                     NULLTell                                                     ??
NULLWrite                                                    NULLWrite                                                    ??
NullFloatXFORM                                               NullFloatXFORM                                               ??
NullXFORM                                                    NullXFORM                                                    ??
OptimizeByComputingLinearization                             OptimizeByComputingLinearization                             ??
OptimizeByJoiningCurves                                      OptimizeByJoiningCurves                                      ??
OptimizeByResampling                                         OptimizeByResampling                                         ??
OptimizeMatrixShaper                                         OptimizeMatrixShaper                                         ??
Pack1Byte                                                    Pack1Byte                                                    ??
Pack1ByteReversed                                            Pack1ByteReversed                                            ??
Pack1ByteSkip1                                               Pack1ByteSkip1                                               ??
Pack1ByteSkip1SwapFirst                                      Pack1ByteSkip1SwapFirst                                      ??
Pack1Word                                                    Pack1Word                                                    ??
Pack1WordBigEndian                                           Pack1WordBigEndian                                           ??
Pack1WordReversed                                            Pack1WordReversed                                            ??
Pack1WordSkip1                                               Pack1WordSkip1                                               ??
Pack1WordSkip1SwapFirst                                      Pack1WordSkip1SwapFirst                                      ??
Pack3Bytes                                                   Pack3Bytes                                                   ??
Pack3BytesAndSkip1                                           Pack3BytesAndSkip1                                           ??
Pack3BytesAndSkip1Optimized                                  Pack3BytesAndSkip1Optimized                                  ??
Pack3BytesAndSkip1Swap                                       Pack3BytesAndSkip1Swap                                       ??
Pack3BytesAndSkip1SwapFirst                                  Pack3BytesAndSkip1SwapFirst                                  ??
Pack3BytesAndSkip1SwapFirstOptimized                         Pack3BytesAndSkip1SwapFirstOptimized                         ??
Pack3BytesAndSkip1SwapOptimized                              Pack3BytesAndSkip1SwapOptimized                              ??
Pack3BytesAndSkip1SwapSwapFirst                              Pack3BytesAndSkip1SwapSwapFirst                              ??
Pack3BytesAndSkip1SwapSwapFirstOptimized                     Pack3BytesAndSkip1SwapSwapFirstOptimized                     ??
Pack3BytesOptimized                                          Pack3BytesOptimized                                          ??
Pack3BytesSwap                                               Pack3BytesSwap                                               ??
Pack3BytesSwapOptimized                                      Pack3BytesSwapOptimized                                      ??
Pack3Words                                                   Pack3Words                                                   ??
Pack3WordsAndSkip1                                           Pack3WordsAndSkip1                                           ??
Pack3WordsAndSkip1Swap                                       Pack3WordsAndSkip1Swap                                       ??
Pack3WordsAndSkip1SwapFirst                                  Pack3WordsAndSkip1SwapFirst                                  ??
Pack3WordsAndSkip1SwapSwapFirst                              Pack3WordsAndSkip1SwapSwapFirst                              ??
Pack3WordsBigEndian                                          Pack3WordsBigEndian                                          ??
Pack3WordsSwap                                               Pack3WordsSwap                                               ??
Pack4Bytes                                                   Pack4Bytes                                                   ??
Pack4BytesReverse                                            Pack4BytesReverse                                            ??
Pack4BytesSwap                                               Pack4BytesSwap                                               ??
Pack4BytesSwapFirst                                          Pack4BytesSwapFirst                                          ??
Pack4BytesSwapSwapFirst                                      Pack4BytesSwapSwapFirst                                      ??
Pack4Words                                                   Pack4Words                                                   ??
Pack4WordsBigEndian                                          Pack4WordsBigEndian                                          ??
Pack4WordsReverse                                            Pack4WordsReverse                                            ??
Pack4WordsSwap                                               Pack4WordsSwap                                               ??
Pack6Bytes                                                   Pack6Bytes                                                   ??
Pack6BytesSwap                                               Pack6BytesSwap                                               ??
Pack6Words                                                   Pack6Words                                                   ??
Pack6WordsSwap                                               Pack6WordsSwap                                               ??
PackALabV2_8                                                 PackALabV2_8                                                 ??
PackAnyBytes                                                 PackAnyBytes                                                 ??
PackAnyWords                                                 PackAnyWords                                                 ??
PackDoubleFrom16                                             PackDoubleFrom16                                             ??
PackDoublesFromFloat                                         PackDoublesFromFloat                                         ??
PackFloatFrom16                                              PackFloatFrom16                                              ??
PackFloatsFromFloat                                          PackFloatsFromFloat                                          ??
PackHalfFrom16                                               PackHalfFrom16                                               ??
PackHalfFromFloat                                            PackHalfFromFloat                                            ??
PackLabDoubleFrom16                                          PackLabDoubleFrom16                                          ??
PackLabDoubleFromFloat                                       PackLabDoubleFromFloat                                       ??
PackLabFloatFrom16                                           PackLabFloatFrom16                                           ??
PackLabFloatFromFloat                                        PackLabFloatFromFloat                                        ??
PackLabV2_16                                                 PackLabV2_16                                                 ??
PackLabV2_8                                                  PackLabV2_8                                                  ??
PackPlanarBytes                                              PackPlanarBytes                                              ??
PackPlanarWords                                              PackPlanarWords                                              ??
PackXYZDoubleFrom16                                          PackXYZDoubleFrom16                                          ??
PackXYZDoubleFromFloat                                       PackXYZDoubleFromFloat                                       ??
PackXYZFloatFrom16                                           PackXYZFloatFrom16                                           ??
PackXYZFloatFromFloat                                        PackXYZFloatFromFloat                                        ??
PreOptimize                                                  PreOptimize                                                  ??
PrecalculatedXFORM                                           PrecalculatedXFORM                                           ??
PrecalculatedXFORMGamutCheck                                 PrecalculatedXFORMGamutCheck                                 ??
Prelin16dup                                                  Prelin16dup                                                  ??
Prelin8dup                                                   Prelin8dup                                                   ??
Prelin8free                                                  Prelin8free                                                  ??
PrelinEval16                                                 PrelinEval16                                                 ??
PrelinEval8                                                  PrelinEval8                                                  ??
PrelinOpt16free                                              PrelinOpt16free                                              ??
R_SSL_clear_options                                          R_SSL_clear_options                                          ??
R_SSL_clear_options_by_type                                  R_SSL_clear_options_by_type                                  ??
ReadMPECurve                                                 ReadMPECurve                                                 ??
ReadMPEElem                                                  ReadMPEElem                                                  ??
ReadSeqID                                                    ReadSeqID                                                    ??
ReadSetOfCurves                                              ReadSetOfCurves                                              ??
SaveTags                                                     SaveTags                                                     ??
TetrahedralInterp16                                          TetrahedralInterp16                                          ??
TetrahedralInterpFloat                                       TetrahedralInterpFloat                                       ??
TrilinearInterp16                                            TrilinearInterp16                                            ??
TrilinearInterpFloat                                         TrilinearInterpFloat                                         ??
Type_Chromaticity_Dup                                        Type_Chromaticity_Dup                                        ??
Type_Chromaticity_Free                                       Type_Chromaticity_Free                                       ??
Type_Chromaticity_Read                                       Type_Chromaticity_Read                                       ??
Type_Chromaticity_Write                                      Type_Chromaticity_Write                                      ??
Type_ColorantOrderType_Dup                                   Type_ColorantOrderType_Dup                                   ??
Type_ColorantOrderType_Free                                  Type_ColorantOrderType_Free                                  ??
Type_ColorantOrderType_Read                                  Type_ColorantOrderType_Read                                  ??
Type_ColorantOrderType_Write                                 Type_ColorantOrderType_Write                                 ??
Type_ColorantTable_Dup                                       Type_ColorantTable_Dup                                       ??
Type_ColorantTable_Free                                      Type_ColorantTable_Free                                      ??
Type_ColorantTable_Read                                      Type_ColorantTable_Read                                      ??
Type_ColorantTable_Write                                     Type_ColorantTable_Write                                     ??
Type_CrdInfo_Dup                                             Type_CrdInfo_Dup                                             ??
Type_CrdInfo_Free                                            Type_CrdInfo_Free                                            ??
Type_CrdInfo_Read                                            Type_CrdInfo_Read                                            ??
Type_CrdInfo_Write                                           Type_CrdInfo_Write                                           ??
Type_Curve_Dup                                               Type_Curve_Dup                                               ??
Type_Curve_Free                                              Type_Curve_Free                                              ??
Type_Curve_Read                                              Type_Curve_Read                                              ??
Type_Curve_Write                                             Type_Curve_Write                                             ??
Type_Data_Dup                                                Type_Data_Dup                                                ??
Type_Data_Free                                               Type_Data_Free                                               ??
Type_Data_Read                                               Type_Data_Read                                               ??
Type_Data_Write                                              Type_Data_Write                                              ??
Type_DateTime_Dup                                            Type_DateTime_Dup                                            ??
Type_DateTime_Free                                           Type_DateTime_Free                                           ??
Type_DateTime_Read                                           Type_DateTime_Read                                           ??
Type_DateTime_Write                                          Type_DateTime_Write                                          ??
Type_Dictionary_Dup                                          Type_Dictionary_Dup                                          ??
Type_Dictionary_Free                                         Type_Dictionary_Free                                         ??
Type_Dictionary_Read                                         Type_Dictionary_Read                                         ??
Type_Dictionary_Write                                        Type_Dictionary_Write                                        ??
Type_LUT16_Dup                                               Type_LUT16_Dup                                               ??
Type_LUT16_Free                                              Type_LUT16_Free                                              ??
Type_LUT16_Read                                              Type_LUT16_Read                                              ??
Type_LUT16_Write                                             Type_LUT16_Write                                             ??
Type_LUT8_Dup                                                Type_LUT8_Dup                                                ??
Type_LUT8_Free                                               Type_LUT8_Free                                               ??
Type_LUT8_Read                                               Type_LUT8_Read                                               ??
Type_LUT8_Write                                              Type_LUT8_Write                                              ??
Type_LUTA2B_Dup                                              Type_LUTA2B_Dup                                              ??
Type_LUTA2B_Free                                             Type_LUTA2B_Free                                             ??
Type_LUTA2B_Read                                             Type_LUTA2B_Read                                             ??
Type_LUTA2B_Write                                            Type_LUTA2B_Write                                            ??
Type_LUTB2A_Dup                                              Type_LUTB2A_Dup                                              ??
Type_LUTB2A_Free                                             Type_LUTB2A_Free                                             ??
Type_LUTB2A_Read                                             Type_LUTB2A_Read                                             ??
Type_LUTB2A_Write                                            Type_LUTB2A_Write                                            ??
Type_MLU_Dup                                                 Type_MLU_Dup                                                 ??
Type_MLU_Free                                                Type_MLU_Free                                                ??
Type_MLU_Read                                                Type_MLU_Read                                                ??
Type_MLU_Write                                               Type_MLU_Write                                               ??
Type_MPE_Dup                                                 Type_MPE_Dup                                                 ??
Type_MPE_Free                                                Type_MPE_Free                                                ??
Type_MPE_Read                                                Type_MPE_Read                                                ??
Type_MPE_Write                                               Type_MPE_Write                                               ??
Type_MPEclut_Read                                            Type_MPEclut_Read                                            ??
Type_MPEclut_Write                                           Type_MPEclut_Write                                           ??
Type_MPEcurve_Read                                           Type_MPEcurve_Read                                           ??
Type_MPEcurve_Write                                          Type_MPEcurve_Write                                          ??
Type_MPEmatrix_Read                                          Type_MPEmatrix_Read                                          ??
Type_MPEmatrix_Write                                         Type_MPEmatrix_Write                                         ??
Type_Measurement_Dup                                         Type_Measurement_Dup                                         ??
Type_Measurement_Free                                        Type_Measurement_Free                                        ??
Type_Measurement_Read                                        Type_Measurement_Read                                        ??
Type_Measurement_Write                                       Type_Measurement_Write                                       ??
Type_NamedColor_Dup                                          Type_NamedColor_Dup                                          ??
Type_NamedColor_Free                                         Type_NamedColor_Free                                         ??
Type_NamedColor_Read                                         Type_NamedColor_Read                                         ??
Type_NamedColor_Write                                        Type_NamedColor_Write                                        ??
Type_ParametricCurve_Dup                                     Type_ParametricCurve_Dup                                     ??
Type_ParametricCurve_Free                                    Type_ParametricCurve_Free                                    ??
Type_ParametricCurve_Read                                    Type_ParametricCurve_Read                                    ??
Type_ParametricCurve_Write                                   Type_ParametricCurve_Write                                   ??
Type_ProfileSequenceDesc_Dup                                 Type_ProfileSequenceDesc_Dup                                 ??
Type_ProfileSequenceDesc_Free                                Type_ProfileSequenceDesc_Free                                ??
Type_ProfileSequenceDesc_Read                                Type_ProfileSequenceDesc_Read                                ??
Type_ProfileSequenceDesc_Write                               Type_ProfileSequenceDesc_Write                               ??
Type_ProfileSequenceId_Dup                                   Type_ProfileSequenceId_Dup                                   ??
Type_ProfileSequenceId_Free                                  Type_ProfileSequenceId_Free                                  ??
Type_ProfileSequenceId_Read                                  Type_ProfileSequenceId_Read                                  ??
Type_ProfileSequenceId_Write                                 Type_ProfileSequenceId_Write                                 ??
Type_S15Fixed16_Dup                                          Type_S15Fixed16_Dup                                          ??
Type_S15Fixed16_Free                                         Type_S15Fixed16_Free                                         ??
Type_S15Fixed16_Read                                         Type_S15Fixed16_Read                                         ??
Type_S15Fixed16_Write                                        Type_S15Fixed16_Write                                        ??
Type_Screening_Dup                                           Type_Screening_Dup                                           ??
Type_Screening_Free                                          Type_Screening_Free                                          ??
Type_Screening_Read                                          Type_Screening_Read                                          ??
Type_Screening_Write                                         Type_Screening_Write                                         ??
Type_Signature_Dup                                           Type_Signature_Dup                                           ??
Type_Signature_Free                                          Type_Signature_Free                                          ??
Type_Signature_Read                                          Type_Signature_Read                                          ??
Type_Signature_Write                                         Type_Signature_Write                                         ??
Type_Text_Description_Dup                                    Type_Text_Description_Dup                                    ??
Type_Text_Description_Free                                   Type_Text_Description_Free                                   ??
Type_Text_Description_Read                                   Type_Text_Description_Read                                   ??
Type_Text_Description_Write                                  Type_Text_Description_Write                                  ??
Type_Text_Dup                                                Type_Text_Dup                                                ??
Type_Text_Free                                               Type_Text_Free                                               ??
Type_Text_Read                                               Type_Text_Read                                               ??
Type_Text_Write                                              Type_Text_Write                                              ??
Type_U16Fixed16_Dup                                          Type_U16Fixed16_Dup                                          ??
Type_U16Fixed16_Free                                         Type_U16Fixed16_Free                                         ??
Type_U16Fixed16_Read                                         Type_U16Fixed16_Read                                         ??
Type_U16Fixed16_Write                                        Type_U16Fixed16_Write                                        ??
Type_UcrBg_Dup                                               Type_UcrBg_Dup                                               ??
Type_UcrBg_Free                                              Type_UcrBg_Free                                              ??
Type_UcrBg_Read                                              Type_UcrBg_Read                                              ??
Type_UcrBg_Write                                             Type_UcrBg_Write                                             ??
Type_ViewingConditions_Dup                                   Type_ViewingConditions_Dup                                   ??
Type_ViewingConditions_Free                                  Type_ViewingConditions_Free                                  ??
Type_ViewingConditions_Read                                  Type_ViewingConditions_Read                                  ??
Type_ViewingConditions_Write                                 Type_ViewingConditions_Write                                 ??
Type_XYZ_Dup                                                 Type_XYZ_Dup                                                 ??
Type_XYZ_Free                                                Type_XYZ_Free                                                ??
Type_XYZ_Read                                                Type_XYZ_Read                                                ??
Type_XYZ_Write                                               Type_XYZ_Write                                               ??
Type_vcgt_Dup                                                Type_vcgt_Dup                                                ??
Type_vcgt_Free                                               Type_vcgt_Free                                               ??
Type_vcgt_Read                                               Type_vcgt_Read                                               ??
Type_vcgt_Write                                              Type_vcgt_Write                                              ??
Unroll1Byte                                                  Unroll1Byte                                                  ??
Unroll1ByteReversed                                          Unroll1ByteReversed                                          ??
Unroll1ByteSkip1                                             Unroll1ByteSkip1                                             ??
Unroll1ByteSkip2                                             Unroll1ByteSkip2                                             ??
Unroll1Word                                                  Unroll1Word                                                  ??
Unroll1WordReversed                                          Unroll1WordReversed                                          ??
Unroll1WordSkip3                                             Unroll1WordSkip3                                             ??
Unroll2Bytes                                                 Unroll2Bytes                                                 ??
Unroll2Words                                                 Unroll2Words                                                 ??
Unroll3Bytes                                                 Unroll3Bytes                                                 ??
Unroll3BytesSkip1Swap                                        Unroll3BytesSkip1Swap                                        ??
Unroll3BytesSkip1SwapFirst                                   Unroll3BytesSkip1SwapFirst                                   ??
Unroll3BytesSkip1SwapSwapFirst                               Unroll3BytesSkip1SwapSwapFirst                               ??
Unroll3BytesSwap                                             Unroll3BytesSwap                                             ??
Unroll3Words                                                 Unroll3Words                                                 ??
Unroll3WordsSkip1Swap                                        Unroll3WordsSkip1Swap                                        ??
Unroll3WordsSkip1SwapFirst                                   Unroll3WordsSkip1SwapFirst                                   ??
Unroll3WordsSwap                                             Unroll3WordsSwap                                             ??
Unroll4Bytes                                                 Unroll4Bytes                                                 ??
Unroll4BytesReverse                                          Unroll4BytesReverse                                          ??
Unroll4BytesSwap                                             Unroll4BytesSwap                                             ??
Unroll4BytesSwapFirst                                        Unroll4BytesSwapFirst                                        ??
Unroll4BytesSwapSwapFirst                                    Unroll4BytesSwapSwapFirst                                    ??
Unroll4Words                                                 Unroll4Words                                                 ??
Unroll4WordsReverse                                          Unroll4WordsReverse                                          ??
Unroll4WordsSwap                                             Unroll4WordsSwap                                             ??
Unroll4WordsSwapFirst                                        Unroll4WordsSwapFirst                                        ??
Unroll4WordsSwapSwapFirst                                    Unroll4WordsSwapSwapFirst                                    ??
UnrollALabV2_8                                               UnrollALabV2_8                                               ??
UnrollAnyWords                                               UnrollAnyWords                                               ??
UnrollChunkyBytes                                            UnrollChunkyBytes                                            ??
UnrollDouble1Chan                                            UnrollDouble1Chan                                            ??
UnrollDoubleTo16                                             UnrollDoubleTo16                                             ??
UnrollDoublesToFloat                                         UnrollDoublesToFloat                                         ??
UnrollFloatTo16                                              UnrollFloatTo16                                              ??
UnrollFloatsToFloat                                          UnrollFloatsToFloat                                          ??
UnrollHalfTo16                                               UnrollHalfTo16                                               ??
UnrollHalfToFloat                                            UnrollHalfToFloat                                            ??
UnrollLabDoubleTo16                                          UnrollLabDoubleTo16                                          ??
UnrollLabDoubleToFloat                                       UnrollLabDoubleToFloat                                       ??
UnrollLabFloatTo16                                           UnrollLabFloatTo16                                           ??
UnrollLabFloatToFloat                                        UnrollLabFloatToFloat                                        ??
UnrollLabV2_16                                               UnrollLabV2_16                                               ??
UnrollLabV2_8                                                UnrollLabV2_8                                                ??
UnrollPlanarBytes                                            UnrollPlanarBytes                                            ??
UnrollPlanarWords                                            UnrollPlanarWords                                            ??
UnrollXYZDoubleTo16                                          UnrollXYZDoubleTo16                                          ??
UnrollXYZDoubleToFloat                                       UnrollXYZDoubleToFloat                                       ??
UnrollXYZFloatTo16                                           UnrollXYZFloatTo16                                           ??
UnrollXYZFloatToFloat                                        UnrollXYZFloatToFloat                                        ??
WriteMPECurve                                                WriteMPECurve                                                ??
WriteSeqID                                                   WriteSeqID                                                   ??
WriteSetOfCurves                                             WriteSetOfCurves                                             ??
XFormSampler16                                               XFormSampler16                                               ??
_LUTeval16                                                   _LUTeval16                                                   ??
_LUTevalFloat                                                _LUTevalFloat                                                ??
_cms15Fixed16toDouble                                        (_cms)15Fixed16toDouble                                      color management system (little cms) ??
_cms8Fixed8toDouble                                          (_cms)8Fixed8toDouble                                        color management system (little cms) ??
_cmsAdaptationMatrix                                         (_cms)AdaptationMatrix                                       color management system (little cms) ??
_cmsAdjustEndianess16                                        (_cms)AdjustEndianess16                                      color management system (little cms) ??
_cmsAdjustEndianess32                                        (_cms)AdjustEndianess32                                      color management system (little cms) ??
_cmsAdjustEndianess64                                        (_cms)AdjustEndianess64                                      color management system (little cms) ??
_cmsAllocAdaptationStateChunk                                (_cms)AllocAdaptationStateChunk                              color management system (little cms) ??
_cmsAllocAlarmCodesChunk                                     (_cms)AllocAlarmCodesChunk                                   color management system (little cms) ??
_cmsAllocCurvesPluginChunk                                   (_cms)AllocCurvesPluginChunk                                 color management system (little cms) ??
_cmsAllocFormattersPluginChunk                               (_cms)AllocFormattersPluginChunk                             color management system (little cms) ??
_cmsAllocIntentsPluginChunk                                  (_cms)AllocIntentsPluginChunk                                color management system (little cms) ??
_cmsAllocInterpPluginChunk                                   (_cms)AllocInterpPluginChunk                                 color management system (little cms) ??
_cmsAllocLogErrorChunk                                       (_cms)AllocLogErrorChunk                                     color management system (little cms) ??
_cmsAllocMPETypePluginChunk                                  (_cms)AllocMPETypePluginChunk                                color management system (little cms) ??
_cmsAllocMemPluginChunk                                      (_cms)AllocMemPluginChunk                                    color management system (little cms) ??
_cmsAllocMutexPluginChunk                                    (_cms)AllocMutexPluginChunk                                  color management system (little cms) ??
_cmsAllocOptimizationPluginChunk                             (_cms)AllocOptimizationPluginChunk                           color management system (little cms) ??
_cmsAllocTagPluginChunk                                      (_cms)AllocTagPluginChunk                                    color management system (little cms) ??
_cmsAllocTagTypePluginChunk                                  (_cms)AllocTagTypePluginChunk                                color management system (little cms) ??
_cmsAllocTransformPluginChunk                                (_cms)AllocTransformPluginChunk                              color management system (little cms) ??
_cmsBuildKToneCurve                                          (_cms)BuildKToneCurve                                        color management system (little cms) ??
_cmsBuildRGB2XYZtransferMatrix                               (_cms)BuildRGB2XYZtransferMatrix                             color management system (little cms) ??
_cmsCalloc                                                   (_cms)Calloc                                                 color management system (little cms) ??
_cmsCallocDefaultFn                                          (_cms)CallocDefaultFn                                        color management system (little cms) ??
_cmsChain2Lab                                                (_cms)Chain2Lab                                              color management system (little cms) ??
_cmsCompileProfileSequence                                   (_cms)CompileProfileSequence                                 color management system (little cms) ??
_cmsComputeInterpParams                                      (_cms)ComputeInterpParams                                    color management system (little cms) ??
_cmsComputeInterpParamsEx                                    (_cms)ComputeInterpParamsEx                                  color management system (little cms) ??
_cmsContextGetClientChunk                                    (_cms)ContextGetClientChunk                                  color management system (little cms) ??
_cmsCreateGamutCheckPipeline                                 (_cms)CreateGamutCheckPipeline                               color management system (little cms) ??
_cmsCreateMutex                                              (_cms)CreateMutex                                            color management system (little cms) ??
_cmsCreateSubAlloc                                           (_cms)CreateSubAlloc                                         color management system (little cms) ??
_cmsDecodeDateTimeNumber                                     (_cms)DecodeDateTimeNumber                                   color management system (little cms) ??
_cmsDefaultICCintents                                        (_cms)DefaultICCintents                                      color management system (little cms) ??
_cmsDestroyMutex                                             (_cms)DestroyMutex                                           color management system (little cms) ??
_cmsDoubleTo15Fixed16                                        (_cms)DoubleTo15Fixed16                                      color management system (little cms) ??
_cmsDoubleTo8Fixed8                                          (_cms)DoubleTo8Fixed8                                        color management system (little cms) ??
_cmsDupDefaultFn                                             (_cms)DupDefaultFn                                           color management system (little cms) ??
_cmsDupMem                                                   (_cms)DupMem                                                 color management system (little cms) ??
_cmsEncodeDateTimeNumber                                     (_cms)EncodeDateTimeNumber                                   color management system (little cms) ??
_cmsEndPointsBySpace                                         (_cms)EndPointsBySpace                                       color management system (little cms) ??
_cmsFloat2Half                                               (_cms)Float2Half                                             color management system (little cms) ??
_cmsFormatterIs8bit                                          (_cms)FormatterIs8bit                                        color management system (little cms) ??
_cmsFormatterIsFloat                                         (_cms)FormatterIsFloat                                       color management system (little cms) ??
_cmsFree                                                     (_cms)Free                                                   color management system (little cms) ??
_cmsFreeDefaultFn                                            (_cms)FreeDefaultFn                                          color management system (little cms) ??
_cmsFreeInterpParams                                         (_cms)FreeInterpParams                                       color management system (little cms) ??
_cmsGetContext                                               (_cms)GetContext                                             color management system (little cms) ??
_cmsGetFormatter                                             (_cms)GetFormatter                                           color management system (little cms) ??
_cmsGetTagDescriptor                                         (_cms)GetTagDescriptor                                       color management system (little cms) ??
_cmsGetTagTrueType                                           (_cms)GetTagTrueType                                         color management system (little cms) ??
_cmsGetTagTypeHandler                                        (_cms)GetTagTypeHandler                                      color management system (little cms) ??
_cmsGetTransformFormatters16                                 (_cms)GetTransformFormatters16                               color management system (little cms) ??
_cmsGetTransformFormattersFloat                              (_cms)GetTransformFormattersFloat                            color management system (little cms) ??
_cmsGetTransformUserData                                     (_cms)GetTransformUserData                                   color management system (little cms) ??
_cmsHalf2Float                                               (_cms)Half2Float                                             color management system (little cms) ??
_cmsHandleExtraChannels                                      (_cms)HandleExtraChannels                                    color management system (little cms) ??
_cmsICCcolorSpace                                            (_cms)ICCcolorSpace                                          color management system (little cms) ??
_cmsIOPrintf                                                 (_cms)IOPrintf                                               color management system (little cms) ??
_cmsInstallAllocFunctions                                    (_cms)InstallAllocFunctions                                  color management system (little cms) ??
_cmsLCMScolorSpace                                           (_cms)LCMScolorSpace                                         color management system (little cms) ??
_cmsLinkProfiles                                             (_cms)LinkProfiles                                           color management system (little cms) ??
_cmsLockMutex                                                (_cms)LockMutex                                              color management system (little cms) ??
_cmsMAT3eval                                                 (_cms)MAT3eval                                               color management system (little cms) ??
_cmsMAT3identity                                             (_cms)MAT3identity                                           color management system (little cms) ??
_cmsMAT3inverse                                              (_cms)MAT3inverse                                            color management system (little cms) ??
_cmsMAT3isIdentity                                           (_cms)MAT3isIdentity                                         color management system (little cms) ??
_cmsMAT3per                                                  (_cms)MAT3per                                                color management system (little cms) ??
_cmsMAT3solve                                                (_cms)MAT3solve                                              color management system (little cms) ??
_cmsMalloc                                                   (_cms)Malloc                                                 color management system (little cms) ??
_cmsMallocDefaultFn                                          (_cms)MallocDefaultFn                                        color management system (little cms) ??
_cmsMallocZero                                               (_cms)MallocZero                                             color management system (little cms) ??
_cmsMallocZeroDefaultFn                                      (_cms)MallocZeroDefaultFn                                    color management system (little cms) ??
_cmsOptimizePipeline                                         (_cms)OptimizePipeline                                       color management system (little cms) ??
_cmsPipelineSetOptimizationParameters                        (_cms)PipelineSetOptimizationParameters                      color management system (little cms) ??
_cmsPluginMalloc                                             (_cms)PluginMalloc                                           color management system (little cms) ??
_cmsQuantizeVal                                              (_cms)QuantizeVal                                            color management system (little cms) ??
_cmsRead15Fixed16Number                                      (_cms)Read15Fixed16Number                                    color management system (little cms) ??
_cmsReadAlignment                                            (_cms)ReadAlignment                                          color management system (little cms) ??
_cmsReadCHAD                                                 (_cms)ReadCHAD                                               color management system (little cms) ??
_cmsReadDevicelinkLUT                                        (_cms)ReadDevicelinkLUT                                      color management system (little cms) ??
_cmsReadFloat32Number                                        (_cms)ReadFloat32Number                                      color management system (little cms) ??
_cmsReadHeader                                               (_cms)ReadHeader                                             color management system (little cms) ??
_cmsReadInputLUT                                             (_cms)ReadInputLUT                                           color management system (little cms) ??
_cmsReadMediaWhitePoint                                      (_cms)ReadMediaWhitePoint                                    color management system (little cms) ??
_cmsReadOutputLUT                                            (_cms)ReadOutputLUT                                          color management system (little cms) ??
_cmsReadProfileSequence                                      (_cms)ReadProfileSequence                                    color management system (little cms) ??
_cmsReadTypeBase                                             (_cms)ReadTypeBase                                           color management system (little cms) ??
_cmsReadUInt16Array                                          (_cms)ReadUInt16Array                                        color management system (little cms) ??
_cmsReadUInt16Number                                         (_cms)ReadUInt16Number                                       color management system (little cms) ??
_cmsReadUInt32Number                                         (_cms)ReadUInt32Number                                       color management system (little cms) ??
_cmsReadUInt64Number                                         (_cms)ReadUInt64Number                                       color management system (little cms) ??
_cmsReadUInt8Number                                          (_cms)ReadUInt8Number                                        color management system (little cms) ??
_cmsReadXYZNumber                                            (_cms)ReadXYZNumber                                          color management system (little cms) ??
_cmsRealloc                                                  (_cms)Realloc                                                color management system (little cms) ??
_cmsReallocDefaultFn                                         (_cms)ReallocDefaultFn                                       color management system (little cms) ??
_cmsReasonableGridpointsByColorspace                         (_cms)ReasonableGridpointsByColorspace                       color management system (little cms) ??
_cmsRegisterFormattersPlugin                                 (_cms)RegisterFormattersPlugin                               color management system (little cms) ??
_cmsRegisterInterpPlugin                                     (_cms)RegisterInterpPlugin                                   color management system (little cms) ??
_cmsRegisterMemHandlerPlugin                                 (_cms)RegisterMemHandlerPlugin                               color management system (little cms) ??
_cmsRegisterMultiProcessElementPlugin                        (_cms)RegisterMultiProcessElementPlugin                      color management system (little cms) ??
_cmsRegisterMutexPlugin                                      (_cms)RegisterMutexPlugin                                    color management system (little cms) ??
_cmsRegisterOptimizationPlugin                               (_cms)RegisterOptimizationPlugin                             color management system (little cms) ??
_cmsRegisterParametricCurvesPlugin                           (_cms)RegisterParametricCurvesPlugin                         color management system (little cms) ??
_cmsRegisterRenderingIntentPlugin                            (_cms)RegisterRenderingIntentPlugin                          color management system (little cms) ??
_cmsRegisterTagPlugin                                        (_cms)RegisterTagPlugin                                      color management system (little cms) ??
_cmsRegisterTagTypePlugin                                    (_cms)RegisterTagTypePlugin                                  color management system (little cms) ??
_cmsRegisterTransformPlugin                                  (_cms)RegisterTransformPlugin                                color management system (little cms) ??
_cmsSearchTag                                                (_cms)SearchTag                                              color management system (little cms) ??
_cmsSetInterpolationRoutine                                  (_cms)SetInterpolationRoutine                                color management system (little cms) ??
_cmsSetTransformUserData                                     (_cms)SetTransformUserData                                   color management system (little cms) ??
_cmsStageAllocIdentityCLut                                   (_cms)StageAllocIdentityCLut                                 color management system (little cms) ??
_cmsStageAllocIdentityCurves                                 (_cms)StageAllocIdentityCurves                               color management system (little cms) ??
_cmsStageAllocLab2XYZ                                        (_cms)StageAllocLab2XYZ                                      color management system (little cms) ??
_cmsStageAllocLabPrelin                                      (_cms)StageAllocLabPrelin                                    color management system (little cms) ??
_cmsStageAllocLabV2ToV4                                      (_cms)StageAllocLabV2ToV4                                    color management system (little cms) ??
_cmsStageAllocLabV2ToV4curves                                (_cms)StageAllocLabV2ToV4curves                              color management system (little cms) ??
_cmsStageAllocLabV4ToV2                                      (_cms)StageAllocLabV4ToV2                                    color management system (little cms) ??
_cmsStageAllocNamedColor                                     (_cms)StageAllocNamedColor                                   color management system (little cms) ??
_cmsStageAllocPlaceholder                                    (_cms)StageAllocPlaceholder                                  color management system (little cms) ??
_cmsStageAllocXYZ2Lab                                        (_cms)StageAllocXYZ2Lab                                      color management system (little cms) ??
_cmsStageClipNegatives                                       (_cms)StageClipNegatives                                     color management system (little cms) ??
_cmsStageGetPtrToCurveSet                                    (_cms)StageGetPtrToCurveSet                                  color management system (little cms) ??
_cmsStageNormalizeFromLabFloat                               (_cms)StageNormalizeFromLabFloat                             color management system (little cms) ??
_cmsStageNormalizeFromXyzFloat                               (_cms)StageNormalizeFromXyzFloat                             color management system (little cms) ??
_cmsStageNormalizeToLabFloat                                 (_cms)StageNormalizeToLabFloat                               color management system (little cms) ??
_cmsStageNormalizeToXyzFloat                                 (_cms)StageNormalizeToXyzFloat                               color management system (little cms) ??
_cmsSubAlloc                                                 (_cms)SubAlloc                                               color management system (little cms) ??
_cmsSubAllocDestroy                                          (_cms)SubAllocDestroy                                        color management system (little cms) ??
_cmsSubAllocDup                                              (_cms)SubAllocDup                                            color management system (little cms) ??
_cmsTagSignature2String                                      (_cms)TagSignature2String                                    color management system (little cms) ??
_cmsTransform2toTransformAdaptor                             (_cms)Transform2toTransformAdaptor                           color management system (little cms) ??
_cmsUnlockMutex                                              (_cms)UnlockMutex                                            color management system (little cms) ??
_cmsVEC3cross                                                (_cms)VEC3cross                                              color management system (little cms) ??
_cmsVEC3distance                                             (_cms)VEC3distance                                           color management system (little cms) ??
_cmsVEC3dot                                                  (_cms)VEC3dot                                                color management system (little cms) ??
_cmsVEC3init                                                 (_cms)VEC3init                                               color management system (little cms) ??
_cmsVEC3length                                               (_cms)VEC3length                                             color management system (little cms) ??
_cmsVEC3minus                                                (_cms)VEC3minus                                              color management system (little cms) ??
_cmsWrite15Fixed16Number                                     (_cms)Write15Fixed16Number                                   color management system (little cms) ??
_cmsWriteAlignment                                           (_cms)WriteAlignment                                         color management system (little cms) ??
_cmsWriteFloat32Number                                       (_cms)WriteFloat32Number                                     color management system (little cms) ??
_cmsWriteHeader                                              (_cms)WriteHeader                                            color management system (little cms) ??
_cmsWriteProfileSequence                                     (_cms)WriteProfileSequence                                   color management system (little cms) ??
_cmsWriteTypeBase                                            (_cms)WriteTypeBase                                          color management system (little cms) ??
_cmsWriteUInt16Array                                         (_cms)WriteUInt16Array                                       color management system (little cms) ??
_cmsWriteUInt16Number                                        (_cms)WriteUInt16Number                                      color management system (little cms) ??
_cmsWriteUInt32Number                                        (_cms)WriteUInt32Number                                      color management system (little cms) ??
_cmsWriteUInt64Number                                        (_cms)WriteUInt64Number                                      color management system (little cms) ??
_cmsWriteUInt8Number                                         (_cms)WriteUInt8Number                                       color management system (little cms) ??
_cmsWriteXYZNumber                                           (_cms)WriteXYZNumber                                         color management system (little cms) ??
bchswSampler                                                 bchswSampler                                                 ??
cmsAdaptToIlluminant                                         (cms)AdaptToIlluminant                                       color management system (little cms) ??
cmsAllocNamedColorList                                       (cms)AllocNamedColorList                                     color management system (little cms) ??
cmsAllocProfileSequenceDescription                           (cms)AllocProfileSequenceDescription                         color management system (little cms) ??
cmsAppendNamedColor                                          (cms)AppendNamedColor                                        color management system (little cms) ??
cmsBFDdeltaE                                                 (cms)BFDdeltaE                                               color management system (little cms) ??
cmsBuildGamma                                                (cms)BuildGamma                                              color management system (little cms) ??
cmsBuildParametricToneCurve                                  (cms)BuildParametricToneCurve                                color management system (little cms) ??
cmsBuildSegmentedToneCurve                                   (cms)BuildSegmentedToneCurve                                 color management system (little cms) ??
cmsBuildTabulatedToneCurve16                                 (cms)BuildTabulatedToneCurve16                               color management system (little cms) ??
cmsBuildTabulatedToneCurveFloat                              (cms)BuildTabulatedToneCurveFloat                            color management system (little cms) ??
cmsCIE2000DeltaE                                             (cms)CIE2000DeltaE                                           color management system (little cms) ??
cmsCIE94DeltaE                                               (cms)CIE94DeltaE                                             color management system (little cms) ??
cmsCMCdeltaE                                                 (cms)CMCdeltaE                                               color management system (little cms) ??
cmsChangeBuffersFormat                                       (cms)ChangeBuffersFormat                                     color management system (little cms) ??
cmsChannelsOf                                                (cms)ChannelsOf                                              color management system (little cms) ??
cmsCloseIOhandler                                            (cms)CloseIOhandler                                          color management system (little cms) ??
cmsCloseProfile                                              (cms)CloseProfile                                            color management system (little cms) ??
cmsCreateBCHSWabstractProfile                                (cms)CreateBCHSWabstractProfile                              color management system (little cms) ??
cmsCreateBCHSWabstractProfileTHR                             (cms)CreateBCHSWabstractProfileTHR                           color management system (little cms) ??
cmsCreateContext                                             (cms)CreateContext                                           color management system (little cms) ??
cmsCreateExtendedTransform                                   (cms)CreateExtendedTransform                                 color management system (little cms) ??
cmsCreateGrayProfile                                         (cms)CreateGrayProfile                                       color management system (little cms) ??
cmsCreateGrayProfileTHR                                      (cms)CreateGrayProfileTHR                                    color management system (little cms) ??
cmsCreateInkLimitingDeviceLink                               (cms)CreateInkLimitingDeviceLink                             color management system (little cms) ??
cmsCreateInkLimitingDeviceLinkTHR                            (cms)CreateInkLimitingDeviceLinkTHR                          color management system (little cms) ??
cmsCreateLab2Profile                                         (cms)CreateLab2Profile                                       color management system (little cms) ??
cmsCreateLab2ProfileTHR                                      (cms)CreateLab2ProfileTHR                                    color management system (little cms) ??
cmsCreateLab4Profile                                         (cms)CreateLab4Profile                                       color management system (little cms) ??
cmsCreateLab4ProfileTHR                                      (cms)CreateLab4ProfileTHR                                    color management system (little cms) ??
cmsCreateLinearizationDeviceLink                             (cms)CreateLinearizationDeviceLink                           color management system (little cms) ??
cmsCreateLinearizationDeviceLinkTHR                          (cms)CreateLinearizationDeviceLinkTHR                        color management system (little cms) ??
cmsCreateMultiprofileTransform                               (cms)CreateMultiprofileTransform                             color management system (little cms) ??
cmsCreateMultiprofileTransformTHR                            (cms)CreateMultiprofileTransformTHR                          color management system (little cms) ??
cmsCreateNULLProfile                                         (cms)CreateNULLProfile                                       color management system (little cms) ??
cmsCreateNULLProfileTHR                                      (cms)CreateNULLProfileTHR                                    color management system (little cms) ??
cmsCreateProfilePlaceholder                                  (cms)CreateProfilePlaceholder                                color management system (little cms) ??
cmsCreateProofingTransform                                   (cms)CreateProofingTransform                                 color management system (little cms) ??
cmsCreateProofingTransformTHR                                (cms)CreateProofingTransformTHR                              color management system (little cms) ??
cmsCreateRGBProfile                                          (cms)CreateRGBProfile                                        color management system (little cms) ??
cmsCreateRGBProfileTHR                                       (cms)CreateRGBProfileTHR                                     color management system (little cms) ??
cmsCreateTransform                                           (cms)CreateTransform                                         color management system (little cms) ??
cmsCreateTransformTHR                                        (cms)CreateTransformTHR                                      color management system (little cms) ??
cmsCreateXYZProfile                                          (cms)CreateXYZProfile                                        color management system (little cms) ??
cmsCreateXYZProfileTHR                                       (cms)CreateXYZProfileTHR                                     color management system (little cms) ??
cmsCreate_sRGBProfile                                        (cms)Create_sRGBProfile                                      color management system (little cms) ??
cmsCreate_sRGBProfileTHR                                     (cms)Create_sRGBProfileTHR                                   color management system (little cms) ??
cmsD50_XYZ                                                   (cms)D50_XYZ                                                 color management system (little cms) ??
cmsD50_xyY                                                   (cms)D50_xyY                                                 color management system (little cms) ??
cmsDeleteContext                                             (cms)DeleteContext                                           color management system (little cms) ??
cmsDeleteTransform                                           (cms)DeleteTransform                                         color management system (little cms) ??
cmsDeltaE                                                    (cms)DeltaE                                                  color management system (little cms) ??
cmsDesaturateLab                                             (cms)DesaturateLab                                           color management system (little cms) ??
cmsDetectBlackPoint                                          (cms)DetectBlackPoint                                        color management system (little cms) ??
cmsDetectDestinationBlackPoint                               (cms)DetectDestinationBlackPoint                             color management system (little cms) ??
cmsDetectTAC                                                 (cms)DetectTAC                                               color management system (little cms) ??
cmsDictAddEntry                                              (cms)DictAddEntry                                            color management system (little cms) ??
cmsDictAlloc                                                 (cms)DictAlloc                                               color management system (little cms) ??
cmsDictDup                                                   (cms)DictDup                                                 color management system (little cms) ??
cmsDictFree                                                  (cms)DictFree                                                color management system (little cms) ??
cmsDictGetEntryList                                          (cms)DictGetEntryList                                        color management system (little cms) ??
cmsDictNextEntry                                             (cms)DictNextEntry                                           color management system (little cms) ??
cmsDoTransform                                               (cms)DoTransform                                             color management system (little cms) ??
cmsDoTransformLineStride                                     (cms)DoTransformLineStride                                   color management system (little cms) ??
cmsDoTransformStride                                         (cms)DoTransformStride                                       color management system (little cms) ??
cmsDupContext                                                (cms)DupContext                                              color management system (little cms) ??
cmsDupNamedColorList                                         (cms)DupNamedColorList                                       color management system (little cms) ??
cmsDupProfileSequenceDescription                             (cms)DupProfileSequenceDescription                           color management system (little cms) ??
cmsDupToneCurve                                              (cms)DupToneCurve                                            color management system (little cms) ??
cmsEstimateGamma                                             (cms)EstimateGamma                                           color management system (little cms) ??
cmsEvalToneCurve16                                           (cms)EvalToneCurve16                                         color management system (little cms) ??
cmsEvalToneCurveFloat                                        (cms)EvalToneCurveFloat                                      color management system (little cms) ??
cmsFloat2LabEncoded                                          (cms)Float2LabEncoded                                        color management system (little cms) ??
cmsFloat2LabEncodedV2                                        (cms)Float2LabEncodedV2                                      color management system (little cms) ??
cmsFloat2XYZEncoded                                          (cms)Float2XYZEncoded                                        color management system (little cms) ??
cmsFormatterForColorspaceOfProfile                           (cms)FormatterForColorspaceOfProfile                         color management system (little cms) ??
cmsFormatterForPCSOfProfile                                  (cms)FormatterForPCSOfProfile                                color management system (little cms) ??
cmsFreeNamedColorList                                        (cms)FreeNamedColorList                                      color management system (little cms) ??
cmsFreeProfileSequenceDescription                            (cms)FreeProfileSequenceDescription                          color management system (little cms) ??
cmsFreeToneCurve                                             (cms)FreeToneCurve                                           color management system (little cms) ??
cmsFreeToneCurveTriple                                       (cms)FreeToneCurveTriple                                     color management system (little cms) ??
cmsGetAlarmCodes                                             (cms)GetAlarmCodes                                           color management system (little cms) ??
cmsGetAlarmCodesTHR                                          (cms)GetAlarmCodesTHR                                        color management system (little cms) ??
cmsGetColorSpace                                             (cms)GetColorSpace                                           color management system (little cms) ??
cmsGetContextUserData                                        (cms)GetContextUserData                                      color management system (little cms) ??
cmsGetDeviceClass                                            (cms)GetDeviceClass                                          color management system (little cms) ??
cmsGetEncodedCMMversion                                      (cms)GetEncodedCMMversion                                    color management system (little cms) ??
cmsGetEncodedICCversion                                      (cms)GetEncodedICCversion                                    color management system (little cms) ??
cmsGetHeaderAttributes                                       (cms)GetHeaderAttributes                                     color management system (little cms) ??
cmsGetHeaderCreationDateTime                                 (cms)GetHeaderCreationDateTime                               color management system (little cms) ??
cmsGetHeaderCreator                                          (cms)GetHeaderCreator                                        color management system (little cms) ??
cmsGetHeaderFlags                                            (cms)GetHeaderFlags                                          color management system (little cms) ??
cmsGetHeaderManufacturer                                     (cms)GetHeaderManufacturer                                   color management system (little cms) ??
cmsGetHeaderModel                                            (cms)GetHeaderModel                                          color management system (little cms) ??
cmsGetHeaderProfileID                                        (cms)GetHeaderProfileID                                      color management system (little cms) ??
cmsGetHeaderRenderingIntent                                  (cms)GetHeaderRenderingIntent                                color management system (little cms) ??
cmsGetNamedColorList                                         (cms)GetNamedColorList                                       color management system (little cms) ??
cmsGetPCS                                                    (cms)GetPCS                                                  color management system (little cms) ??
cmsGetPipelineContextID                                      (cms)GetPipelineContextID                                    color management system (little cms) ??
cmsGetProfileContextID                                       (cms)GetProfileContextID                                     color management system (little cms) ??
cmsGetProfileIOhandler                                       (cms)GetProfileIOhandler                                     color management system (little cms) ??
cmsGetProfileInfo                                            (cms)GetProfileInfo                                          color management system (little cms) ??
cmsGetProfileInfoASCII                                       (cms)GetProfileInfoASCII                                     color management system (little cms) ??
cmsGetProfileVersion                                         (cms)GetProfileVersion                                       color management system (little cms) ??
cmsGetSupportedIntents                                       (cms)GetSupportedIntents                                     color management system (little cms) ??
cmsGetSupportedIntentsTHR                                    (cms)GetSupportedIntentsTHR                                  color management system (little cms) ??
cmsGetTagCount                                               (cms)GetTagCount                                             color management system (little cms) ??
cmsGetTagSignature                                           (cms)GetTagSignature                                         color management system (little cms) ??
cmsGetToneCurveEstimatedTable                                (cms)GetToneCurveEstimatedTable                              color management system (little cms) ??
cmsGetToneCurveEstimatedTableEntries                         (cms)GetToneCurveEstimatedTableEntries                       color management system (little cms) ??
cmsGetToneCurveParametricType                                (cms)GetToneCurveParametricType                              color management system (little cms) ??
cmsGetTransformContextID                                     (cms)GetTransformContextID                                   color management system (little cms) ??
cmsGetTransformInputFormat                                   (cms)GetTransformInputFormat                                 color management system (little cms) ??
cmsGetTransformOutputFormat                                  (cms)GetTransformOutputFormat                                color management system (little cms) ??
cmsIsCLUT                                                    (cms)IsCLUT                                                  color management system (little cms) ??
cmsIsIntentSupported                                         (cms)IsIntentSupported                                       color management system (little cms) ??
cmsIsMatrixShaper                                            (cms)IsMatrixShaper                                          color management system (little cms) ??
cmsIsTag                                                     (cms)IsTag                                                   color management system (little cms) ??
cmsIsToneCurveDescending                                     (cms)IsToneCurveDescending                                   color management system (little cms) ??
cmsIsToneCurveLinear                                         (cms)IsToneCurveLinear                                       color management system (little cms) ??
cmsIsToneCurveMonotonic                                      (cms)IsToneCurveMonotonic                                    color management system (little cms) ??
cmsIsToneCurveMultisegment                                   (cms)IsToneCurveMultisegment                                 color management system (little cms) ??
cmsJoinToneCurve                                             (cms)JoinToneCurve                                           color management system (little cms) ??
cmsLCh2Lab                                                   (cms)LCh2Lab                                                 color management system (little cms) ??
cmsLab2LCh                                                   (cms)Lab2LCh                                                 color management system (little cms) ??
cmsLab2XYZ                                                   (cms)Lab2XYZ                                                 color management system (little cms) ??
cmsLabEncoded2Float                                          (cms)LabEncoded2Float                                        color management system (little cms) ??
cmsLabEncoded2FloatV2                                        (cms)LabEncoded2FloatV2                                      color management system (little cms) ??
cmsLinkTag                                                   (cms)LinkTag                                                 color management system (little cms) ??
cmsMLUalloc                                                  (cms)MLUalloc                                                color management system (little cms) ??
cmsMLUdup                                                    (cms)MLUdup                                                  color management system (little cms) ??
cmsMLUfree                                                   (cms)MLUfree                                                 color management system (little cms) ??
cmsMLUgetASCII                                               (cms)MLUgetASCII                                             color management system (little cms) ??
cmsMLUgetTranslation                                         (cms)MLUgetTranslation                                       color management system (little cms) ??
cmsMLUgetWide                                                (cms)MLUgetWide                                              color management system (little cms) ??
cmsMLUsetASCII                                               (cms)MLUsetASCII                                             color management system (little cms) ??
cmsMLUsetWide                                                (cms)MLUsetWide                                              color management system (little cms) ??
cmsMLUtranslationsCodes                                      (cms)MLUtranslationsCodes                                    color management system (little cms) ??
cmsMLUtranslationsCount                                      (cms)MLUtranslationsCount                                    color management system (little cms) ??
cmsNamedColorCount                                           (cms)NamedColorCount                                         color management system (little cms) ??
cmsNamedColorIndex                                           (cms)NamedColorIndex                                         color management system (little cms) ??
cmsNamedColorInfo                                            (cms)NamedColorInfo                                          color management system (little cms) ??
cmsOpenIOhandlerFromFile                                     (cms)OpenIOhandlerFromFile                                   color management system (little cms) ??
cmsOpenIOhandlerFromMem                                      (cms)OpenIOhandlerFromMem                                    color management system (little cms) ??
cmsOpenIOhandlerFromNULL                                     (cms)OpenIOhandlerFromNULL                                   color management system (little cms) ??
cmsOpenIOhandlerFromStream                                   (cms)OpenIOhandlerFromStream                                 color management system (little cms) ??
cmsOpenProfileFromFile                                       (cms)OpenProfileFromFile                                     color management system (little cms) ??
cmsOpenProfileFromFileTHR                                    (cms)OpenProfileFromFileTHR                                  color management system (little cms) ??
cmsOpenProfileFromIOhandler2THR                              (cms)OpenProfileFromIOhandler2THR                            color management system (little cms) ??
cmsOpenProfileFromIOhandlerTHR                               (cms)OpenProfileFromIOhandlerTHR                             color management system (little cms) ??
cmsOpenProfileFromMem                                        (cms)OpenProfileFromMem                                      color management system (little cms) ??
cmsOpenProfileFromMemTHR                                     (cms)OpenProfileFromMemTHR                                   color management system (little cms) ??
cmsOpenProfileFromStream                                     (cms)OpenProfileFromStream                                   color management system (little cms) ??
cmsOpenProfileFromStreamTHR                                  (cms)OpenProfileFromStreamTHR                                color management system (little cms) ??
cmsPipelineAlloc                                             (cms)PipelineAlloc                                           color management system (little cms) ??
cmsPipelineCat                                               (cms)PipelineCat                                             color management system (little cms) ??
cmsPipelineCheckAndRetreiveStages                            (cms)PipelineCheckAndRetreiveStages                          color management system (little cms) ??
cmsPipelineDup                                               (cms)PipelineDup                                             color management system (little cms) ??
cmsPipelineEval16                                            (cms)PipelineEval16                                          color management system (little cms) ??
cmsPipelineEvalFloat                                         (cms)PipelineEvalFloat                                       color management system (little cms) ??
cmsPipelineEvalReverseFloat                                  (cms)PipelineEvalReverseFloat                                color management system (little cms) ??
cmsPipelineFree                                              (cms)PipelineFree                                            color management system (little cms) ??
cmsPipelineGetPtrToFirstStage                                (cms)PipelineGetPtrToFirstStage                              color management system (little cms) ??
cmsPipelineGetPtrToLastStage                                 (cms)PipelineGetPtrToLastStage                               color management system (little cms) ??
cmsPipelineInputChannels                                     (cms)PipelineInputChannels                                   color management system (little cms) ??
cmsPipelineInsertStage                                       (cms)PipelineInsertStage                                     color management system (little cms) ??
cmsPipelineOutputChannels                                    (cms)PipelineOutputChannels                                  color management system (little cms) ??
cmsPipelineSetSaveAs8bitsFlag                                (cms)PipelineSetSaveAs8bitsFlag                              color management system (little cms) ??
cmsPipelineStageCount                                        (cms)PipelineStageCount                                      color management system (little cms) ??
cmsPipelineUnlinkStage                                       (cms)PipelineUnlinkStage                                     color management system (little cms) ??
cmsPlugin                                                    (cms)Plugin                                                  color management system (little cms) ??
cmsPluginTHR                                                 (cms)PluginTHR                                               color management system (little cms) ??
cmsReadRawTag                                                (cms)ReadRawTag                                              color management system (little cms) ??
cmsReadTag                                                   (cms)ReadTag                                                 color management system (little cms) ??
cmsReverseToneCurve                                          (cms)ReverseToneCurve                                        color management system (little cms) ??
cmsReverseToneCurveEx                                        (cms)ReverseToneCurveEx                                      color management system (little cms) ??
cmsSaveProfileToFile                                         (cms)SaveProfileToFile                                       color management system (little cms) ??
cmsSaveProfileToIOhandler                                    (cms)SaveProfileToIOhandler                                  color management system (little cms) ??
cmsSaveProfileToMem                                          (cms)SaveProfileToMem                                        color management system (little cms) ??
cmsSaveProfileToStream                                       (cms)SaveProfileToStream                                     color management system (little cms) ??
cmsSetAdaptationState                                        (cms)SetAdaptationState                                      color management system (little cms) ??
cmsSetAdaptationStateTHR                                     (cms)SetAdaptationStateTHR                                   color management system (little cms) ??
cmsSetAlarmCodes                                             (cms)SetAlarmCodes                                           color management system (little cms) ??
cmsSetAlarmCodesTHR                                          (cms)SetAlarmCodesTHR                                        color management system (little cms) ??
cmsSetColorSpace                                             (cms)SetColorSpace                                           color management system (little cms) ??
cmsSetDeviceClass                                            (cms)SetDeviceClass                                          color management system (little cms) ??
cmsSetEncodedICCversion                                      (cms)SetEncodedICCversion                                    color management system (little cms) ??
cmsSetHeaderAttributes                                       (cms)SetHeaderAttributes                                     color management system (little cms) ??
cmsSetHeaderFlags                                            (cms)SetHeaderFlags                                          color management system (little cms) ??
cmsSetHeaderManufacturer                                     (cms)SetHeaderManufacturer                                   color management system (little cms) ??
cmsSetHeaderModel                                            (cms)SetHeaderModel                                          color management system (little cms) ??
cmsSetHeaderProfileID                                        (cms)SetHeaderProfileID                                      color management system (little cms) ??
cmsSetHeaderRenderingIntent                                  (cms)SetHeaderRenderingIntent                                color management system (little cms) ??
cmsSetLogErrorHandler                                        (cms)SetLogErrorHandler                                      color management system (little cms) ??
cmsSetLogErrorHandlerTHR                                     (cms)SetLogErrorHandlerTHR                                   color management system (little cms) ??
cmsSetPCS                                                    (cms)SetPCS                                                  color management system (little cms) ??
cmsSetProfileVersion                                         (cms)SetProfileVersion                                       color management system (little cms) ??
cmsSignalError                                               (cms)SignalError                                             color management system (little cms) ??
cmsSliceSpace16                                              (cms)SliceSpace16                                            color management system (little cms) ??
cmsSliceSpaceFloat                                           (cms)SliceSpaceFloat                                         color management system (little cms) ??
cmsSmoothToneCurve                                           (cms)SmoothToneCurve                                         color management system (little cms) ??
cmsStageAllocCLut16bit                                       (cms)StageAllocCLut16bit                                     color management system (little cms) ??
cmsStageAllocCLut16bitGranular                               (cms)StageAllocCLut16bitGranular                             color management system (little cms) ??
cmsStageAllocCLutFloat                                       (cms)StageAllocCLutFloat                                     color management system (little cms) ??
cmsStageAllocCLutFloatGranular                               (cms)StageAllocCLutFloatGranular                             color management system (little cms) ??
cmsStageAllocIdentity                                        (cms)StageAllocIdentity                                      color management system (little cms) ??
cmsStageAllocMatrix                                          (cms)StageAllocMatrix                                        color management system (little cms) ??
cmsStageAllocToneCurves                                      (cms)StageAllocToneCurves                                    color management system (little cms) ??
cmsStageData                                                 (cms)StageData                                               color management system (little cms) ??
cmsStageDup                                                  (cms)StageDup                                                color management system (little cms) ??
cmsStageFree                                                 (cms)StageFree                                               color management system (little cms) ??
cmsStageInputChannels                                        (cms)StageInputChannels                                      color management system (little cms) ??
cmsStageNext                                                 (cms)StageNext                                               color management system (little cms) ??
cmsStageOutputChannels                                       (cms)StageOutputChannels                                     color management system (little cms) ??
cmsStageSampleCLut16bit                                      (cms)StageSampleCLut16bit                                    color management system (little cms) ??
cmsStageSampleCLutFloat                                      (cms)StageSampleCLutFloat                                    color management system (little cms) ??
cmsStageType                                                 (cms)StageType                                               color management system (little cms) ??
cmsTagLinkedTo                                               (cms)TagLinkedTo                                             color management system (little cms) ??
cmsTempFromWhitePoint                                        (cms)TempFromWhitePoint                                      color management system (little cms) ??
cmsTransform2DeviceLink                                      (cms)Transform2DeviceLink                                    color management system (little cms) ??
cmsUnregisterPlugins                                         (cms)UnregisterPlugins                                       color management system (little cms) ??
cmsUnregisterPluginsTHR                                      (cms)UnregisterPluginsTHR                                    color management system (little cms) ??
cmsWhitePointFromTemp                                        (cms)WhitePointFromTemp                                      color management system (little cms) ??
cmsWriteRawTag                                               (cms)WriteRawTag                                             color management system (little cms) ??
cmsWriteTag                                                  (cms)WriteTag                                                color management system (little cms) ??
cmsXYZ2Lab                                                   (cms)XYZ2Lab                                                 color management system (little cms) ??
cmsXYZ2xyY                                                   (cms)XYZ2xyY                                                 color management system (little cms) ??
cmsXYZEncoded2Float                                          (cms)XYZEncoded2Float                                        color management system (little cms) ??
cmsfilelength                                                (cms)filelength                                              color management system (little cms) ??
cmsstrcasecmp                                                (cms)strcasecmp                                              color management system (little cms) ??
cmsxyY2XYZ                                                   (cms)xyY2XYZ                                                 color management system (little cms) ??
copy16                                                       copy16                                                       ??
copy32                                                       copy32                                                       ??
copy64                                                       copy64                                                       ??
copy8                                                        copy8                                                        ??
defMtxCreate                                                 defMtxCreate                                                 ??
defMtxDestroy                                                defMtxDestroy                                                ??
defMtxLock                                                   defMtxLock                                                   ??
defMtxUnlock                                                 defMtxUnlock                                                 ??
errorHandler                                                 (err)orHandler                                               error recovery ??
free_ex_data_arg                                             free_ex_data_arg                                             ??
from16to8                                                    from16to8                                                    ??
from16toDBL                                                  from16toDBL                                                  ??
from16toFLT                                                  from16toFLT                                                  ??
from16toHLF                                                  from16toHLF                                                  ??
from8to16                                                    from8to16                                                    ??
from8toDBL                                                   from8toDBL                                                   ??
from8toFLT                                                   from8toFLT                                                   ??
from8toHLF                                                   from8toHLF                                                   ??
fromDBLto16                                                  fromDBLto16                                                  ??
fromDBLto8                                                   fromDBLto8                                                   ??
fromDBLtoFLT                                                 fromDBLtoFLT                                                 ??
fromDBLtoHLF                                                 fromDBLtoHLF                                                 ??
fromFLTto16                                                  fromFLTto16                                                  ??
fromFLTto8                                                   fromFLTto8                                                   ??
fromFLTtoDBL                                                 fromFLTtoDBL                                                 ??
fromFLTtoHLF                                                 fromFLTtoHLF                                                 ??
fromHLFto16                                                  fromHLFto16                                                  ??
fromHLFto8                                                   fromHLFto8                                                   ??
fromHLFtoDBL                                                 fromHLFtoDBL                                                 ??
fromHLFtoFLT                                                 fromHLFtoFLT                                                 ??
getILData                                                    getILData                                                    ??
kcfis_oss_wait_per_ctx                                       (kcfis)_oss_wait_per_ctx                                     kernel cache file management intelligent storage ??
kcfis_set_appliance_ctx                                      (kcfis)_set_appliance_ctx                                    kernel cache file management intelligent storage ??
kcfis_tablespace_is_smart_scannable                          (kcfis)_tablespace_is_smart_scannable                        kernel cache file management intelligent storage ??
kdmoCheckGbyAgg                                              (kdmo)CheckGbyAgg                                            kernel data in-memory data layer optimizer ??
kdmoCheckGbyOpn                                              (kdmo)CheckGbyOpn                                            kernel data in-memory data layer optimizer ??
kdmoColsUlvlIsDict                                           (kdmo)ColsUlvlIsDict                                         kernel data in-memory data layer optimizer ??
kdmoGByPushdownCardEst                                       (kdmo)GByPushdownCardEst                                     kernel data in-memory data layer optimizer ??
kdmoGByPushdownValid                                         (kdmo)GByPushdownValid                                       kernel data in-memory data layer optimizer ??
kdmoGenericPcodeCheck                                        (kdmo)GenericPcodeCheck                                      kernel data in-memory data layer optimizer ??
kdmoStorageColCheck                                          (kdmo)StorageColCheck                                        kernel data in-memory data layer optimizer ??
kdmoStorageConstantCheck                                     (kdmo)StorageConstantCheck                                   kernel data in-memory data layer optimizer ??
kdmoStorageOpnCheckAux                                       (kdmo)StorageOpnCheckAux                                     kernel data in-memory data layer optimizer ??
kdmoTabUlvlIsDict                                            (kdmo)TabUlvlIsDict                                          kernel data in-memory data layer optimizer ??
kdmoValidGbyPushdownPredCB                                   (kdmo)ValidGbyPushdownPredCB                                 kernel data in-memory data layer optimizer ??
kdzfCheckDbaMatch                                            (kdz)fCheckDbaMatch                                          kernel data archive compression ??
kdzsDumpKaf                                                  (kdzs)DumpKaf                                                kernel data archive compression decompression ??
kfgbCheckInterrupt                                           (kfgb)CheckInterrupt                                         kernel automatic storage management diskgroups background ??
kgh_check_simple_free_canary                                 (kgh)_check_simple_free_canary                               kernel generic heap manager ??
kgh_get_max_size_on_list                                     (kgh)_get_max_size_on_list                                   kernel generic heap manager ??
kgh_get_visit_limit                                          (kgh)_get_visit_limit                                        kernel generic heap manager ??
kgh_set_fl_effort                                            (kgh)_set_fl_effort                                          kernel generic heap manager ??
kghsrch_best_fit                                             (kghsrch)_best_fit                                           kernel generic heap manager search freelists for a memory chunk ??
kghunalo                                                     (kgh)unalo                                                   kernel generic heap manager ??
kgzm_encode_identify_ctype                                   (kg)zm_encode_identify_ctype                                 kernel generic ??
kgzm_encode_version_with_reid                                (kg)zm_encode_version_with_reid                              kernel generic ??
kjbmisgdbabast                                               (kjb)misgdbabast                                             kernel lock management global cache service ??
kjbmrejectbast                                               (kjb)mrejectbast                                             kernel lock management global cache service ??
kkesrcCard                                                   (kke)srcCard                                                 kernel compile cost engine ??
kkqjpdrace                                                   (kkqjpd)race                                                 kernel compile query  join analysis predicate push down ??
kkqoreIsUnsupported                                          (kkqore)IsUnsupported                                        kernel compile query  or-expansion ??
klxGetBuffer                                                 (kl)xGetBuffer                                               kernel loader ??
klxGetLength                                                 (kl)xGetLength                                               kernel loader ??
koklc_screate                                                (kokl)c_screate                                              kernel objects kernel side lob access ??
kolaCreateFromExternalSrc                                    (kola)CreateFromExternalSrc                                  kernel objects lob ??
kolaetAssign                                                 (kola)etAssign                                               kernel objects lob ??
kolaetChkSize                                                (kola)etChkSize                                              kernel objects lob ??
kolaetCreate                                                 (kola)etCreate                                               kernel objects lob ??
kolaetCreateCtx                                              (kola)etCreateCtx                                            kernel objects lob ??
kolaetCreateExternalInline                                   (kola)etCreateExternalInline                                 kernel objects lob ??
kolaetCreateFromExternalInline                               (kola)etCreateFromExternalInline                             kernel objects lob ??
kolaetDmpData                                                (kola)etDmpData                                              kernel objects lob ??
kolaetFree                                                   (kola)etFree                                                 kernel objects lob ??
kolaetGetLength                                              (kola)etGetLength                                            kernel objects lob ??
kolaetGetRawData                                             (kola)etGetRawData                                           kernel objects lob ??
kolaetIsExternalInlineLob                                    (kola)etIsExternalInlineLob                                  kernel objects lob ??
kolaetRead                                                   (kola)etRead                                                 kernel objects lob ??
kolaetWrite                                                  (kola)etWrite                                                kernel objects lob ??
kolamalSage                                                  (kola)malSage                                                kernel objects lob ??
kolamfrSage                                                  (kola)mfrSage                                                kernel objects lob ??
kolasugc                                                     (kola)sugc                                                   kernel objects lob ??
kolasugi                                                     (kola)sugi                                                   kernel objects lob ??
kolrsdesht                                                   (kolr)sdesht                                                 kernel objects lob refcount ??
kolrsugi                                                     (kolr)sugi                                                   kernel objects lob refcount ??
kqltprom                                                     (kql)tprom                                                   kernel query library cache ??
krbCreateGuidStr                                             (krb)CreateGuidStr                                           kernel redo backup/restore ??
krbrCheckLogical                                             (krbr)CheckLogical                                           kernel redo backup/restore restore and recovery ??
krvxdsr                                                      (krvx)dsr                                                    kernel redo recovery extract ??
ksmsq_memlock_limit_alert_error                              (ksmsq)_memlock_limit_alert_error                            kernel service (VOS) memory sga heap message queue services ??
ksmsq_transport_require_memlock                              (ksmsq)_transport_require_memlock                            kernel service (VOS) memory sga heap message queue services ??
ksz_oss_get_ip_info                                          (ksz)_oss_get_ip_info                                        kernel service (VOS) oracle storage server (OSS) server layer ??
ksz_skgxp_check_default_query                                (ksz)_skgxp_check_default_query                              kernel service (VOS) oracle storage server (OSS) server layer ??
ktmaLgwrHeartbeat                                            (ktma)LgwrHeartbeat                                          kernel transaction transaction monitor (smon) IM transaction ADG ??
ktsttsn_to_tsnpdb                                            (ktst)tsn_to_tsnpdb                                          kernel transaction segment management sort management ??
ktsttsnpdb_to_pdb                                            (ktst)tsnpdb_to_pdb                                          kernel transaction segment management sort management ??
ktsttsnpdb_to_tsn                                            (ktst)tsnpdb_to_tsn                                          kernel transaction segment management sort management ??
kubsjniSkip                                                  (ku)bsjniSkip                                                kernel utility ??
kubsxiMapCluUser                                             (ku)bsxiMapCluUser                                           kernel utility ??
kubsxiSetCpx                                                 (ku)bsxiSetCpx                                               kernel utility ??
kxfrGetNextSplit                                             (kxfr)GetNextSplit                                           kernel execution parallel query granules ??
kzvdvechk_ownerid_1                                          (kzvd)vechk_ownerid_1                                        kernel security data vault ??
oracle_storidx_recompute_ridx_summary                        oracle_storidx_recompute_ridx_summary                        ??
prsUnusable                                                  (prs)Unusable                                                parse ??
prscViewColAlias                                             (prs)cViewColAlias                                           parse ??
prsc_edition                                                 (prs)c_edition                                               parse ??
prscal                                                       (prs)cal                                                     parse ??
prscap                                                       (prs)cap                                                     parse ??
prscbl                                                       (prs)cbl                                                     parse ??
prsccc                                                       (prs)ccc                                                     parse ??
prscct                                                       (prs)cct                                                     parse ??
prscdg                                                       (prs)cdg                                                     parse ??
prscdr                                                       (prs)cdr                                                     parse ??
prscet_parse_extab                                           (prs)cet_parse_extab                                         parse ??
prscipc_index_placement_clause                               (prs)cipc_index_placement_clause                             parse ??
prscisfbdrp                                                  (prs)cisfbdrp                                                parse ??
prscisfbtbl                                                  (prs)cisfbtbl                                                parse ??
prscnd                                                       (prs)cnd                                                     parse ??
prscpfil                                                     (prs)cpfil                                                   parse ??
prscrefv                                                     (prs)crefv                                                   parse ??
prscrely                                                     (prs)crely                                                   parse ??
prscspfi                                                     (prs)cspfi                                                   parse ??
prscvac                                                      (prs)cvac                                                    parse ??
prsdfl                                                       (prs)dfl                                                     parse ??
prsfbdrp                                                     (prs)fbdrp                                                   parse ??
prsfbdrptab                                                  (prs)fbdrptab                                                parse ??
prsfla                                                       (prs)fla                                                     parse ??
prshve_high_val_expr                                         (prs)hve_high_val_expr                                       parse ??
prsicsl                                                      (prs)icsl                                                    parse ??
prsident                                                     (prs)ident                                                   parse ??
prsini_iot_init                                              (prs)ini_iot_init                                            parse ??
prsref                                                       (prs)ref                                                     parse ??
prssoasn                                                     (prs)soasn                                                   parse ??
prssptoa                                                     (prs)sptoa                                                   parse ??
qcpiSetFnInOpn                                               (qcpi)SetFnInOpn                                             query compile parse interim ??
qcpiSetFnInSel                                               (qcpi)SetFnInSel                                             query compile parse interim ??
qertbFilter_qertbs                                           (qertb)Filter_qertbs                                         query execute rowsource table access ??
qertbInitXTFilter                                            (qertb)InitXTFilter                                          query execute rowsource table access ??
qertbQueryHXTFilter                                          (qertb)QueryHXTFilter                                        query execute rowsource table access ??
qertbQueryLXTFilter                                          (qertb)QueryLXTFilter                                        query execute rowsource table access ??
qertbUpdateXTFilter                                          (qertb)UpdateXTFilter                                        query execute rowsource table access ??
qertbXTCbk                                                   (qertb)XTCbk                                                 query execute rowsource table access ??
qesSageIsHDFSTsn                                             (qes)SageIsHDFSTsn                                           query execute services ??
qesxtcCloseScan                                              (qes)xtcCloseScan                                            query execute services ??
qesxtcFetchCurrScan                                          (qes)xtcFetchCurrScan                                        query execute services ??
qesxtcFetchNextScan                                          (qes)xtcFetchNextScan                                        query execute services ??
qesxtcOpenScan                                               (qes)xtcOpenScan                                             query execute services ??
qjsnGenerateCapabilitySet                                    (qjsn)GenerateCapabilitySet                                  query json ??
qjsngCloseSageUga                                            (qjsn)gCloseSageUga                                          query json ??
qjsngCloseUga                                                (qjsn)gCloseUga                                              query json ??
qjsngInitSageUga                                             (qjsn)gInitSageUga                                           query json ??
qjsngStreamFromLobInSage                                     (qjsn)gStreamFromLobInSage                                   query json ??
qjsngStreamFromLob_h                                         (qjsn)gStreamFromLob_h                                       query json ??
qkaQknSTPruneKaf                                             (qka)QknSTPruneKaf                                           query kernel allocation ??
qksSageFlushSgaCIC                                           (qksSage)FlushSgaCIC                                         query kernel sql exadata ??
qksSageFlushSgaCICDump                                       (qksSage)FlushSgaCICDump                                     query kernel sql exadata ??
qksSageFlushSgaImpl                                          (qksSage)FlushSgaImpl                                        query kernel sql exadata ??
qksSageLoadInternalExternalMD                                (qksSage)LoadInternalExternalMD                              query kernel sql exadata ??
qksSageMakeTbsHDFS                                           (qksSage)MakeTbsHDFS                                         query kernel sql exadata ??
qkssage_kkxbct                                               (qkssa)ge_kkxbct                                             query kernel sql sampling ??
r_ex_data_clear                                              r_ex_data_clear                                              ??
r_ex_data_update                                             r_ex_data_update                                             ??
r_ssl_ctx_ex_data_clear                                      r_ssl_ctx_ex_data_clear                                      ??
r_ssl_ctx_get_dh_uses                                        r_ssl_ctx_get_dh_uses                                        ??
r_ssl_ctx_set_dh_uses                                        r_ssl_ctx_set_dh_uses                                        ??
r_ssl_ec_cert_algs_are_equal                                 r_ssl_ec_cert_algs_are_equal                                 ??
r_ssl_get_dh_uses                                            r_ssl_get_dh_uses                                            ??
r_ssl_set_dh_uses                                            r_ssl_set_dh_uses                                            ??
releaseILData                                                releaseILData                                                ??
ri_ssl3_base                                                 ri_ssl3_base                                                 ??
ri_ssl3_ctx_dh_tmp                                           ri_ssl3_ctx_dh_tmp                                           ??
ri_ssl3_dh_tmp                                               ri_ssl3_dh_tmp                                               ??
ri_ssl_cert_dup_params                                       ri_ssl_cert_dup_params                                       ??
ri_ssl_cipher_ctx_cipher_size                                ri_ssl_cipher_ctx_cipher_size                                ??
ri_ssl_cipher_ctx_is_aead                                    ri_ssl_cipher_ctx_is_aead                                    ??
sageDataReInitCtx                                            (sageData)ReInitCtx                                          exadata specific data layer ??
sageetProcessInternalTable                                   (sage)etProcessInternalTable                                 exadata specific ??
sageet_fp_buf_corrupt                                        (sage)et_fp_buf_corrupt                                      exadata specific ??
sageet_fp_buf_noalloc                                        (sage)et_fp_buf_noalloc                                      exadata specific ??
sageet_fp_buf_noresize                                       (sage)et_fp_buf_noresize                                     exadata specific ??
sagesqlFindCidInStoridx                                      (sagesql)FindCidInStoridx                                    exadata specific sql ??
seqSetStartWith                                              (seq)SetStartWith                                            sequence numbers ??
seqclp                                                       (seq)clp                                                     sequence numbers ??
seqmmv                                                       (seq)mmv                                                     sequence numbers ??
seqsetmin                                                    (seq)setmin                                                  sequence numbers ??

Recently, I was trying to setup TDE. Doing that I found out the Oracle provided documentation isn’t overly clear, and there is a way to do it in pre-Oracle 12, which is done using ‘alter system’ commands, and a new-ish way to do it in Oracle 12, using ‘administer key management’ commands. I am using version 12.1.0.2.170117, so decided to use the ‘administer key management’ commands. This blogpost is about an exception which I see is encountered in the Januari 2017 (170117) PSU of the Oracle database, which is NOT happening in Oracle 12.2 (no PSU’s for Oracle 12.2 at the time of writing) and Oracle 12.1.0.2 April 2016 and October 2016 PSU’s.

In order to test the wallet functionality for TDE, I used the following commands:

SQL> select status, wrl_parameter from v$encryption_wallet;

STATUS
------------------------------
WRL_PARAMETER
--------------------------------------------------------------------------------
NOT_AVAILABLE
/u01/app/oracle/admin/test/wallet

SQL> !mkdir /u01/app/oracle/admin/test/wallet

SQL> administer key management create keystore '/u01/app/oracle/admin/test/wallet' identified by "this_is_the_keystore_password";

keystore altered.

SQL> administer key management set keystore open identified by "this_is_the_keystore_password";

keystore altered.

SQL> administer key management set key identified by "this_is_the_keystore_password" with backup;
administer key management set key identified by "this_is_the_keystore_password" with backup
*
ERROR at line 1:
ORA-28374: typed master key not found in wallet

SQL> select status, wrl_parameter from v$encryption_wallet;

STATUS
------------------------------
WRL_PARAMETER
--------------------------------------------------------------------------------
CLOSED
/u01/app/oracle/admin/test/wallet

SQL> administer key management set keystore open identified by "this_is_the_keystore_password";

keystore altered.

SQL> select status, wrl_parameter from v$encryption_wallet;

STATUS
------------------------------
WRL_PARAMETER
--------------------------------------------------------------------------------
OPEN
/u01/app/oracle/admin/test/wallet

Notes:
Line 1-10: The DB_UNIQUE_NAME of the instance is ‘test’, and therefore the default wallet location is /u01/app/oracle/admin/test/wallet (ORACLE_BASE/admin/DB_UNIQUE_NAME/wallet). The wallet directory doesn’t exist by default, so I created it (line 10).
Line 12: Here the keystore/wallet is created with a password.
Line 16: After the wallet is created without auto-login, the wallet must be opened using the ‘set keystore open’ command.
Line 20: After the wallet has been created, it does not contain a master key. This is done using the ‘set key’ command. However, this throws an ORA-28374 error.
Line 26: After an error involving the wallet has occurred, the wallet closes.
Line 35: The wallet can simply be opened using the earlier used ‘set keystore open’ command.
Line 39: This is where the surprise is: after opening, the master key “magically” appeared (visible by the status ‘OPEN’, without a master key this would be ‘OPEN_NO_MASTER_KEY’).

I yet have to start creating encrypted table spaces. There might be more surprises, I can’t tell at this moment because I didn’t try it. However, once I discovered this oddity, I talked to my colleague Matt who gave me his own runbook for enabling TDE, which turned out to be the exact same list of commands as I compiled, however he did not encounter the ORA-28374 which I did. I tested the same sequence of commands on 12.2.0.1, 12.1.0.2.161018 (October 2016) and 12.1.0.2.160419 (April 2016) and there the ORA-28374 was not raised during execution of the ‘set key’ command.

Update!
Reading through My Oracle Support note Master Note For Transparent Data Encryption ( TDE ) (Doc ID 1228046.1), I found the following text:

All the versions after 12.1.0.2

=====================

As of 12.1.0.2 If the key associated with the SYSTEM, SYSAUX or UNDO tablespaces is not present in the wallet you cannot associate a new master key with the database (i.e. you cannot activate that master key for the database) unless you set a hidden parameter :

SQL> administer key management use key ‘AUQukK/ZR0/iv26nuN9vIqcAAAAAAAAAAAAAAAAAAAAAAAAAAAAA’ identified by “welcome1” with backup;
administer key management use key ‘AUQukK/ZR0/iv26nuN9vIqcAAAAAAAAAAAAAAAAAAAAAAAAAAAAA’ identified by “welcome1” with backup
*
ERROR at line 1:
ORA-28374: typed master key not found in wallet

alter system set “_db_discard_lost_masterkey”=true;

SQL> administer key management use key ‘AUQukK/ZR0/iv26nuN9vIqcAAAAAAAAAAAAAAAAAAAAAAAAAAAAA’ identified by “welcome1” with backup;

The heading and first line read weird, the heading indicates the paragraph is about ‘all the versions after 12.1.0.2’ (which to me means 12.2), and the first line in the paragraph says ‘as of 12.1.0.2’, which very clearly says this is about version 12.1.0.2 and higher. However, a little further it shows the exact error (ORA-28374) I encountered, and explains that if a current key is used in the data dictionary (mind data dictionary, not wallet), you must set “_db_discard_lost_masterkey” to true before you can create and use another master key for a wallet if you start over (wipe or move the wallet directory).

This makes sense to me now! I tried dropping and creating new wallets in my current 170117 PSU instance, and only tried creating an encryption wallet in a brand new freshly created instance. So if I would have EXACTLY done the same in the instances with the other PSU’s, which is repeatedly create and drop a wallet for TDE, I would have encountered the same ORA-28374 error. Well…I see this as a safety mechanism, be it not a very obvious one, not exuberant documented, and probably causing more grief than it would save if you run into the need the change the master key.

This post is about memory management on the operating system level of an Oracle database. The first question that might pop in your head is: isn’t this a solved problem? The answer is: yes, if you use Oracle’s AMM (Automatic Memory Management) feature, which let’s you set a limit for the Oracle datababase’s two main memory area’s: SGA and PGA. But in my opinion any serious, real life, usage of an Oracle database on Linux will be (severely) constrained in performance because of the lack of huge pages with AMM, and I personally witnessed very strange behaviour and process deaths with the AMM feature and high demand for memory.

This means that I strongly advise customers to use Oracle’s ASMM (Automatic Shared Memory Management) feature. In the newer versions of 11.2 I found this to be working very well. Earlier versions like 10.2 could suffer from an ever growing shared pool (which also means an ever shrinking buffer cache), especially when bind variables weren’t used. This still could happen, but it seems the SGA memory management feature in 11.2 handles this well in most cases. The ASMM feature means a fixed memory area is allocated for the SGA. SGA allocation has always been fixed outside of the AMM feature, as far as I know.

When ASMM doesn’t work, meaning the memory areas are getting sized wrong and performance is influenced by that, the last option is to size the memory area’s yourself. However, since version 11.2.0.2 Oracle will resize when the memory manager thinks it’s feasible. See Kurt van Meerbeek’s article about that.

That leaves the PGA (Process Global Area) as a memory area on itself. Most databases are using the automatic PGA memory management, which is enabled once the PGA_AGGREGATE_TARGET parameter is set to a non zero value. A common misunderstanding is this setting is actually limiting the overall PGA usage of an instance. The truth is automatic PGA memory management will make attempts to adhere to the PGA_AGGREGATE_TARGET value. These are the actual words in the official Oracle documentation: ‘attempts to adhere’!

This means sort memory, hash memory and bitmap memory will be actively limited in size per process by automatic PGA memory management, any attempt to allocate more than automatic PGA memory management allows will result in moving some contents of these memory areas to the assigned temporary tablespace of the database user, to make room for new data.

However, there are more memory area’s allocatable per process, which are never swapped to disk, thus always will stay in memory, and these could not be limited in an officially supported way prior to Oracle version 12. Two structures which are allocated in PGA and never swapped to disk are PL/SQL collections and PL/SQL tables. Creating and filling these requires the usage of PL/SQL (hence their names); the reason for mentioning this is that if your database is not used by PL/SQL but only SQL, you almost certainly will not run into the problem I describe below.

You might be thinking: wait a minute! Does this mean a developer can just create such a structure, and allocate whatever he/she likes, with all the consequences that it can have, like the operating system starting to swap, and can do that for every single process? Yes, this is what this means. This is why Oracle introduced a parameter called PGA_AGGREGATE_LIMIT with Oracle 12, to effectively limit the overall PGA heap size.

In case you wonder what this means, or even doubting my words, I have written a little program to demonstrate this behaviour.

This is the source code to create my test table T2:

exec dbms_random.seed('abracadabra');
create table t2
as
with generator as (
    select      rownum      id
    from        dual
    connect by
                rownum <= 1000
)
select
    rownum                                                id,
    trunc((rownum-1)/50)                            clustered,
    mod(rownum,20000)                               scattered,
    trunc(dbms_random.value(0,20000))               randomized,
    trunc(sysdate) + dbms_random.value(-180, 180)   random_date,
    dbms_random.string('l',6)                       random_string,
    lpad(rownum,10,0)                               vc_small,
    rpad('x',100,'x')                               vc_padding
from
    generator   g1,
    generator   g2
where
    rownum <= 1000000
;
exec dbms_stats.gather_table_stats(null,'T2');

This is a very smart way to generate a table. I actually borrowed this from Jonathan Lewis.

Next up, I created a small anonymous PL/SQL block to take the contents from the T2 table, and store them in a collection until I hit the limit in the variable ‘grow_until’.

declare
	type sourcetab is table of t2%ROWTYPE;
	c_tmp		sourcetab;
	c_def		sourcetab	:= sourcetab();
	v_b_p		number		:= 0;
	v_c_p		number		:= 0;
	v_b_u		number		:= 0;
	v_c_u		number		:= 0;
	grow_until	number		:= 700000000;
	p_a_t		number;
begin
	select value into v_b_p from v$mystat m, v$statname n where m.statistic#=n.statistic# and name = 'session pga memory max';
	select value into v_b_u from v$mystat m, v$statname n where m.statistic#=n.statistic# and name = 'session uga memory max';
	select value into p_a_t from v$parameter where name = 'pga_aggregate_target';
	select * bulk collect into c_tmp from t2;
	while v_c_p < grow_until loop
		for c in c_tmp.first .. c_tmp.last loop
			c_def.extend(1);
			c_def(c_def.last) := c_tmp(c);
			select value into v_c_p from v$mystat m, v$statname n where m.statistic#=n.statistic# and name = 'session pga memory max';
			select value into v_c_u from v$mystat m, v$statname n where m.statistic#=n.statistic# and name = 'session uga memory max';
			if v_c_p >= grow_until then
				exit;
			end if;
		end loop;
	end loop;
	dbms_output.put_line('vbp : '||v_b_p);
	dbms_output.put_line('vcp : '||v_c_p);
	dbms_output.put_line('vbu : '||v_b_u);
	dbms_output.put_line('vcu : '||v_c_u);
	dbms_output.put_line('pat : '||p_a_t);
end;
/

Please mind the session needs to have create table, create session granted, enough quota in the default tablespace and select on v_$mystat, v_$parameter and v_$statname granted.

This is run on an Oracle 12.1.0.2 database:

TS@v12102 > @pga_filler
vbp : 3535368
vcp : 700051976
vbu : 1103192
vcu : 4755704
pat : 524288000

PL/SQL procedure successfully completed.

The begin sizes of the UGA (vbu) and PGA (vbp) are 1’103’192 and 3’535’368. The PGA_AGGREGATE_TARGET size is set to 524’288’000 (500MB). I did set the grow_until variable to 700’000’000 (roughly 700MB), which is more than PGA_AGGREGATE_TARGET. After running this, it’s easy to spot the values of vcu (UGA allocation) and vcp (PGA allocation). vcu grew to 4’755’704 during the run, however vcp grew to 700’051’976, a little more than 700MB! This shows that the collection is stored in the PGA, and that the collection grew beyond the value set with PGA_AGGREGATE_TARGET.

This behaviour is consistent in versions 12.1.0.1, 11.2.0.4, 11.2.0.3, 11.2.0.2 and 11.2.0.1.

Let me emphasise once again that the above proof of concept code managed to allocate more memory than was set for the overall PGA usage of the entire instance. This can have an enormous, devastating impact on a consolidated database setup (meaning having multiple instances running on a single machine). Typically, once memory consumption of all the processes exceeds physically available memory, the operating system tries to use the swap device, to which it will swap memory pages in and out depending on memory usage of active (=on CPU) processes. Mild swapping shows as severely slowed-down processing (because a number of memory pages for processing need to be read from the swap device and placed in memory, from which the former contents need to be written to the swap device), heavy swapping shows as the machine coming down to a standstill.

Please mind that a diagnosis on the state of memory usage (alias swapping), just by looking at the amount of used swap (as can be seen in the ‘top’ output, or ‘swapon -s’) could be misleading. It’s also important to look at actual swapping in and out, as can be seen with ‘vmstat 1’ (si/so columns) or swap -W. I’ve found several systems which had been running for some time (approximately longer than a month) that had swap usage, sometimes up to 40%, while no ‘active swapping’, so memory pages being transfered to and from the swap device, was happening.

Luckily, starting with Oracle 12 you can actually limit overall PGA usage using the parameter PGA_AGGREGATE_LIMIT. The default value is the greater of (list from Oracle documentation):
a) 2GB
b) 200% of PGA_AGGREGATE_TARGET parameter (or lower if 200% > (90% of physical memory – total SGA size) but not below 100%)
c) 3MB * PROCESSES parameter
The parameter can not set below it’s default value, except when set in a pfile or spfile.

Let’s set the PGA_AGGREGATE_LIMIT to 600MB and see what happens when we start doing a large allocation again:

SQL> alter system set pga_aggregate_limit=600m scope=spfile;

System altered.

SQL> startup force;

Okay, let’s run the pga_filler.sql script again, and try to allocate 900MB. This means the “grow_until” variable must be set to 900000000.
PLEASE MIND this is done as a regular user, the SYS user and background processes other than job queue processes are not subject to the limiting.

TS@v12102 > @pga_filler
declare
*
ERROR at line 1:
ORA-01423: error encountered while checking for extra rows in exact fetch
ORA-00039: error during periodic action
ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT
ORA-06512: at line 21

Great! Exactly like we expect, right?
Well…yes, but let’s look at the alert.log

Sat Dec 13 15:08:57 2014
Errors in file /u01/app/oracle/diag/rdbms/v12102/v12102/trace/v12102_ora_4147.trc  (incident=46599):
ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT
Incident details in: /u01/app/oracle/diag/rdbms/v12102/v12102/incident/incdir_46599/v12102_ora_4147_i46599.trc
Sat Dec 13 15:09:07 2014
Dumping diagnostic data in directory=[cdmp_20141213150907], requested by (instance=1, osid=4147), summary=[incident=46599].
Sat Dec 13 15:09:09 2014
Sweep [inc][46599]: completed
Sweep [inc2][46599]: completed

Okay, essentially, this tells us nothing interesting, except for the tracefile. Let’s look in/u01/app/oracle/diag/rdbms/v12102/v12102/trace/v12102_ora_4147.trc, being the tracefile as indicated in the above alert.log snippet:

*** 2014-12-13 15:08:57.351
Process may have gone over pga_aggregate_limit
Just allocated 65536 bytes
Dumping short stack in preparation for potential ORA-4036
----- Abridged Call Stack Trace -----
ksedsts()+244<-ksm_pga_limit_short_stack()+1016<-ksm_check_over_limit()+469<-ksmarfg()+574<-kghgex()+1376<-kghfnd()+361<-kghalo()+4422<-kghgex()+414<-kghfnd()+361<-kghalo()+4422<-kghgex()+414<-kghfnd()+361<-kghalo()+4422<-kghgex()+414<-kghalf()+1003<-klmalf()+103
<-kllcqas()+194<-kcblasm()+108<-kxhfNewBuffer()+607<-qerhjSplitBuild()+632
----- End of Abridged Call Stack Trace -----
=======================================
PRIVATE MEMORY SUMMARY FOR THIS PROCESS
---------------------------------------
******************************************************
PRIVATE HEAP SUMMARY DUMP
781 MB total:
   781 MB commented, 646 KB permanent
   208 KB free (0 KB in empty extents),
     779 MB,   2 heaps:   "koh-kghu call  "            57 KB free held
------------------------------------------------------
Summary of subheaps at depth 1
779 MB total:
   778 MB commented, 110 KB permanent
    63 KB free (0 KB in empty extents),
     667 MB, 42786 chunks:  "pmuccst: adt/record       "
      83 MB, 5333 chunks:  "pl/sql vc2                "

Actually, this is the end of the tracefile. It seems that the pga limit dump (the text in between “Process may have gone over pga_aggregate_limit” to the private memory summary heap dumps) occurs several times before an actual ORA-4036 is triggered. In my private test instance, where I am obviously the only user process doing something, I get a pga limit dump approximately 20 times before the ORA-4036 is actually triggered:

sending 4036 interrupt
Incident 46599 created, dump file: /u01/app/oracle/diag/rdbms/v12102/v12102/incident/incdir_46599/v12102_ora_4147_i46599.trc
ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT

Did you actually spot the oddity here?

Remember the PGA_AGGREGATE_LIMIT was set to 600M. Now look at the process’ PGA/Private heap summary dump above: it says 781M. Please mind the 781M is the PGA heap of a SINGLE process! When looking at the total PGA allocated for the entire instance, it’s even more:

SYS@v12102 AS SYSDBA> select value/power(1024,2) "MB" from v$pgastat where name = 'maximum PGA allocated';

	MB
----------
1041.16699

So…despite PGA_AGGREGATE_LIMIT set to 600M, according to the v$pgastat view, there’s 1041MB allocated for PGA. Please mind I haven’t looked into how accurate v$pgastat is, but I tend to believe this.

Summary.
I’ve seen PGA_AGGREGATE_TARGET being used as a calculation value for actual PGA usage of an instance. This is simply wrong. The actual amount of PGA memory allocated by the instance is highly depended on what is done, and can be less than PGA_AGGREGATE_TARGET, or more. Automatic PGA can control three per process memory area’s: the sort, hash and bitmap memory area’s. These are sized based on the setting of PGA_AGGREGATE_TARGET and the actual PGA memory usage instance wide. If more memory is needed for sort, hash or bitmap memory than is made available by the memory manager, excess memory needed is allocated in the temporary tablespace. Any other PGA memory allocation is always done, regardless of the setting of PGA_AGGREGATE_TARGET.

Starting with Oracle 12, it seems the actual PGA allocation now can actually be limited with the new parameter PGA_AGGREGATE_LIMIT. However, during some simple testing it shows that actually more memory is allocated than set with PGA_AGGREGATE_LIMIT as limit. I haven’t tested it in more situations, this post is meant to grow awareness that the actual limit as set by PGA_AGGREGATE_LIMIT might not be that hard as you would expect.

Please mind, PGA_AGGREGATE_LIMIT seems to truly limit PGA usage instance wide, not limit the PGA heap per process, as event 10251 (PGA usage limiting way for Oracle 11.2) does. However, once again: PGA_AGGREGATE_LIMIT seems to try to be smart and actually does not limit at the exact size set, but beyond that.

The next post will introduce a way to limit PGA usage in Oracle 11.2. Stay tuned!

I guess everybody who is working with Oracle databases and has been involved with Oracle Exadata in any way knows about smartscans. It is the smartscan who makes the magic happen of full segment scans with sometimes enormously reduced scan times. The Oracle database does smartscans which something that is referred to as ‘offloading’. This is all general known information.

But how does that work? I assume more people are like me, and are anxious to understand how that exactly works. But the information on smartscans is extremely scarce. Of course there is the Oracle public material, which looks technical, but is little/nothing more than marketing. On My Oracle Support, I can’t find anything on the inner working. Even in the ‘Expert Oracle Exadata’ book (which I still regard as the best source of Exadata related information) there is no material on the mechanics of smartscans.

I’ve written a couple of articles on smartscans, of which this article already lays some groundwork, it describes the phases I could see with the available information at that time: oss_open, oss_ioctl followed by oss_wait and oss_cread followed by oss_wait. This is actually a summary of a smartscan, but a very brief one. In this article I described that a smartscan can only happen with a full segment scan (kdstf* functions, Oracle’s fast full scan routines) and if it chooses to use direct path (kcbld* functions, direct path loader) access, which is actually mandatory to get smart scans.

The following investigation is done on an Exadata X2-2 Quarter rack, with Image version: 12.1.1.1.0.131219, and database version 12.1.0.1.3.

In order to get more understand on smartscans, we can use Oracle’s new debugging syntax. The part we are going to look at is called ‘KXD’:

SYS@db12c2 AS SYSDBA> oradebug doc component kxd

  KXD			       Exadata specific Kernel modules (kxd)
    KXDAM		       Exadata Disk Auto Manage (kxdam)
    KCFIS		       Exadata Predicate Push (kcfis)
    NSMTIO		       Trace Non Smart I/O (nsmtio)
    KXDBIO		       Exadata Block level Intelligent Operations (kxdbio)
    KXDRS		       Exadata Resilvering Layer (kxdrs)
    KXDOFL		       Exadata Offload (kxdofl)
    KXDMISC		       Exadata Misc (kxdmisc)

In order to get the tracing of the Exadata (database-) kernel modules, along with regular sql tracing with waits (to understand when something is happening, use the following events:

FRITS@db12c2 > alter session set events 'trace[kxd.*]';

Session altered.

FRITS@db12c2 > alter session set events 'sql_trace level 8';

Session altered.

Now execute a SQL that does smartscans. I’ve made sure table ‘T’ is big enough to invoke a direct path full table scan:

FRITS@db12c2 > select count(*) from t;

  COUNT(*)
----------
   1000000

Now take a peek in the trace file! The first part is normal looking:

PARSING IN CURSOR #139755290955216 len=22 dep=0 uid=201 oct=3 lid=201 tim=1593707470857 hv=2763161912 ad='224d632b8' sqlid='cyzznbykb509s'
select count(*) from t
END OF STMT
PARSE #139755290955216:c=2000,e=2417,p=0,cr=1,cu=0,mis=1,r=0,dep=0,og=1,plh=2966233522,tim=1593707470856
EXEC #139755290955216:c=0,e=25,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=2966233522,tim=1593707470920
WAIT #139755290955216: nam='SQL*Net message to client' ela= 2 driver id=1650815232 #bytes=1 p3=0 obj#=14 tim=1593707470968
WAIT #139755290955216: nam='enq: KO - fast object checkpoint' ela= 215 name|mode=1263468550 2=131242 0=2 obj#=14 tim=1593707471374
WAIT #139755290955216: nam='reliable message' ela= 1035 channel context=10126085744 channel handle=10164799536 broadcast message=10179010104 obj#=14 tim=1593707472530
WAIT #139755290955216: nam='enq: KO - fast object checkpoint' ela= 108 name|mode=1263468550 2=131242 0=1 obj#=14 tim=1593707472684
WAIT #139755290955216: nam='enq: KO - fast object checkpoint' ela= 101 name|mode=1263468545 2=131242 0=2 obj#=14 tim=1593707472829

We see the parsing of the simple select statement, and the execution, which yields some waits which are always there (the sqlnet message wait), and then some ‘enq: KO – fast object checkpoint’ waits, indicating a checkpoint, which is a sign of a direct path read.

The next part is interesting, because this is the smartscan-specific tracing:

Caching: Global context initialized 0x7f1b50ca6d20
kcfis_alloc_so 0x24d8994d8
In kcfis initialize: new init: app_state: 0x7f1b50ca67b0 app_type: 1
kcfis rcv update : op: 7 val: 1 so_numses 1 ovhdmem 0 mdmem 0 bufmem 0
kcfis_reinitialize: initializing queues
Set work des: global_ctx: 0x7f1b50ca6d20 app_state: 0x7f1b50ca67b0, mmwds: 0x22446a1a8
Automem enabled: app_state: 0x7f1b50ca67b0, mmwds: 0x22446a1a8
No match found for mmwds. Allocated wds 0x7f1b50ca6768, mmwds 0x22446a1a8
Cache version is 1 start cache version is 1
kcfis rcv update : op: 1 val: 1496 so_numses 1 ovhdmem 0 mdmem 1496 bufmem 0
kcfis rcv update : op: 1 val: 68 so_numses 1 ovhdmem 0 mdmem 1564 bufmem 0
oss_state->oss_context is 0x109d2db0
kcfis rcv update : op: 3 val: 69656 so_numses 1 ovhdmem 69656 mdmem 1564 bufmem 0
kcfis_initialize done

What we see here is the kcfis layer initialising memory. It’s interesting to see where the allocations are done. In general, on this system, the allocations in the 0x7f1bxxxxxxxx are in the PGA heap, kcfis_alloc_so/0x24d8994d8 in the SGA, and the mmwds/0x22446a1a8 is in the SGA too.
This information can be obtained by dumping heaps. Dumping the heaps at level 7 will show sga, session, pga, call and uga heaps. Another way to get insight into a memory locations is using Tanel Poder’s fcha script (Find Chunk Address). Please mind that if you are reading the kxd trace file and want to look up the addresses in a second window in a sqlplus / as sysdba session, this will allow you to see the SGA chunks, but probably not the PGA chunks, because these are private to the traced session.

Let’s get on to the next chunk of trace lines:

In kcfis read: app_state: 0x7f1b50ca67b0 being_ret=(nil)
kcfis_translate: source: 1
kcfis_get_new_request: obtained new piece to translate:fob: 0x24901a2a0 startblk: 1188499 blkcnt: 13 rdba: 55714451 Fno: 13 Bno: 1188499 
kcfis_get_translation:before: kcfis_req: fob: 0x24901a2a0 startblk: 1188499 blkcnt: 13 rdba: 55714451 Fno: 13 Bno: 1188499 reqid=1 cver=1 source=1
OSSIPC:SKGXP:[109be210.0]{0}: (25797 <- 13487)SKGXPDOAINVALCON: connection 0x109c8c30 admno 0x5ac93f1d scoono 0x321bdad5 acconn 0x63cd0678 getting closed. inactive: 0
OSSIPC:SKGXP:[109be210.9]{obj}: SKGXPCNH: 0x109c8390 SKGXPCON_OPEN (2) sconno 321bdad4 accono 1206dea3 admno 75f4dfcd ospid 13487 ANT
OSSIPC:SKGXP:[109be210.10]{obj}:   cookie [04030201010100001f2f65030b000000:..........e.....(16)]
OSSIPC:SKGXP:[109be210.11]{obj}:   Remote admin port
OSSIPC:SKGXP:[109be210.12]{obj}:        SSKGXPT 0x109c83d0 flags 0x2 { WRITE } sockno 12 IP 192.168.12.8 RDS 22774 lerr 0
OSSIPC:SKGXP:[109be210.13]{obj}:   Remote data port
OSSIPC:SKGXP:[109be210.14]{obj}:        SSKGXPT 0x109c84a0 flags 0x2 { WRITE } sockno 12 IP 192.168.12.8 RDS 51931 lerr 0
OSSIPC:SKGXP:[109be210.15]{obj}:   next seqno 32768 last ack 32763 credits 2 total credits 2 ertt 16 resends on con 0
OSSIPC:SKGXP:[109be210.16]{obj}: SKGXPCNH: 0x109c8c30 SKGXPCON_CLOSED (1) sconno 321bdad5 accono 63cd0678 admno 7b54ac1c ospid 13487 ANT
OSSIPC:SKGXP:[109be210.17]{obj}:   cookie [04030201010100001f2f65030b000000:..........e.....(16)]
OSSIPC:SKGXP:[109be210.18]{obj}:   Remote admin port
OSSIPC:SKGXP:[109be210.19]{obj}:        SSKGXPT 0x109c8c70 flags 0x2 { WRITE } sockno 12 IP 192.168.12.8 RDS 29832 lerr 0
OSSIPC:SKGXP:[109be210.20]{obj}:   Remote data port
OSSIPC:SKGXP:[109be210.21]{obj}:        SSKGXPT 0x109c8d40 flags 0x2 { WRITE } sockno 12 IP 192.168.12.8 RDS 65116 lerr 0
OSSIPC:SKGXP:[109be210.22]{obj}:   next seqno 32765 last ack 32763 credits 2 total credits 2 ertt 16 resends on con 0
kcfis_get_translation:after: Translation (disk,off,len of (0x24901a2a0, 1188499, 13) to (o/192.168.12.8/DATA_CD_06_enkcel04, 513723752448, 106496) Mirr_num: 0 reqid=1 cver=1 source=1
kcfis_get_disk_for_translation: Appliance 192.168.12.8/ does not exist
kcfis rcv update : op: 5 val: 8192 so_numses 1 ovhdmem 69656 mdmem 1564 bufmem 8192
kcfis_open_appliance: 
throttle: initialized for appliance 0x7f1b5078b3a8
kcfis_find_appliance_fd: appliance fd not found. appliance 192.168.12.8/ Cached appliance fd count 0
kcfis_open_appliance_fd: 0x7f1b5078b3a8
WAIT #139755290955216: nam='cell smart table scan' ela= 120 cellhash#=3249924569 p2=0 p3=0 obj#=61458 tim=1593707489569
Predicate device intelligent IO opened. fd 5
kcfis_init_appliance_fd: appliance fd 0x7f1b50da2b88 initialized for appliance 0x7f1b5078b3a8
kcfis_add_disk: Adding disk 0x7f1b50c953f0 name = o/192.168.12.8/DATA_CD_06_enkcel04 under appliance = 0x7f1b5078b3a8
initialize disk for disk o/192.168.12.8/DATA_CD_06_enkcel04
kcfis_initialize_disk_fd: Disk initialized. appliance: 192.168.12.8/ disk: o/192.168.12.8/DATA_CD_06_enkcel04 fd: 4 disknumber: 1 incarnation: 6 prev_disknumber: 0 num_inits: 1 init_cache_ver: 1
Translated fields: disk = 0x7f1b50c953f0, blkno = 1188499, numblks = 13, disk_handle = 4, diskoffs = 513723752448, len = 106496, path_asmname = o/192.168.12.8/DATA_CD_06_enkcel04, disk_num = 1, req_element = 0x7f1b50d0a000 reqid=1 cver=1 source=1
Default: calc numbufs mem 1048576
Final: calc numbufs mem 1048576 buflen 1048576
Num buffers: 1 buf per appliance: 1 num active appliance: 1
Appliance 0x7f1b5078b3a8 active. Active count 1

There’s a lot to see here. The next step in doing a smartscan is the translation of the data dictionary information on the segment to be smartscanned to cell server and grid disk extents. The data dictionary information is shown in line 3; ‘kcfis_get_new_request’, in line 4; ‘kcfis_get_translation:before’ the session tries to translate the data dictionary information to cell server and grid disk. Because the kcfis context is just initialised, there is no cell related information yet. For that reason, the information is requested from the cell server (the OSSIPC:SKGXP lines). Please mind this request will send all disk related information to the kcfis context of the process. With this information, the process can make the translation, as can be seen in line 20: ‘kcfis_get_translation:after’, it shows where the extent is located in the well known exadata notation: ‘o/cell ip/grid disk name’, together with offset and chunk length. Next the disk needs to be initialised (a disk is a combination of cell server or ‘appliance’ and grid disk), and given a hash value, as is indicated by ‘kcfis_get_disk_for_translation’ in line 21. However, in order for a disk to be initialised, the cell server or appliance must be initialised too. That is what the process is indicating in line 21: Appliance 192.168.12.8/ does not exist. The appliance (cell server) is initialised/opened, which is what is shown in lines 23-29. The wait here is initialising a connection with the cell server, waiting for an acknowledgement. Now the appliance is initialised, the disk is initialised, as indicated in lines 30-32. Next line 33 shows the translation which was started earlier new is finally been done. Line 34-37 show something about the buffering which seems to be arranged per appliance, and is 1MB.

Now that the process has initialised the appliance and the disk, the next translation is done. The translations are done per extent of the segment, and the disk is depended on the placing of the extent. Please mind the maximum size of the allocation is depended on the AU (allocation unit) size of ASM, which is set to 4MB by default with Exadata. If a non-initialised appliance is encountered, it is initialised and opened, and if a non-initialised disk is encountered, this is initialised.

The translation looks like this if everything is initialised:

kcfis_get_new_request: obtained new piece to translate:fob: 0x24901a2a0 startblk: 1188513 blkcnt: 15 rdba: 55714465 Fno: 13 Bno: 1188513 
kcfis_get_translation:before: kcfis_req: fob: 0x24901a2a0 startblk: 1188513 blkcnt: 15 rdba: 55714465 Fno: 13 Bno: 1188513 reqid=2 cver=1 source=1
kcfis_get_translation:after: Translation (disk,off,len of (0x24901a2a0, 1188513, 15) to (o/192.168.12.8/DATA_CD_06_enkcel04, 513723867136, 122880) Mirr_num: 0 reqid=2 cver=1 source=1
Translated fields: disk = 0x7f1b50c953f0, blkno = 1188513, numblks = 15, disk_handle = 4, diskoffs = 513723867136, len = 122880, path_asmname = o/192.168.12.8/DATA_CD_06_enkcel04, disk_num = 1, req_element = 0x7f1b50d0a220 reqid=2 cver=1 source=1

Another thing which is important to notice is the ‘reqid’, which obviously means ‘request id’. This process is repeated until the complete segment is translated into requests.

Once the requests are translated, the next step in the smartscan is to send (‘push’) the requests to the appliances. This apparently is called a ‘payload map’.

kcfis_push: num-appliances 4. payload_size 0x7f1b50da3130 ioctl_issued 0x7f1b50da3108 results 0x7f1b50da3068 payload_type 0x7f1b50da3160
kcfis_create_maps_payload. appliance 0x7f1b5078b3a8 num_disks 12
disk=0x7f1b50c953f0 state=1
trans_req_element = 0x7f1b50d0a000
Pushing request : disknumber = 1, offset = 513723752448, len = 106496 rdba: 55714451 version 0 reqid=1 cver=1
disk=0x7f1b507908e0 state=1

The first line is the start of the pushing of the payload maps. The next line shows a specific appliance being chosen. What is shown next is a line showing ‘disk’ and the hash value of the disk. At the start of a maps push, the disk lines are followed by two lines saying ‘trans_req_element’ and ‘Pushing request’. These two lines probably are some kind of state object for the request, and the actual pushing of the request. Here we see the request id back which we saw in the translation phase.

This is repeated, until some of the disk lines are starting to get followed immediately by another disk line:

disk=0x7f1b50cfdf80 state=1
trans_req_element = 0x7f1b50d12c40
Pushing request : disknumber = 7, offset = 513817985024, len = 1032192 rdba: 55716738 version 0 reqid=67 cver=1
disk=0x7f1b50cfc930 state=1
disk=0x7f1b50cc8118 state=1
trans_req_element = 0x7f1b50d2ffa0
Pushing request : disknumber = 9, offset = 513890353152, len = 4161536 rdba: 55729668 version 0 reqid=174 cver=1
disk=0x7f1b50cc7ae0 state=1
disk=0x7f1b50cc5ab0 state=1
disk=0x7f1b50cc3a80 state=1

Further down in the trace file, the trans_req_element and Pushing request lines are becoming scarce:

disk=0x7f1b50cc8118 state=1
disk=0x7f1b50cc7ae0 state=1
disk=0x7f1b50cc5ab0 state=1
disk=0x7f1b50cc3a80 state=1
disk=0x7f1b50cc30a0 state=1
disk=0x7f1b50c953f0 state=1
trans_req_element = 0x7f1b50d31920
Pushing request : disknumber = 1, offset = 498446925824, len = 4161536 rdba: 41208836 version 0 reqid=186 cver=1
disk=0x7f1b507908e0 state=1
disk=0x7f1b507903f0 state=1
disk=0x7f1b50d04fc0 state=1
disk=0x7f1b50cfe960 state=1
disk=0x7f1b50cfdf80 state=1
disk=0x7f1b50cfc930 state=1

Inspection of these lines show that the process is going through a strict sequence of disks of that appliance, and picks up one request per disk which (obviously) belongs to that disk. If the requests are not evenly divided between the disks, some disks will have all the requests already pushed to that disk, while other disks still need additional requests. In that case, the disk which already have their requests pushed will not get a request, so no trans_req_element/Pushing request combination. The process goes through this until all the requests for that appliance are pushed.

After the push of all the requests for that appliance, the following is happening:

kcfis_create_maps_payload. alloc_len 4088 num maps 55
throttle: mappayload: maps_to_push 7 iosize_being_pushed 150953984
kcfis_metadata_payload_len: app_state 0x7f1b50ca67b0 appliance 0x7f1b5078b3a8 payload_len 5968 payload_hdr_len 96 sessiondata_payload_len 144
metadata_payload_len 1536 fmetadata_payload_len 100 maps_len 3992 exthdr_len 8 planpayload_len 48 oflgrppayload_plen 40
kcfis_create_metadata_payload. appliance 0x7f1b5078b3a8 payload 0x7f1b50da3190 payload_memlen 5968 maps 0x7f1b50d07fe0 mapslen 3992
kcfis_create_metadata_payload: pushing sessiondata: appliance 0x7f1b5078b3a8
kcfis_create_metadata_payload: pushing capability payload: appliance 0x7f1b5078b3a8
kcfis_create_metadata_payload: dop: 1kcfis_create_metadata_payload: pushing metadata: appliance 0x7f1b5078b3a8
kcfis_create_metadata_payload: pushing fast metadata: appliance 0x7f1b5078b3a8
kcfis_push: pushing metadata to appliance 0x7f1b5078b3a8. metadata 0x7f1b50da3190
kcfis_issue_ioctl: payload_type 1
WAIT #139755290955216: nam='cell smart table scan' ela= 178 cellhash#=3249924569 p2=0 p3=0 obj#=61458 tim=1593707571673
Ioctl completed. Payload type 1
Ioctl quarantine response 1 for appliance 0x7f1b5078b3a8
appliance 0x7f1b5078b3a8 : cellsrv pid: 13487: predicate: /box/predicate735745

The first line shows 55 maps have been pushed to the appliance. The other lines are various memory locations which are needed for gathering the reads which will be send back by the appliances. Further things which seem important are line 11, which issues a ioctl (IO control) request to the appliance, and waits for acknowledgement. Mind the wait is always ‘cell smart table scan’. Line 15 shows this request gets a predicate, which is ‘/box/predicate735745’.

This is repeated for every appliance.

Then the next thing happens:

kcfis_create_maps_payload. appliance 0x7f1b5078b3a8 num_disks 12
throttle: allowing map push: appliance 0x7f1b5078b3a8, maps_to_push 7
disk=0x7f1b50c953f0 state=1
disk=0x7f1b507908e0 state=1
disk=0x7f1b507903f0 state=1
disk=0x7f1b50d04fc0 state=1
disk=0x7f1b50cfe960 state=1
disk=0x7f1b50cfdf80 state=1
disk=0x7f1b50cfc930 state=1
disk=0x7f1b50cc8118 state=1
disk=0x7f1b50cc7ae0 state=1
disk=0x7f1b50cc5ab0 state=1
disk=0x7f1b50cc3a80 state=1
disk=0x7f1b50cc30a0 state=1
kcfis_create_maps_payload. alloc_len 200 num maps 0

The maps payload push is done again for all the appliances, without any request being pushed. The last line confirms no maps/requests having been pushed: num maps 0. However, there is one line which hasn’t been there before: line 2 ‘throttle’, more specifically important in this line is ‘allowing map push’, the previous throttle during ‘kcfis_create_maps_payload’ had the remark ‘mappayload’.

This means that at this point the physical extents to be scanned on the appliances (cell servers) have been identified, translated to appliance, grid disk, offset and size, and the requests for these extents have been send to the appliances. The last snippet actually means that the appliance is notified to start preparing for sending results back.

After the appliances have been notified, memory is initialised again.

Default: calc numbufs mem 4194304
Final: calc numbufs mem 4194304 buflen 1048576
Alloc buffer: target_freebufs 4 allocated_freebufs 0
Get additional mem: app_state: 0x7f1b50ca67b0 kcfis wds 0x7f1b50ca6768
Starting work area: app_state: 0x7f1b50ca67b0 wds 0x7f1b50ca6768, mmwds: 0x22446a1a8
Started work area: wds: 0x7f1b50ca6768 mmwds 0x22446a1a8
Get additional mem for pga_aggregate_target: max 4195552 min 1048888, wds 0x7f1b50ca6768 mmwds 0x22446a1a8
cur size 0
Change req: expected size 4196352 cur size 0 max_mem (KB) 4098
Max memory allocation ok: max 4195552, expected 4196352, cur 0
Memlen allowed 4195552 io_buflen 1048576 chunk_len 1048888
kcfis_alloc_readmem_chunk: sz=1048888
incr_kcfis_mem_wds: wds 0x7f1b50ca6768 mmwds 0x22446a1a8 size 1053007 refcnt 1 memsize 1053007
kcfis rcv update : op: 5 val: 1053007 so_numses 1 ovhdmem 139312 mdmem 1564 bufmem 1085775
kcfis_alloc_readmem_chunk: sz=1048888
incr_kcfis_mem_wds: wds 0x7f1b50ca6768 mmwds 0x22446a1a8 size 1053007 refcnt 1 memsize 2106014
kcfis rcv update : op: 5 val: 1053007 so_numses 1 ovhdmem 139312 mdmem 1564 bufmem 2138782
kcfis_alloc_readmem_chunk: sz=1048888
incr_kcfis_mem_wds: wds 0x7f1b50ca6768 mmwds 0x22446a1a8 size 1053007 refcnt 1 memsize 3159021
kcfis rcv update : op: 5 val: 1053007 so_numses 1 ovhdmem 139312 mdmem 1564 bufmem 3191789
kcfis_alloc_readmem_chunk: sz=1048888
incr_kcfis_mem_wds: wds 0x7f1b50ca6768 mmwds 0x22446a1a8 size 1053007 refcnt 1 memsize 4212028
kcfis rcv update : op: 5 val: 1053007 so_numses 1 ovhdmem 139312 mdmem 1564 bufmem 4244796
Set workarea size: app_state: 0x7f1b50ca67b0 kcfis wds 0x7f1b50ca6768 mmwds: 0x22446a1a8, global_ctx: 0x7f1b50ca6d20, size: 4212028
Calling oss_cread: appliance 0x7f1b5078b3a8 app_buffer: 0x7f1b50659000 databuf: 0x7f1b50559000 buflen: 1048576 (posted to 192.168.12.8/)
appliance 0x7f1b5078b3a8 total creads 1 new creads 1 read seqno 0 pending reads 0
Calling oss_cread: appliance 0x7f1b50c94e78 app_buffer: 0x7f1b50549000 databuf: 0x7f1b50449000 buflen: 1048576 (posted to 192.168.12.11/)
appliance 0x7f1b50c94e78 total creads 1 new creads 1 read seqno 0 pending reads 0
Calling oss_cread: appliance 0x7f1b50d059a0 app_buffer: 0x7f1b50439000 databuf: 0x7f1b50339000 buflen: 1048576 (posted to 192.168.12.9/)
appliance 0x7f1b50d059a0 total creads 1 new creads 1 read seqno 0 pending reads 0
Calling oss_cread: appliance 0x7f1b50d02948 app_buffer: 0x7f1b50a7a000 databuf: 0x7f1b5097a000 buflen: 1048576 (posted to 192.168.12.10/)
appliance 0x7f1b50d02948 total creads 1 new creads 1 read seqno 0 pending reads 0
kcfis wait: buf: 0x7f1b50549000 app_state: 0x7f1b50ca67b0 err: (0) Success
kcfis wait: buf: 0x7f1b50659000 app_state: 0x7f1b50ca67b0 err: (0) Success
WAIT #139755290955216: nam='cell smart table scan' ela= 59 cellhash#=822451848 p2=0 p3=0 obj#=61458 tim=1593707578411
kcfis_push: num-appliances 4. payload_size 0x7f1b50da3130 ioctl_issued 0x7f1b50da3108 results 0x7f1b50da3068 payload_type 0x7f1b50da3160

First memory areas are initialised (lines 1-24), then we see lines showing ‘Calling oss_cread’. The oss_cread call is the call to the appliances to start sending a resultset back. Please mind that despite the calls addressing specific extents, this is a smartscan, so resultsets are send back instead of Oracle blocks. Also, since this is exadata using the iDB/RDS protocol over infiniband, the appliances can use RDMA to send the results back, which means the cells can fill the memory in the server process’ memory directly.

After oss_cread being called, the ‘kcfis_create_maps_payload’ routine (shown in the snippet above the last snippet) being executed to every appliance apparently to indicate all the disks being enabled, possibly trying to throttle activity, and/or to indicate requests will be called from this session. This seems to be repeated for every roundtrip during the entire smartscan for all the appliances that are still needed.

Whenever a result (result set) is ready to be processed, the following sequence happens:

kcfis reaped i/o: app_state: 0x7f1b50ca67b0
kcfis reaped i/o: buf: 0x7f1b50549000 err: (0) Success
Returning non-pt payload
appliance 0x7f1b50c94e78 read seqno 1 pending reads 0
appliance 0x7f1b50c94e78 total creads 0 re-adjusted: read seqno 1 pending reads 0
throttle: received: maps_to_push 7 total 14
kcfis_process_completed_buffer: App Buffer 0x7f1b50549000, databuf: 0x7f1b50449000, nelem: 1
Dump of memory from 0x00007F1B5044B000 to 0x00007F1B5044B040
7F1B5044B000 0000A23C 03344401 5BB31709 0402099A  [<....D4....[....]
7F1B5044B010 00986F3F 001E0001 0000000F 0000000F  [?o..............]
7F1B5044B020 0000000F 00000000 00000000 00000000  [................]
7F1B5044B030 00000000 00000000 00000000 00000000  [................]
kcfis_process_completed_buffer: 0 elem: 0x7f1b50449080
kcfis_validate_translation: request 0x7f1b50d0aee0
Req completed is : err = 0, disknumber = 1, off = 562871410688, len = 122880 data_len = 3992 bufoff = 8192 version = 0 reqid=8 cver=1 block_id=53756929
 flags = 4
kcfis_oss_block_verify: bp: 0x7f1b5044b000 afn 12 rdba 53756929 dlen: 3992 blksz: 8192 nblks: 0
kcfis_oss_block_verify: corrupt checkcb: rdba: 53756929 good: 1
Request 0x7f1b50d0aee0 done
Returning non-pt payload
kcfis_get_next_data: elem = 0, err = 0, disknumber = 1, off = 562871410688, len = 122880 data_len = 3992 bufoff = 8192 version = 0 reqid=8 cver=1
memptr (nil) len 0, blockid = 53756929
kcfis_get_next_data: dptr: 0x7f1b5044b000 len: 3992 err_code: 0
In kcfis read: app_state: 0x7f1b50ca67b0 being_ret=0x7f1b50549000
Returning non-pt payload
kcfis_read: OSS I/O: freeing buffer=0x7f1b50549000
WAIT #139755290955216: nam='cell smart table scan' ela= 9 cellhash#=822451848 p2=0 p3=0 obj#=61458 tim=1593707579132

This shows the reap of result set returned by an appliance. Line 5-6 show appliance specific information. Line 7 is showing important information; ‘nelem’ shows the amount of extents (called ‘elements’ in this context) for which the result or results are returned. Of course ‘nelem’ means ‘number of elements’. In this case it’s 1 (resultset from a specific element/extent). Line 15 shows the actual extent from which the result set came back, because the reqid is exposed, the reqid was defined during the translation phase. This snippet ends with a WAIT line (again: all the waits are ‘cell smart table scan’). I consider this a cyclic process: first the ‘kcfis_create_maps_payload’, then calling oss_cread for one or multiple cells, then a wait, or the above processing of results for one or multiple appliances, and a wait.

I’ve created this snippet to be as simple as possible, in real life result sets of multiple appliances could be reaped (in my case I had to remove a second result). The processing of the result set is done in a few stages, so a resultset is not processed per appliance, but the processing stages are done for all the result sets of all the appliances.

Also this example shows only one request in the reaped result. There can be multiple requests (reqid’s/extents) returned.

kcfis reaped i/o: app_state: 0x7f1b50ca67b0
kcfis reaped i/o: buf: 0x7f1b50439000 err: (0) Success
Returning non-pt payload
appliance 0x7f1b50d059a0 read seqno 1 pending reads 1
appliance 0x7f1b50d059a0 total creads 0 re-adjusted: read seqno 1 pending reads 1
throttle: received: maps_to_push 7 total 14
kcfis_process_completed_buffer: App Buffer 0x7f1b50439000, databuf: 0x7f1b50339000, nelem: 2
Dump of memory from 0x00007F1B5033B000 to 0x00007F1B5033B040
7F1B5033B000 0000A23C 02748C02 5BB31713 0402099A  [<.....t....[....]
7F1B5033B010 00988672 001E0001 0000007E 0000007E  [r.......~...~...]
7F1B5033B020 0000007E 00000000 00000000 00000000  [~...............]
7F1B5033B030 00000000 00000000 00000000 00000000  [................]
kcfis_process_completed_buffer: 0 elem: 0x7f1b50339080
kcfis_validate_translation: request 0x7f1b50d0b980
Req completed is : err = 0, disknumber = 1, off = 465244798976, len = 1032192 data_len = 32408 bufoff = 8192 version = 0 reqid=13 cver=1 block_id=41192450
 flags = 0
kcfis_oss_block_verify: bp: 0x7f1b5033b000 afn 9 rdba 41192450 dlen: 32408 blksz: 8192 nblks: 3
kcfis_oss_block_verify: corrupt checkcb: rdba: 41192450 good: 1
Request 0x7f1b50d0b980 done
kcfis_process_completed_buffer: 1 elem: 0x7f1b503390c8
kcfis_validate_translation: request 0x7f1b50d114e0
Req completed is : err = 0, disknumber = 6, off = 498161696768, len = 1032192 data_len = 32408 bufoff = 40600 version = 0 reqid=56 cver=1 block_id=53758466
 flags = 0
kcfis_oss_block_verify: bp: 0x7f1b50342e98 afn 12 rdba 53758466 dlen: 32408 blksz: 8192 nblks: 3
kcfis_oss_block_verify: corrupt checkcb: rdba: 53758466 good: 1
Request 0x7f1b50d114e0 done
Returning non-pt payload
kcfis_get_next_data: elem = 0, err = 0, disknumber = 1, off = 465244798976, len = 1032192 data_len = 32408 bufoff = 8192 version = 0 reqid=13 cver=1
memptr (nil) len 0, blockid = 41192450
kcfis_get_next_data: elem = 1, err = 0, disknumber = 6, off = 498161696768, len = 1032192 data_len = 32408 bufoff = 40600 version = 0 reqid=56 cver=1
memptr 0x7f1b5033b000 len 32408, blockid = 53758466
kcfis_get_next_data: dptr: 0x7f1b5033b000 len: 64816 err_code: 0
In kcfis read: app_state: 0x7f1b50ca67b0 being_ret=0x7f1b50439000
Returning non-pt payload
kcfis_read: OSS I/O: freeing buffer=0x7f1b50439000
WAIT #139755290955216: nam='cell smart table scan' ela= 10 cellhash#=1034800054 p2=0 p3=0 obj#=61458 tim=1593707616790

(example with ‘nelem’=2)

kcfis reaped i/o: app_state: 0x7f1b50ca67b0
kcfis reaped i/o: buf: 0x7f1b50549000 err: (0) Success
Returning non-pt payload
appliance 0x7f1b50c94e78 read seqno 1 pending reads 0
appliance 0x7f1b50c94e78 total creads 0 re-adjusted: read seqno 1 pending reads 0
throttle: received: maps_to_push 7 total 14
kcfis_process_completed_buffer: App Buffer 0x7f1b50549000, databuf: 0x7f1b50449000, nelem: 1
Dump of memory from 0x00007F1B5044B000 to 0x00007F1B5044B040
7F1B5044B000 0000A23C 03344401 5BB31709 0402099A  [<....D4....[....]
7F1B5044B010 00986F3F 001E0001 0000000F 0000000F  [?o..............]
7F1B5044B020 0000000F 00000000 00000000 00000000  [................]
7F1B5044B030 00000000 00000000 00000000 00000000  [................]
kcfis_process_completed_buffer: 0 elem: 0x7f1b50449080
kcfis_validate_translation: request 0x7f1b50d0aee0
Req completed is : err = 0, disknumber = 1, off = 562871410688, len = 122880 data_len = 3992 bufoff = 8192 version = 0 reqid=8 cver=1 block_id=53756929
 flags = 4
kcfis_oss_block_verify: bp: 0x7f1b5044b000 afn 12 rdba 53756929 dlen: 3992 blksz: 8192 nblks: 0
kcfis_oss_block_verify: corrupt checkcb: rdba: 53756929 good: 1
Request 0x7f1b50d0aee0 done
kcfis reaped i/o: buf: 0x7f1b50659000 err: (0) Success
Returning non-pt payload
appliance 0x7f1b5078b3a8 read seqno 1 pending reads 0
appliance 0x7f1b5078b3a8 total creads 0 re-adjusted: read seqno 1 pending reads 0
throttle: received: maps_to_push 7 total 14
kcfis_process_completed_buffer: App Buffer 0x7f1b50659000, databuf: 0x7f1b50559000, nelem: 1
Dump of memory from 0x00007F1B5055B000 to 0x00007F1B5055B040
7F1B5055B000 0000A23C 03522293 5BB316ED 0402099A  [<...."R....[....]
7F1B5055B010 009849CB 001E0001 0000000D 0000000D  [.I..............]
7F1B5055B020 0000000D 00000000 00000000 00000000  [................]
7F1B5055B030 00000000 00000000 00000000 00000000  [................]
kcfis_process_completed_buffer: 0 elem: 0x7f1b50559080
kcfis_validate_translation: request 0x7f1b50d0a000
Req completed is : err = 0, disknumber = 1, off = 513723752448, len = 106496 data_len = 3480 bufoff = 8192 version = 0 reqid=1 cver=1 block_id=55714451
 flags = 4
kcfis_oss_block_verify: bp: 0x7f1b5055b000 afn 13 rdba 55714451 dlen: 3480 blksz: 8192 nblks: 0
kcfis_oss_block_verify: corrupt checkcb: rdba: 55714451 good: 1
Request 0x7f1b50d0a000 done
Returning non-pt payload
kcfis_get_next_data: elem = 0, err = 0, disknumber = 1, off = 513723752448, len = 106496 data_len = 3480 bufoff = 8192 version = 0 reqid=1 cver=1
memptr (nil) len 0, blockid = 55714451
kcfis_get_next_data: dptr: 0x7f1b5055b000 len: 3480 err_code: 0
In kcfis read: app_state: 0x7f1b50ca67b0 being_ret=0x7f1b50659000
Returning non-pt payload
kcfis_read: OSS I/O: freeing buffer=0x7f1b50659000
Returning non-pt payload
kcfis_get_next_data: elem = 0, err = 0, disknumber = 1, off = 562871410688, len = 122880 data_len = 3992 bufoff = 8192 version = 0 reqid=8 cver=1
memptr (nil) len 0, blockid = 53756929
kcfis_get_next_data: dptr: 0x7f1b5044b000 len: 3992 err_code: 0
In kcfis read: app_state: 0x7f1b50ca67b0 being_ret=0x7f1b50549000
Returning non-pt payload
kcfis_read: OSS I/O: freeing buffer=0x7f1b50549000
WAIT #139755290955216: nam='cell smart table scan' ela= 9 cellhash#=822451848 p2=0 p3=0 obj#=61458 tim=1593707579132

(example of result sets returned of two appliances, for which the returned results are processed)

Also, when the extent to be scanned is larger, the result set is processed in multiple steps:

kcfis reaped i/o: app_state: 0x7f1b50ca67b0
kcfis reaped i/o: buf: 0x7f1b50659000 err: (0) Success
Returning non-pt payload
appliance 0x7f1b50c94e78 read seqno 2 pending reads 1
appliance 0x7f1b50c94e78 total creads 0 re-adjusted: read seqno 2 pending reads 1
throttle: received: maps_to_push 0 total 14
kcfis_process_completed_buffer: App Buffer 0x7f1b50659000, databuf: 0x7f1b50559000, nelem: 3
Dump of memory from 0x00007F1B5055B000 to 0x00007F1B5055B040
7F1B5055B000 0000A23C 03523080 5BB3178D 0402099A  [<....0R....[....]
7F1B5055B010 0098FBC9 001E0001 00000080 00000080  [................]
7F1B5055B020 00000080 00000000 00000000 00000000  [................]
7F1B5055B030 00000000 00000000 00000000 00000000  [................]
kcfis_process_completed_buffer: 0 elem: 0x7f1b50559080
kcfis_validate_translation: request 0x7f1b50d145c0
Req completed is : err = 0, disknumber = 6, off = 650650845184, len = 1048576 data_len = 32920 bufoff = 8192 version = 0 reqid=79 cver=1 block_id=55718016
 flags = 0
kcfis_validate_translation: REQ2: splitting from top
kcfis_do_kf_trans_and_queue_in_push1: discard: 0 fob: 0x24901a2a0 sblk: 1191936 nblk: 128 aubyteoffs 0 disk: o/192.168.12.11/DATA_CD_05_enkcel07 off: 650649796608 sz: 1048576 mnum: 0 res: 1 parent req: 0x7f1b50d145c0, req 0x7f1b50d114e0, preqid=79 reqid=79 cver=1
kcfis_do_kf_trans_and_queue_in_push4: discard: 0 fob: 0x24901a2a0 sblk: 1191936 nblk: 128 aubyteoffs 0 disk: o/192.168.12.11/DATA_CD_05_enkcel07 off: 650649796608 sz: 1048576 mnum: 0 res: 1 parent req: 0x7f1b50d145c0, req 0x7f1b50d114e0, preqid=79 reqid=79 cver=1
kcfis_validate_translation: REQ3: splitting from bottom
kcfis_do_kf_trans_and_queue_in_push1: discard: 0 fob: 0x24901a2a0 sblk: 1192192 nblk: 256 aubyteoffs 2097152 disk: o/192.168.12.11/DATA_CD_05_enkcel07 off: 650651893760 sz: 2097152 mnum: 0 res: 1 parent req: 0x7f1b50d145c0, req 0x7f1b50d0b980, preqid=79 reqid=79 cver=1
kcfis_do_kf_trans_and_queue_in_push4: discard: 0 fob: 0x24901a2a0 sblk: 1192192 nblk: 256 aubyteoffs 2097152 disk: o/192.168.12.11/DATA_CD_05_enkcel07 off: 650651893760 sz: 2097152 mnum: 0 res: 1 parent req: 0x7f1b50d145c0, req 0x7f1b50d0b980, preqid=79 reqid=79 cver=1
kcfis_oss_block_verify: bp: 0x7f1b5055b000 afn 13 rdba 55718016 dlen: 32920 blksz: 8192 nblks: 4
kcfis_oss_block_verify: corrupt checkcb: rdba: 55718016 good: 1
Request 0x7f1b50d145c0 done
kcfis_process_completed_buffer: 1 elem: 0x7f1b505590c8
kcfis_validate_translation: request 0x7f1b50d13d40
Req completed is : err = 0, disknumber = 4, off = 533539586048, len = 1048576 data_len = 32920 bufoff = 41112 version = 0 reqid=75 cver=1 block_id=41195392
 flags = 0
kcfis_validate_translation: REQ2: splitting from top
kcfis_do_kf_trans_and_queue_in_push1: discard: 0 fob: 0x24901a520 sblk: 3446272 nblk: 384 aubyteoffs 0 disk: o/192.168.12.11/DATA_CD_06_enkcel07 off: 533536440320 sz: 3145728 mnum: 0 res: 1 parent req: 0x7f1b50d13d40, req 0x7f1b50d145c0, preqid=75 reqid=75 cver=1
kcfis_do_kf_trans_and_queue_in_push4: discard: 0 fob: 0x24901a520 sblk: 3446272 nblk: 384 aubyteoffs 0 disk: o/192.168.12.11/DATA_CD_06_enkcel07 off: 533536440320 sz: 3145728 mnum: 0 res: 1 parent req: 0x7f1b50d13d40, req 0x7f1b50d145c0, preqid=75 reqid=75 cver=1
kcfis_oss_block_verify: bp: 0x7f1b50563098 afn 9 rdba 41195392 dlen: 32920 blksz: 8192 nblks: 4
kcfis_oss_block_verify: corrupt checkcb: rdba: 41195392 good: 1
Request 0x7f1b50d13d40 done
kcfis_process_completed_buffer: 2 elem: 0x7f1b50559110
kcfis_validate_translation: request 0x7f1b50d13f60
Req completed is : err = 0, disknumber = 5, off = 655073738752, len = 1048576 data_len = 32920 bufoff = 74032 version = 0 reqid=76 cver=1 block_id=59912064
 flags = 0
kcfis_validate_translation: REQ2: splitting from top
kcfis_do_kf_trans_and_queue_in_push1: discard: 0 fob: 0x24901a008 sblk: 1191428 nblk: 380 aubyteoffs 32768 disk: o/192.168.12.11/DATA_CD_07_enkcel07 off: 655070625792 sz: 3112960 mnum: 0 res: 1 parent req: 0x7f1b50d13f60, req 0x7f1b50d13d40, preqid=76 reqid=76 cver=1
kcfis_do_kf_trans_and_queue_in_push4: discard: 0 fob: 0x24901a008 sblk: 1191428 nblk: 380 aubyteoffs 32768 disk: o/192.168.12.11/DATA_CD_07_enkcel07 off: 655070625792 sz: 3112960 mnum: 0 res: 1 parent req: 0x7f1b50d13f60, req 0x7f1b50d13d40, preqid=76 reqid=76 cver=1
kcfis_oss_block_verify: bp: 0x7f1b5056b130 afn 14 rdba 59912064 dlen: 32920 blksz: 8192 nblks: 4
kcfis_oss_block_verify: corrupt checkcb: rdba: 59912064 good: 1
Request 0x7f1b50d13f60 done
Returning non-pt payload
kcfis_get_next_data: elem = 0, err = 0, disknumber = 6, off = 650650845184, len = 1048576 data_len = 32920 bufoff = 8192 version = 0 reqid=79 cver=1
memptr (nil) len 0, blockid = 55718016
kcfis_get_next_data: elem = 1, err = 0, disknumber = 4, off = 533539586048, len = 1048576 data_len = 32920 bufoff = 41112 version = 0 reqid=75 cver=1
memptr 0x7f1b5055b000 len 32920, blockid = 41195392
kcfis_get_next_data: elem = 2, err = 0, disknumber = 5, off = 655073738752, len = 1048576 data_len = 32920 bufoff = 74032 version = 0 reqid=76 cver=1
memptr 0x7f1b5055b000 len 65840, blockid = 59912064
kcfis_get_next_data: dptr: 0x7f1b5055b000 len: 98760 err_code: 0
In kcfis read: app_state: 0x7f1b50ca67b0 being_ret=0x7f1b50659000
Returning non-pt payload
kcfis_read: OSS I/O: freeing buffer=0x7f1b50659000
WAIT #139755290955216: nam='cell smart table scan' ela= 9 cellhash#=3249924569 p2=0 p3=0 obj#=61458 tim=1593707638819

(example with nelem=3, with bigger extents, for which the result is splitted)

Once the result is reaped for a certain appliance, a new oss_cread call must be done in order for another result to be pushed to the server process. Request (reqid or extents) are not sent back in order.

Once the requests are exhausted for a certain appliance, the appliance is excluded from the ‘kcfis_create_maps_payload’ procedure.

Once all smartscan is complete, the memory areas are cleaned up, and the sessions are closed. Apparently a session or some session state information is kept per grid disk, which are all closed (closing of one appliance is shown in this snippet):

kcfis_read DONE - ret NULL
kcfis_finalize: app_state 0x7f1b50ca67b0 permflags 0x9000 clnflags 0x1f
Appliance 0x7f1b5078b3a8 in-active. Active count 3
Appliance 0x7f1b50c94e78 in-active. Active count 2
Appliance 0x7f1b50d059a0 in-active. Active count 1
Appliance 0x7f1b50d02948 in-active. Active count 0
Caching: session at the end of scan 1
Work area gc: 0x7f1b50ca6768, app_state: 0x7f1b50ca67b0
Freeing read buffer chunk (GC) 0x7f1b50448eb0. count 4
decr_kcfis_mem_wds: wds 0x7f1b50ca6768 mmwds 0x22446a1a8 size 1053007 refcnt 1 memsize 3159021
kcfis rcv update : op: 6 val: 1053007 so_numses 1 ovhdmem 139312 mdmem 1564 bufmem 3191789
Freeing read buffer chunk (GC) 0x7f1b50338eb0. count 3
decr_kcfis_mem_wds: wds 0x7f1b50ca6768 mmwds 0x22446a1a8 size 1053007 refcnt 1 memsize 2106014
kcfis rcv update : op: 6 val: 1053007 so_numses 1 ovhdmem 139312 mdmem 1564 bufmem 2138782
Freeing read buffer chunk (GC) 0x7f1b50558eb0. count 2
decr_kcfis_mem_wds: wds 0x7f1b50ca6768 mmwds 0x22446a1a8 size 1053007 refcnt 1 memsize 1053007
kcfis rcv update : op: 6 val: 1053007 so_numses 1 ovhdmem 139312 mdmem 1564 bufmem 1085775
Freeing read buffer chunk (GC) 0x7f1b50979eb0. count 1
decr_kcfis_mem_wds: wds 0x7f1b50ca6768 mmwds 0x22446a1a8 size 1053007 refcnt 1 memsize 0
kcfis rcv update : op: 6 val: 1053007 so_numses 1 ovhdmem 139312 mdmem 1564 bufmem 32768
Freeing memory to top level heap: num_freed 4
Set workarea size: app_state: 0x7f1b50ca67b0 kcfis wds 0x7f1b50ca6768 mmwds: 0x22446a1a8, global_ctx: 0x7f1b50ca6d20, size: 0
Work area cleanup start: global_ctx: 0x7f1b50ca6d20
Close work area: kcfis wds 0x7f1b50ca6768, mmwds: 0x22446a1a8 app_state: 0x7f1b50ca67b0 refcnt: 0
Close work area: kcfis wds 0x7f1b50ca6768, mmwds: 0x22446a1a8 refcnt: 0
Closed work area: kcfis wds 0x7f1b50ca6768, mmwds: 0x22446a1a8
Deallocating kcfis wds: 0x7f1b50ca6768
Cleanup work area for app_state: 0x7f1b50ca67b0
kcfis_finalize_cached_sessions: global_ctx 0x7f1b50ca6d20
Caching: in kcfis_finalize_cached_sessions global ctx 0x7f1b50ca6d20 total cached 1 cached in ctx 1
Deallocating session. app state 0x7f1b50ca67b0 num cached sessions 0
In kcfis_deallocate_session: app_state: 0x7f1b50ca67b0
KCFIS: [NSMTIO]:SQL for this (non)Smart I/O session is: 
select count(*) from t
kcfis_cache_appliance_fd: Cached appliance fd 0x7f1b50da2b88  for appliance 0x7f1b5078b3a8
Appliance 0x7f1b5078b3a8 was not active. Active count 0
kcfis_deallocate_session: Freeing disk 0x7f1b50c953f0 name = o/192.168.12.8/DATA_CD_06_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis_deallocate_session: Freeing disk 0x7f1b507908e0 name = o/192.168.12.8/DATA_CD_01_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis_deallocate_session: Freeing disk 0x7f1b507903f0 name = o/192.168.12.8/DATA_CD_05_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis_deallocate_session: Freeing disk 0x7f1b50d04fc0 name = o/192.168.12.8/DATA_CD_10_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis_deallocate_session: Freeing disk 0x7f1b50cfe960 name = o/192.168.12.8/DATA_CD_03_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis_deallocate_session: Freeing disk 0x7f1b50cfdf80 name = o/192.168.12.8/DATA_CD_09_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis_deallocate_session: Freeing disk 0x7f1b50cfc930 name = o/192.168.12.8/DATA_CD_00_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis_deallocate_session: Freeing disk 0x7f1b50cc8118 name = o/192.168.12.8/DATA_CD_11_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis_deallocate_session: Freeing disk 0x7f1b50cc7ae0 name = o/192.168.12.8/DATA_CD_08_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis_deallocate_session: Freeing disk 0x7f1b50cc5ab0 name = o/192.168.12.8/DATA_CD_02_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis_deallocate_session: Freeing disk 0x7f1b50cc3a80 name = o/192.168.12.8/DATA_CD_04_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis_deallocate_session: Freeing disk 0x7f1b50cc30a0 name = o/192.168.12.8/DATA_CD_07_enkcel04 state = 0 appliance = 0x7f1b5078b3a8
kcfis rcv update : op: 6 val: 8192 so_numses 1 ovhdmem 139312 mdmem 1564 bufmem 24576

And the appliances are notified the smartscan has ended:

kcfis_close_all_appliance_fds. Cache count 4
kcfis_close_appliance_fd: appliance_fd 0x7f1b50da2b88
WAIT #139755290955216: nam='cell smart table scan' ela= 241 cellhash#=3249924569 p2=0 p3=0 obj#=61458 tim=1593707978094
kcfis_close_appliance_fd: appliance_fd 0x7f1b51f5db00
WAIT #139755290955216: nam='cell smart table scan' ela= 237 cellhash#=822451848 p2=0 p3=0 obj#=61458 tim=1593707978356
kcfis_close_appliance_fd: appliance_fd 0x7f1b51f5df58
WAIT #139755290955216: nam='cell smart table scan' ela= 280 cellhash#=674246789 p2=0 p3=0 obj#=61458 tim=1593707978668
kcfis_close_appliance_fd: appliance_fd 0x7f1b51f5e3b0
WAIT #139755290955216: nam='cell smart table scan' ela= 242 cellhash#=1034800054 p2=0 p3=0 obj#=61458 tim=1593707978935

Summary.
This blogpost tries to summarise (..) the different steps in an Exadata smartscan. One of the most important things which this shows is that anything which is done, is covered by a single wait event (‘cell smart table scan’). In other words: profiling this wait event tells you little to nothing about what is actually happening, except that a smartscan is being processed. In other words: if you get high ‘cell smart table scan’ waits, either for all the waits or for a few, the only way to pinpoint what it is the wait is showing waiting time is executing this trace again, and see which step it is. Of course you can pinpoint if the waiting is happening for a specific cell/appliance by looking at cellhash# in the wait line.

Disclaimer:
Please mind I tried to use my knowledge on Oracle and Exadata processing together with the information the trace provided to build this description. If you encounter anything which is incorrect, please comment on this post, and I try to get it fixed. No bits where harmed during testing.

This is the fourth post on a serie of postings on how to get measurements out of the cell server, which is the storage layer of the Oracle Exadata database machine. Up until now, I have looked at the measurement of the kind of IOs Exadata receives, the latencies of the IOs as as done by the cell server, and the mechanism Exadata uses to overcome overloaded CPUs on the cell layer.

This post is about the statistics on the disk devices on the operating system, which the cell server also collects and uses. The disk statistics are ideal to combine with the IO latency statistics.

This is how a dump of the collected statistics (which is called “devio_stats”) is invoked on the cell server, using cellcli:

alter cell events="immediate cellsrv.cellsrv_dump('devio_stats',0)"; 

This will output the name of the thread-log file, in which the “devio_stats” dump has been made.

This is a quick peek at the statistics this dump provides (first 10 lines):

[IOSTAT] Dump IO device stats for the last 1800 seconds
2013-10-28 04:57:39.679590*: Dump sequence #34:
[IOSTAT] Device - /dev/sda
ServiceTime Latency AverageRQ numReads numWrites DMWG numDmwgPeers numDmwgPeersFl trigerConfine avgSrvcTimeDmwg avgSrvcTimeDmwgFl
0.000000 0.000000 10 0 6 0 0 0 0 0.000000 0.000000
0.111111 0.111111 15 7 38 0 0 0 0 0.000000 0.000000
0.000000 0.000000 8 4 8 0 0 0 0 0.000000 0.000000
0.000000 0.000000 31 0 23 0 0 0 0 0.000000 0.000000
0.000000 0.000000 8 0 1 0 0 0 0 0.000000 0.000000
0.058824 0.058824 25 0 17 0 0 0 0 0.000000 0.000000
etc.

These are the devices for which the cell server keeps statistics:

grep \/dev\/ /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/log/diag/asm/cell/enkcel01/trace/svtrc_15737_85.trc
[IOSTAT] Device - /dev/sda
[IOSTAT] Device - /dev/sda3
[IOSTAT] Device - /dev/sdb
[IOSTAT] Device - /dev/sdb3
[IOSTAT] Device - /dev/sdc
[IOSTAT] Device - /dev/sde
[IOSTAT] Device - /dev/sdd
[IOSTAT] Device - /dev/sdf
[IOSTAT] Device - /dev/sdg
[IOSTAT] Device - /dev/sdh
[IOSTAT] Device - /dev/sdi
[IOSTAT] Device - /dev/sdj
[IOSTAT] Device - /dev/sdk
[IOSTAT] Device - /dev/sdl
[IOSTAT] Device - /dev/sdm
[IOSTAT] Device - /dev/sdn
[IOSTAT] Device - /dev/sdo
[IOSTAT] Device - /dev/sdp
[IOSTAT] Device - /dev/sdq
[IOSTAT] Device - /dev/sdr
[IOSTAT] Device - /dev/sds
[IOSTAT] Device - /dev/sdt
[IOSTAT] Device - /dev/sdu

What is of interest here is that if the cell disk is allocated inside a partition instead of the whole disk, the cell server will keep statistics on both the entire device (/dev/sda, dev/sdb) and the partition (/dev/sda3, dev/sdb3). Also, the statistics are kept on both the rotating disks and the flash disks, as you would expect.

When looking in the “devio_stats” dump, there are a few other things which are worthy to notice. The lines with statistics do not have timestamp or other time indicator, it’s only statistics. The lines are displayed per device, with the newest line on top. The dump indicates it dumps the IO device statistics which the cell keeps for the last 1800 seconds (30 minutes). If you count the number of lines which (apparently) are kept by the cell server, the count is 599, not 1800. If you divide the time by the number of samples, it appears the cell takes a device statistics snapshot every 3 seconds. The cell server picks up the disk statistics from /proc/diskstats. Also, mind the cell measures the differences between two periods in time, which means the numbers are averages over a period of 3 seconds.

Two other things are listed in the statistics: ‘trigerConfine’ (which probably should be “triggerConfine”), which is a mechanism for Oracle to manage under performing disks.
The other thing is “DMWG”. At this moment I am aware DMWG means “Disk Media Working Group”, and works with the concept of peers.

To get a better understanding of what the difference is between the ServiceTime and Latency columns, see this excellent writeup on IO statistics from Bart Sjerps. You can exchange the ServiceTime for svctm of iostat or storage wait as Bart calls it, and Latency for await or host wait as Bart calls it.

Recently we upgraded an Exadata to the currently latest version, 11.2.3.2.0. The Exadata software itself consists of an image for the storage servers (the storage servers are essentially re-imaged), and a set of updates for the database/computing nodes, including: firmware for ILOM (lights out adapter), BIOS, LSI RAID adapter, Infiniband adapter, linux kernel, drivers, mandatory packages, to name some.

One of the exceptional things this upgrade does, is remove the hot-spare out of the RAID set on the database/compute nodes. This is documented in MOS note: 1468877.1, as ‘known issue 5: hotspare removed for compute nodes’. For some sites, this actually can be a good thing, if they are really tight on disk space on the compute nodes of Exadata. I must say that we have not encountered this situation. What this means, is that the actual HDD configuration on the compute node is left to the customer, instead of having one mandatory configuration (having 3 disks in a RAID-5 configuration, and one hot-spare).

So if you decide to use the former hot-spare disk as an active part of the RAID configuration, you are effectively trading availability for diskspace. Please mind the RAID set itself already provides redundancy, even without the hot-spare!

On the other hand, I think in most configurations, it makes sense to convert the disk back to being hot-spare.

This is done in the following way:

a) Get an overview of the current disk configuration:
/opt/MegaRAID/MegaCli/MegaCli64 -pdlist -a0 | grep -iE "slot|firmware"
Slot Number: 0
Firmware state: Online, Spun Up
Device Firmware Level: 0B70
Slot Number: 1
Firmware state: Online, Spun Up
Device Firmware Level: 0B70
Slot Number: 2
Firmware state: Online, Spun Up
Device Firmware Level: 0B70
Slot Number: 3
Firmware state: Unconfigured(good), Spun Up
Device Firmware Level: 0B70

This shows the disk in slot number 3 being left “unconfigured”, but in “good” state (of a disk has gone bad because of errors, it will be removed from the RAID set, and will show up as “unconfigured(bad)”!)
This is the state the upgrade to 11.2.3.2.0 leaves your system.

New let’s make the disk hot spare again!

b) Get the inclosure id:
/opt/MegaRAID/MegaCli/MegaCli64 -encinfo -a0 | grep ID
Device ID : 252

This means we know the enclosure id (252) and the slot number (3), which is the information needed for the MegaCli utility to revert the unconfigured disk to hot-spare again!

c) Revert the unconfigured disk back to hot-spare
/opt/MegaRAID/MegaCli/MegaCli64 -PdHsp -set -EnclAffinity -PhysDrv[252:3] -a0
Adapter: 0: Set Physical Drive at EnclId-252 SlotId-3 as Hot Spare Success.
Exit Code: 0x00

d) Check the disk configuration again:
/opt/MegaRAID/MegaCli/MegaCli64 -pdlist -a0 | grep -iE "slot|firmware"
Slot Number: 0
Firmware state: Online, Spun Up
Device Firmware Level: 0B70
Slot Number: 1
Firmware state: Online, Spun Up
Device Firmware Level: 0B70
Slot Number: 2
Firmware state: Online, Spun Up
Device Firmware Level: 0B70
Slot Number: 3
Firmware state: Hotspare, Spun Up
Device Firmware Level: 0B70

It appears that having the disk been removed from the RAID set by the update to 11.2.3.2.0 generates an ASR message. At least it did at our site, despite this being an undocumented bug (7161023,’ASR generating false errors in relation to disks’) which is marked resolved in 11.2.3.1.0 (?). Most sites I encounter have ASR setup, but not having all messages send additionally to local, onsite monitoring. I want to stress it’s very important to have the ASR messages sent to your own monitoring too!

Oracle Support does not list all the specifications from an ASR message it has gotten. Instead, a Service Request is made with enough information for Oracle itself (!!). In our case, the exact error message was NOT specified, only ‘compute server hard disk predictive failure’ and the node name.

Where do you look on an Exadata for that information? The first logical point is the ASR daemon. I didn’t spend too much time on it, but it seems that it’s more a proxy for messages than a database. I wasn’t able to find useful information about the systems which where using this daemon.

What are the sources for ASR with an Exadata? These are:

Computing node:
– “compmon daemon” / Linux level monitoring
– ILOM
Storage node:
– “cell daemon” / Linux level monitoring
– ILOM

For the computing node, it’s quite easy to see if there are any detected failed devices from the viewpoint of the ILOM:
(please mind ipmitool -I open only works on the local system)
# ipmitool -I open sunoem cli "show /SP/logs/event/list Severity==(Major,Critical,Down)"
Connected. Use ^D to exit.
-> show /SP/logs/event/list Severity==(Major,Critical,Down)

ID Date/Time Class Type Severity
----- ------------------------ -------- -------- --------

-> Session closed
Disconnected

This shows no messages with the severity Major, Critical or Down are in the eventlog in the ILOM. Please mind that the logons to the ILOM have severity “Minor”. These are in most system the vast majority of the messages, which are not of interest for this investigation. If you want to know if something has failed, there even a simpler command:
# ipmitool -I open sunoem cli "show faulty"

For the “compmon daemon”, grep the processlist for “compmon”:
# ps -ef | grep compmon
root 12812 1 0 Oct22 ? 00:00:11 /usr/bin/perl -w /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -server

The most important part here is the directory: /opt/oracle/cellos/compmon
If you navigate to that directory, you will see a number of “state files”: asrs.state, traps.state and disks.state.
The disks.state lists the disk status as listed with a) with the firmware state.
The most important file for the ASR message investigation is the traps.state file. This file lists traps it has sent to ASR. In our case:
1 ; Mon Oct 22 14:39:10 2012 ; 86425886-b359-4587-8d46-f31ff2ecb135 ; Physicaldisk : Make Model: is at status predictive failure. Raised fault id: HALRT-02008 ; Physical disk should be replaced. Exadata Compute Server: Disk Serial Number:
Yes, this is pasted correctly, it misses Physicaldisk, Make Model and Disk Serial Number information. This has not been omitted for safety, it just is not listed.
So, the failure which was sent was HALRT-02008 in our case.

For completeness, the ILOM layer can be investigated identically to the description of the ILOM handling on the computing layer. The Linux layer messages can be investigated with: # cellcli -e list alerthistory
32 2012-10-17T02:00:27+02:00 info "HDD disk controller battery on disk contoller at adapter 0 is going into a learn cycle. This is a normal maintenance activity that occurs quarterly and runs for approximately 1 to 12 hours. The disk controller cache might go into WriteThrough caching mode during the learn cycle. Disk write throughput might be temporarily lower during this time. The message is informational only, no action is required."
33 2012-10-22T11:43:21+02:00 info "Factory defaults restored for Adapter 0"
34 2012-10-22T11:43:23+02:00 info "Factory defaults restored for Adapter 0"

%d bloggers like this: