This blogpost is about opatch and how to obtain information about the current oracle home(s), and how to obtain information about the patches to be applied.
Patches that can be applied using opatch are provided by oracle as zip files which have the following naming convention:
p[patchnumber]_[baseversion]_[platform]-[architecture].zip. The patch normally contains an XML file called ‘PatchSearch.xml’ and a directory with the patch number. Inside the patch number directory there is a README.txt which is lame, because it says ‘Refer to README.html’, and a README.html that contains the readme information that is also visible when the [README] button for this patch is selected in MOS.
I spend my time on the CLI exclusively. This is because I spend my time on remote servers all the time, and using the X window system would be unusable. The best part of using the CLI is that when done correctly, it gives you almost infinite control over what you do, while when clicking through an interface toggling selections and filling out fields makes you entering information that you have to copy from a document or make up on the spot, which then is quickly hidden by entering another tab or window. In fact, based on my experience, this is a guaranteed way of generating wrong or inconsistent results.
Because of being on the CLI almost exclusively on servers, I sometimes need to read the README.html. I can do that in MOS, but sometimes you want that information on the spot. You can open up the HTML file in ‘vim’ or ‘less’, but it will show you a lot of HTML making it very hard to read. What I find useful is installing an executable called ‘elinks’ (available on oracle linux via yum), and then read the README.html in this text based browser:
$ elinks README.html
Oracle® Database Patch % psuBug % - Database Release Update % product_version % (1/12) Go to primary content Patch 28655784 - Database Release Update 220.127.116.11.181016 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── The script content on this page is for navigation purposes only and does not alter the content in any way. Oracle® Database Patch 28655784 - Database Release Update 18.104.22.168.181016 This document is accurate at the time of release. For any changes and additional information regarding Database Release Update 22.214.171.124.181016, see these related documents that are available at My Oracle Support (http://support.oracle.com/): * Document 2433586.1 Oracle DB/GI/OJVM Update/Revision R18 Oct 2018 Known Issues This document includes the following sections: * Section 1, "Patch Information" * Section 2, "Prerequisites" * Section 3, "Installation" * Section 4, "Deinstallation" * Section 5, "Known Issues" * Section 6, "References" * Section 7, "Bugs Fixed by This Patch" * Section 8, "Documentation Accessibility" 1 Patch Information Database Release Update 126.96.36.199.181016 patches are cumulative. That is, the content of all previous Database bundles is included in the latest Database bundle patch. To install the Database Release Update 188.8.131.52.181016 patch, the Oracle home must have the 18.1 Database installed. http://support.oracle.com/ [------]
The first thing I do, is obtain information about this patch. Based on my experience, very carefully examine the readme. It contains vital information about the patch, but I found inconsistencies in it in the past. I got the impression the readme’s of patches that are created with a certain interval are simply copied from a previous version and then corrected by hand, and things might be entered wrong or are forgotten and not touched, leaving wrong information.
opatch lspatches on an unzipped patch
Luckily, a lot of information in the readme is actually stored in the metadata of the patch. You can query the metadata of the patch to be applied using opatch lspatches by pointing it to the patch directory:
(in this example I am querying patch 28655784, which is RU 18.4, and my database oracle home is version 18.3)
$ /u01/app/oracle/product/184.108.40.206/dbhome_1/OPatch/opatch lspatches 28655784/ patch_id:28655784 unique_patch_id:22509982 date_of_patch:8 Oct 2018, 21:27:28 hrs PST8PDT patch_description:Database Release Update : 220.127.116.11.181016 (28655784) component:oracle.rdbms.rsf.ic,18.104.22.168.0,optional; oracle.oracore.rsf,22.214.171.124.0,optional; oracle.ctx.atg,126.96.36.199.0,optional; oracle.rdbms.rman,188.8.131.52.0,optional; oracle.rdbms.rsf,184.108.40.206.0,optional; oracle.sdo.locator.jrf,220.127.116.11.0,optional; .... platform:226,Linux x86-64 executable:ORACLE_HOME/lib/libclntsh.so.18.1; ORACLE_HOME/lib/libasmclntsh18.so; ORACLE_HOME/lib/libskgxp11.so; ORACLE_HOME/lib/libskgxp18.so; ORACLE_HOME/lib/libsqlplus.so; ORACLE_HOME/bin/oracle; ... instance_shutdown:true online_rac_installable:true patch_type:singleton product_family:db auto:false bug:28571483, TRACKING BUG TO REGRESS ALL BLR/CIS OF 27502420 ...
(edited for brevity)
Line 2: patch_id: patch number
Line 5: patch_description: description 🙂
Line 6: component: to be patched objects grouped in a ‘component’
Line 7: platform: the platform the patch is intended for
Line 8: executable: the executables that are patched by this patch
Line 9: instance_shutdown: does an instance have to shutdown when it’s ORACLE_HOME is patched?
Line 10: online_rac_installable: does the database have to go down entirely, or can the patch be applied rolling?
Line 11: patch_type: singleton (a single patch). other types that I know: bundle_member (part of a patch bundle).
Line 12: product_family: for which oracle product is this patch? Grid infra also is family ‘db’.
Line 13: auto: can this patch be applied using ‘opatch auto’?
opatch lspatches can also be used to validate if a patch has been applied to the oracle home:
$ opatch lspatches -verify 28655784/ Inventory check failed: Patch ID 28655784 is NOT registered in Oracle Home "/u01/app/oracle/product/18.104.22.168/dbhome_1" inventory. Following patches [ 28655784 ] are NOT registered in Oracle Home "/u01/app/oracle/product/22.214.171.124/dbhome_1" inventory or can't load its meta-data OPatch failed with error code 1
Please mind the argument to -verify is a unzipped patch patch-number directory.
This functionality is also available using ‘opatch query’ (executed as opatch query [argument] directory) opatch query is meant to be run on an unzipped patch patch-number directory, not on an ORACLE_HOME. The next overview takes the above opatch lspatches output, and lists for what lines there is an equivalent opatch query argument to retrieve that information:
Line 2: patch_id: is the patch number, which is the directory that is entered as an argument.
Line 5: patch_description: the description cannot be extracted on its own, but is visible with argument -all.
Line 6: component: argument: -get_component
Line 7: platform: argument: -get_os
Line 8: executable: the executables cannot be fetched on its own, but is visible with argument -all.
Line 9: instance_shutdown: argument: -is_online_patch
Line 10: online_rac_installable: argument: -is_rolling_patch
Line 11: patch_type: argument: -get_patch_type
Line 12: product_family: argument: -get_product_family
Line 13: auto: argument: -is_auto_patch
However, opatch query offers a couple of other switches which can come in handy:
-is_system_patch: System patches are patches that contain several sub-patches and must be applied using opatchauto.
-get_patching_model: The way the patch is applied, normal patching seem to have model “one-off”.
opatch lsinventory (or shorthand: lsinv)
Now the we looked at a patch to be applied, we should also look at the metadata of the oracle home to be patched, which is stored in the inventory. The contents of the inventory can be shown using lsinventory or shorthand lsinv.
$ /u01/app/oracle/product/126.96.36.199/dbhome_1/OPatch/opatch lsinv Oracle Interim Patch Installer version 188.8.131.52.14 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/184.108.40.206/dbhome_1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/220.127.116.11/dbhome_1/oraInst.loc OPatch version : 18.104.22.168.14 OUI version : 22.214.171.124.0 Log file location : /u01/app/oracle/product/126.96.36.199/dbhome_1/cfgtoollogs/opatch/opatch2018-11-13_08-18-30AM_1.log Lsinventory Output file location : /u01/app/oracle/product/188.8.131.52/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2018-11-13_08-18-30AM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: ip-10-0-11-24.local ARU platform id: 226 ARU platform description:: Linux x86-64 Installed Top-level Products (1): Oracle Database 18c 184.108.40.206.0 There are 1 products installed in this Oracle Home. Interim patches (4) : Patch 27908644 : applied on Wed Jul 18 13:44:11 EDT 2018 Unique Patch ID: 22153180 Patch description: "UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171" Created on 4 May 2018, 01:21:02 hrs PST8PDT Bugs fixed: 27908644 Patch 27923415 : applied on Wed Jul 18 13:41:38 EDT 2018 Unique Patch ID: 22239273 Patch description: "OJVM RELEASE UPDATE: 220.127.116.11.180717 (27923415)" Created on 15 Jul 2018, 10:33:22 hrs PST8PDT Bugs fixed: 27304131, 27539876, 27952586, 27642235, 27636900, 27461740 Patch 28090553 : applied on Wed Jul 18 13:40:01 EDT 2018 Unique Patch ID: 22256940 Patch description: "OCW RELEASE UPDATE 18.104.22.168.0 (28090553)" Created on 11 Jul 2018, 19:20:31 hrs PST8PDT Bugs fixed: 12816839, 18701017, 22734786, 23698980, 23840305, 25709124, 25724089 26299684, 26313403, 26433972, 26527054, 26586174, 26587652, 26647619 26827699, 26860285, 26882126, 26882316, 26943660, 26996813, 27012915 ...etc...
Line 6: the ORACLE_HOME for which the contents are listed.
Line 7: the central inventory location.
Line 9: the opatch version
Line 10: the oracle universal install version
Line 17: the hostname
Line 18/19: the platform description and identification
Line 20/24: installed ‘top level’ products
Line 27- : applied interim patches
This is not very detailed data (except for the bugs fixed numbers), however, there is an option to get more information: add ‘-detail’ to the lsinv command:
$ /u01/app/oracle/product/22.214.171.124/dbhome_1/OPatch/opatch lsinv -detail Oracle Interim Patch Installer version 126.96.36.199.14 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/188.8.131.52/dbhome_1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/184.108.40.206/dbhome_1/oraInst.loc OPatch version : 220.127.116.11.14 OUI version : 18.104.22.168.0 Log file location : /u01/app/oracle/product/22.214.171.124/dbhome_1/cfgtoollogs/opatch/opatch2018-11-13_10-15-09AM_1.log Lsinventory Output file location : /u01/app/oracle/product/126.96.36.199/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2018-11-13_10-15-09AM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: ip-10-0-11-24.local ARU platform id: 226 ARU platform description:: Linux x86-64 Installed Top-level Products (1): Oracle Database 18c 188.8.131.52.0 There are 1 products installed in this Oracle Home. Installed Products (125): Assistant Common Files 184.108.40.206.0 BLASLAPACK Component 220.127.116.11.0 Buildtools Common Files 18.104.22.168.0 Cluster Verification Utility Common Files 22.214.171.124.0 Database Configuration and Upgrade Assistants 126.96.36.199.0 Database Migration Assistant for Unicode 188.8.131.52.0 ...etc... Patch 27923415 : applied on Wed Jul 18 13:41:38 EDT 2018 Unique Patch ID: 22239273 Patch description: "OJVM RELEASE UPDATE: 184.108.40.206.180717 (27923415)" Created on 15 Jul 2018, 10:33:22 hrs PST8PDT Bugs fixed: 27304131, 27539876, 27952586, 27642235, 27636900, 27461740 Files Touched: java.security --> ORACLE_HOME/javavm/jdk/jdk8/lib/security/java.security cacerts --> ORACLE_HOME/javavm/jdk/jdk8/lib/security/cacerts sunjce_provider.jar --> ORACLE_HOME/javavm/jdk/jdk8/lib/sunjce_provider.jar jce.jar --> ORACLE_HOME/javavm/jdk/jdk8/lib/jce.jar classes.bin --> ORACLE_HOME/javavm/jdk/jdk8/admin/classes.bin lfclasses.bin --> ORACLE_HOME/javavm/jdk/jdk8/admin/lfclasses.bin jvmpsupii.sql --> ORACLE_HOME/javavm/install/jvmpsupii.sql jvmpsupi.sql --> ORACLE_HOME/javavm/install/jvmpsupi.sql jvmpsu.sql --> ORACLE_HOME/javavm/install/jvmpsu.sql jvmpsui.sql --> ORACLE_HOME/javavm/install/jvmpsui.sql jvmpsupdi.sql --> ORACLE_HOME/javavm/install/jvmpsupdi.sql jvmpsupdii.sql --> ORACLE_HOME/javavm/install/jvmpsupdii.sql libjavavm18.a --> ORACLE_HOME/javavm/jdk/jdk8/lib/libjavavm18.a 27923415.xml --> ORACLE_HOME/sqlpatch/27923415/22239273/27923415.xml 27923415_apply.sql --> ORACLE_HOME/sqlpatch/27923415/22239273/27923415_apply.sql 27923415_rollback.sql --> ORACLE_HOME/sqlpatch/27923415/22239273/27923415_rollback.sql jvmpsupdi.sql --> ORACLE_HOME/sqlpatch/27923415/22239273/rollback_files/220.127.116.11.0/javavm/install/jvmpsupdi.sql jvmpsupdii.sql --> ORACLE_HOME/sqlpatch/27923415/22239273/rollback_files/18.104.22.168.0/javavm/install/jvmpsupdii.sql jvmpsupii.sql --> ORACLE_HOME/sqlpatch/27923415/22239273/rollback_files/22.214.171.124.0/javavm/install/jvmpsupii.sql jvmpsu.sql --> ORACLE_HOME/sqlpatch/27923415/22239273/rollback_files/126.96.36.199.0/javavm/install/jvmpsu.sql jvmpsui.sql --> ORACLE_HOME/sqlpatch/27923415/22239273/rollback_files/188.8.131.52.0/javavm/install/jvmpsui.sql jvmpsupi.sql --> ORACLE_HOME/sqlpatch/27923415/22239273/rollback_files/184.108.40.206.0/javavm/install/jvmpsupi.sql jox.o --> ORACLE_HOME/rdbms/lib/jox.o ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/jox_on ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/ioracle aurora.zip --> ORACLE_HOME/javavm/lib/aurora.zip Patch Location in Inventory: /u01/app/oracle/product/220.127.116.11/dbhome_1/inventory/oneoffs/27923415 Patch Location in Storage area: /u01/app/oracle/product/18.104.22.168/dbhome_1/.patch_storage/27923415_Jul_15_2018_10_33_22 Patch 28090553 : applied on Wed Jul 18 13:40:01 EDT 2018 Unique Patch ID: 22256940 Patch description: "OCW RELEASE UPDATE 22.214.171.124.0 (28090553)" Created on 11 Jul 2018, 19:20:31 hrs PST8PDT Bugs fixed: 12816839, 18701017, 22734786, 23698980, 23840305, 25709124, 25724089 ...etc...
The start of the output is the same as without ‘-detail’. However, on lines 27-34 you see the products, which as far as I understand it are what is meant with ‘components’ when looking at the patch details using ‘opatch lspatches’ and ‘opatch query’.
If we look at the details of patch 27923415 (line 36), at the end of the ‘Files Touched’ list at lines 69-72 it shows where the metadata is stored of this patch; that’s in the inventory directory in oneoffs/27923415. This directory contains information about the currently applied patch contents in the home.
The storage area contains the previous version of the contents that have been patched in case a rollback is needed.
Now the we looked at the patch and the ORACLE_HOME to be patched, the next thing we can look at is if the environment is ready for the patch that was inspected to be applied to the ORACLE_HOME that was inspected.
Please mind that normally you would follow the readme, maybe run the prereq CheckConflictAgainstOHWithDetail (I don’t normally, because I check for conflicts before that), and then run opatch apply. And that is perfectly fine! Opatch performs all the checks that are listed below before applying a patch. That is one of the reasons opatch takes such a long time: it does perform a lot of work to be absolutely certain everything is alright.
The reason for listing the below individual checks is first of all to make the reader aware that all the checks opatch apply executes are available individually to be performed prior to patching. Another reason is to document and logically group these checks.
In fact, the PSU patching documentation advises to run the check CheckConflictAgainstOHWithDetail before applying the database PSU. I hope you now understand this is just a check that would have been performed anyway, but by explicitly executing this check before patching, you can validate the patch is apply-able to the oracle home in the first place.
Once opatch fails these individual checks can be used to validate the error and obtain more information:
If you read through them, you’ll see there are checks which are supersets of other checks.
— os checks
– Is there enough disk space available?
opatch prereq CheckSystemSpace -ph 28655784/
– Are all the operating system executables needed for patching available?
opatch prereq CheckSystemCommandAvailable -ph 28655784/
– Is the patch compatible with the OS/platform?
opatch prereq CheckPatchApplicableOnCurrentPlatform -ph 28655784/
– Are the files and directory structure and rights of the unzipped patch sane?
opatch prereq CheckPatchShipHome -ph 28655784/
– Are any files that are touched by the patch still open by processes?
opatch prereq CheckActiveFilesAndExecutables -ph 28655784/
– Is the current user ‘root’? If so, fail the check
opatch prereq CheckUserAdminPrivilege -ph 28655784/
— inventory checks
– Check if the ORACLE_HOME to be patched is available in the central inventory
opatch prereq CheckCentralInventoryForOH -ph 28655784/
– Is the central inventory writeable?
opatch prereq CheckCentralInventoryForRWSession -ph 28655784/
– Is the central inventory sane?
opatch prereq CheckCentralInventoryLocation -ph 28655784/
— oracle home checks
– Are the required components available in the oracle home?
opatch prereq CheckComponents -ph 28655784/
– Are the required components (products) available in the ORACLE_HOME, and are the actions applicable?
opatch prereq CheckApplicable -ph 28655784/
– Are there patches applied to the ORACLE_HOME already that are in the patches to be applied
opatch prereq CheckForIdenticalPatchInOracleHome -ph 28655784/
– Check if the oracle home is locked for patching (which might be by a previous unsuccessful opatch execution)
opatch prereq CheckIfOHLockedForPatching -ph 28655784/
– Check if the oracle home is valid, check for proper inventory files in the oracle home
opatch prereq CheckOracleHome -ph 28655784/
– Check if all the patches that are required by the patch to be applied are present in the oracle home
opatch prereq CheckPatchApplyDependents -ph 28655784/
– Check if the patching model of the patch is compatible with the oracle home
opatch prereq CheckPatchingModel -ph 28655784/
— conflict checks
– Are there conflicts between the patches in the ORACLE_HOME and the patches to be applied?
opatch prereq CheckConflictAgainstOH -ph 28655784/
– Are there conflicts between the patches in the ORACLE_HOME and the patches to be applied,
– and print out detailed conflict information.
opatch prereq CheckConflictAgainstOHWithDetail -ph 28655784/
– Are there conflicts between the patches to be applied?
– (please mind this requires multiple patches, which can be specified using -phBaseDir)
opatch prereq CheckConflictAmongPatches -phBaseDir
– Are there conflicts between the patches to be applied,
– and print out detailed conflict information.
– (please mind this requires multiple patches, which can be specified using -phBaseDir)
opatch prereq CheckConflictAmongPatchesWithDetail -phBaseDir
— RAC checks
– Check if the central inventory has a CRS home if the database home is RAC
opatch prereq CheckForCRSHomeIfRAC -ph 28655784/
– Check if all the nodes in the RAC setup are valid, up and reachable
opatch prereq CheckRACNodeList -ph 28655784/
– Check if commands can be invoked on the remote machines
opatch prereq CheckRemoteCommandInvocable -ph 28655784/
– Check if files can be copied to and removed from the remote machines
opatch prereq CheckRemoteCopyAndRemove -ph 28655784/
— opatch checks
– Check if the input values provided to opatch are sufficient to run the patch action
opatch prereq CheckForInputValues -ph 28655784/
– Is the opatch version high enough as per patch requirement?
opatch prereq CheckMinimumOPatchVersion -ph 28655784/
— rollback checks
– Check if all the patches provided to rollback are present in the given oracle home
opatch prereq CheckInstalledOneOffs -ph 28655784/
– Check if there are patches applied that depend on the patch that is being rolled back
opatch prereq CheckPatchRollbackDependents -ph 28655784/
– Check if the patch can be rolled back from the oracle home
opatch prereq CheckRollbackable -ph 28655784/
— OUI checks
– Is the OUI available in the ORACLE_HOME?
opatch prereq CheckOUILocation -ph 28655784/
– Is the OUI version high enough?
opatch prereq CheckOUIVersionCompatible -ph 28655784/
– Are all OUI libraries present?
opatch prereq CheckRequiredLibs -ph 28655784/
— oraInst.loc check
– Is oraInst.loc file proper?
opatch prereq CheckOraInstLocation -ph 28655784/
A few options of which I currently am not sure what they exactly do:
– CheckApplicableProduct — ?
opatch prereq CheckApplicableProduct -ph 28655784/
– CheckForNoOpPatches — ?
opatch prereq CheckForNoOpPatches -ph 28655784/
– CheckRollbackSid — ?
opatch prereq CheckRollbackSid -ph 28655784/
This is the second post about opatch, and I didn’t even get to the actual patching yet.
This post showed a number of ways to inspect both the patch to be applied (opatch lspatches/query) and the oracle home to apply the patch to (opatch lspatches/lsinventory), and a way to check specific properties before applying a patch and to validate and check things after a patch (opatch prereq).
This is important once you have to apply a patch to validate a machine in general, the central inventory and the oracle home inventory for readiness, and even more important to check an environment in case of patch failure.