Control file inconsistencies during startup when using ASM (ORA-000214)

During some testing I encountered an ORA-000214 during startup of an Oracle 11.2.0.2 database instance:

ORA-00214: control file '+RECO_XXXX/test/controlfile/current.334.755391511'
version 268 inconsistent with file
'+DATA_XXXX/test/controlfile/current.299.755390399' version 265


This is a RAC instance on Exadata, but all techniques in this article will work on any Oracle 11.2.x database using ASM.

This message means the database found two controlfiles which have a different version. If this message appears when the database is open, the database will crash. If an instance is startup after this message, the same error appears, and the database remains in nomount state. Further diagnosis: the control file version in the recovery area is more recent than the version in the data diskgroup (version 268 versus version 265).

The obvious and correct way of resolving this issue is to copy the latest control file version in the recovery area over the older one in the data diskgroup.

The most simple and obvious way is to use asmcmd:

[oracle@xxxxdb01 [test1] ~]$ +ASM1
The Oracle base for ORACLE_HOME=/u01/app/11.2.0/grid is /u01/app/oracle
[oracle@xxxxdb01 [+ASM1] ~]$ asmcmd

ASMCMD> cp +RECO_XXXX/test/controlfile/current.334.755391511 +DATA_XXXX/test/controlfile/current.299.755390399
copying +RECO_XXXX/test/controlfile/current.334.755391511 -> +DATA_XXXX/test/controlfile/current.299.755390399
ASMCMD-08016: copy source->'+RECO_XXXX/test/controlfile/current.334.755391511' and target->'+DATA_XXXX/test/controlfile/current.299.755390399' failed
ORA-15056: additional error message
ORA-15046: ASM file name '+DATA_XXXX/test/controlfile/current.299.755390399' is not in single-file creation form
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 410
ORA-06512: at line 3 (DBD ERROR: OCIStmtExecute)
ASMCMD>


This means asmcmd can not be used to copy the controlfile.

The resolution for this issue is to use rman to copy the controlfile:

[oracle@xxxxdb01 [test1] ~]$ rman target /

Recovery Manager: Release 11.2.0.2.0 - Production on Sat Jul 2 17:23:32 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.

connected to target database: TEST (not mounted)

RMAN> restore controlfile to '+DATA_XXXX' from '+RECO_XXXX/controlfile/current.334.755391511';

Starting restore at 02-JUL-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=847 instance=test1 device type=DISK

channel ORA_DISK_1: copied control file copy
Finished restore at 02-JUL-11

RMAN> exit


By only specifying the diskgroup, a new controlfile is created using OMF (Oracle Managed Files) naming.

At this point we have a valid, consistent control file in the data diskgroup, in order for the database to be able to use it, we need to do the following tasks:
– Gather the name of the new, consistent controlfile in the data diskgroup.
– Remove the old, inconsistent controlfile in the data diskgroup.
– Set the control_files parameter to use the new, copied version in the data diskgroup and use the copy in the recovery area.

Gather the name of the new controlfile in the data diskgroup:
[oracle@xxxxdb01 [+ASM1] ~]$ asmcmd
ASMCMD> find -t controlfile +DATA_EMC1/test *
WARNING:option 't' is deprecated for 'find'
please use 'type'

+DATA_XXXX/test/CONTROLFILE/current.299.755390399
+DATA_XXXX/test/CONTROLFILE/current.314.755457847
ASMCMD>
We know the old, inconsistent version from the errormessage: current.299.755390399. So the new version is current.314.755457847.

Remove the old one:
ASMCMD> rm +DATA_EMC1/test/CONTROLFILE/current.299.755390399

And set the parameter and bounce the instance to test if it works:
[oracle@xxxxdb01 [+ASM1] ~]$ test1
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is /u01/app/oracle
[oracle@xxxxdb01 [test1] ~]$ sq

SQL*Plus: Release 11.2.0.2.0 Production on Sat Jul 2 17:30:22 2011

Copyright (c) 1982, 2010, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> alter system set control_files='+DATA_XXXX/test/CONTROLFILE/current.314.755457847','+RECO_XXXX/test/controlfile/current.334.755391511' scope=spfile sid='*';

System altered.

SQL> startup force;
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size 2233088 bytes
Variable Size 478153984 bytes
Database Buffers 423624704 bytes
Redo Buffers 139874304 bytes
Database mounted.
Database opened.
SQL>


• ASM
• exadata
• ora-000214
• ora-214
• oracle