Alter database open fails with ORA-00600 kcratr_nab_less_than_odr

APPLIES TO:

SYMPTOMS

After Power Fail Alter database open fails with

ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr]
 

# The Database can't open at this Point. In the corresponding Tracefile we can find this Error Callstack:


dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=1h50ks4ncswfn) -----
ALTER DATABASE OPEN

----- Call Stack Trace -----
ksedst1 <- ksedst <- dbkedDefDump <- ksedmp <- dbgexPhaseII <- dbgexProcessError <- dbgePostErrorKGE  <- kgeasnmierr <- kcratr_odr_check  <- kcratr <- kctrec <- kcvcrv <- kcfopd <- adbdrv <- opiexe <- opiosq0 <- kpoal8 <- opiodr <- ttcpip <- opitsk <- opiino <- opiodr <- opidrv <- sou2o <- opimai_real <- ssthrdmain <- main <- start

CHANGES

Power failure

CAUSE

There was a power failure causing logical corruption in controlfile

This Problem is caused by Storage Problem of the Database Files.
The Subsystem (eg. SAN) crashed while the Database was open.
The Database then crashed because the Database Files were not accessible anymore.
This caused a lost Write into the Online RedoLogs and/or causing logical corruption in controlfile so Instance Recovery is not possible and raising the ORA-600.
 

SOLUTION

Option a:  Do cancel based reocvery, and apply 'current online redolog' manually
------------

SQL>Startup mount ;

SQL>Show parameter control_files

Query 1
------------

SQL> select a.member, a.group#, b.status from v$logfile a ,v$log b where a.group#=b.group# and b.status='CURRENT' ;

Note down the name of the redo log

SQL> Shutdown abort ;

Take a OS Level backup of the controlfile (This is to ensure we have a backup of current state of controlfile)

SQL> Startup mount ;
SQL> recover database using backup controlfile until cancel ;

Enter location of redo log shown as current in Query 1 when prompted for recovery

Hit Enter

SQL> Alter database open resetlogs ;

Option b: Recreate the controlfile using the Controlfile recreation script
-----------

NOTE:  If you are using RMAN backup without a catalog, you will need to catalog the backups after the controlfile is recreated.  For tape backups, capture your backup information before recreating the controlfile as this will be needed.  

With database in mount stage

$ rman target /
rman> spool log to '/tmp/rman.log';
rman> list backup ;
rman> exit

# Keep this log handy

Go to sqlplus

SQL> Show parameter control_files

Keep this location handy.

SQL> oradebug setmypid
SQL> Alter session set tracefile_identifier='controlfilerecreate' ;
SQL> Alter database backup controlfile to trace noresetlogs;
SQL> oradebug tracefile_name ; --> This command will give the path and name of the trace file

Go to this location ,Open this trace file and select the controlfile recreation script with NOResetlogs option

SQL> Shutdown immediate;

Rename the existing controlfile to _old ---> This is Important as we need to have a backup of existing controlfile since we plan to recreate it

SQL> Startup nomount

Now run the Controlfile recreation script with NO Resetlogs option.

SQL> Alter database open ;

For database version 10g and above

Once database is opened you can recatalog the rman backup information present in the list /tmp/rman.log using

Rman> Catalog start with '' ;

If backups are on tape, and you are not using a catalog, backups can be cataloged using information in:
How to Catalog Tape Backup Pieces (Doc ID 550082.1)

Once the database has been opened using the Option a or Option b,  it is recommended to take a hot backup of the database.  The same steps are applicable to RAC if all instance are down with same error.

If  both Options (a|b) fails,  or you do not have the full set of files, then you have to restore and recover the Database from a recent Backup. 

REFERENCES