ORA-00257,UDE-00257 Archiver Error Connect Internal Only, Until Freed in RAC

>> As per the Oracle Support they suggested to bounce the complete cluster and restart all the                  databases for this issue.

Problems Faced.....

>> RMAN backups are not executing

E:\Backup\Scripts> runrmanhr.cmd


>> Data-pump jobs are also not running its showing Archiver Error then i checked Archive destination.

D:\otsexp> expdp dumpfile=OTSLIVE_23_AUG2015FULL directory=otslive full=y


Export: Release 11.2.0.3.0 - Production on Sun Nov 15 18:09:39 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

Username : system
Password : xxxxx

UDE-00257 : operation generated Oracle error 257
ORA-00257 : archiver error. Connect Internal only, until freed



>> I checked the ASM diskgroup status & space for archive destination lot of space is available
   every thing is fine upto here.


SQL> select name,total_mb,free_mb from v$asm_diskgroup;

NAME                             TOTAL_MB         FREE_MB
------------------------------ ---------- -----------------------------
DATA                               511994                385213
OCRVOTE                       102397                102002
RECO                               511997                460833



SQL>  select name,inst_id,state from gv$asm_diskgroup;

NAME                              INST_ID          STATE
------------------------------ ---------- ------------------------
DATA                                    1               MOUNTED
OCRVOTE                            1               MOUNTED
RECO                                   1               MOUNTED
DATA                                    2               MOUNTED
OCRVOTE                            2               MOUNTED
RECO                                   2               MOUNTED

6 rows selected.


>> I checked the datafile status all are ok.


SQL> select file_name,status from dba_data_files

FILE_NAME                                                                        STATUS
--------------------------------------------- ---------------------------------------
+DATA/otslive/datafile/system.258.860628911               AVAILABLE
+DATA/otslive/datafile/sysaux.264.860628911               AVAILABLE
+DATA/otslive/datafile/undotbs1.262.860628913           AVAILABLE
+DATA/otslive/datafile/users.259.860628913                 AVAILABLE
+DATA/otslive/datafile/undotbs2.269.860629051          AVAILABLE
+DATA/otslive/datafile/users.256.860630027                AVAILABLE
+DATA/otslive/datafile/users.274.860630073                AVAILABLE

7 rows selected.


>> When i check CRS resource its showing INTERMEDIATE & CHECK TIMED OUT status for ocrvoting diskgroup and both my production database bablive and otslive...


 E:\Backup\Scripts>crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       oradb1
               ONLINE  ONLINE       oradb2
ora.LISTENER.lsnr
               ONLINE  ONLINE       oradb1
               ONLINE  ONLINE       oradb2
ora.OCRVOTE.dg
               ONLINE  INTERMEDIATE oradb1                    CHECK TIMED OUT
               ONLINE  ONLINE       oradb2
ora.RECO.dg
               ONLINE  ONLINE       oradb1
               ONLINE  ONLINE       oradb2
ora.asm
               ONLINE  ONLINE       oradb1                   Started
               ONLINE  ONLINE       oradb2                   Started
ora.gsd
               OFFLINE OFFLINE      oradb1
               OFFLINE OFFLINE      oradb2
ora.net1.network
               ONLINE  ONLINE       oradb1
               ONLINE  ONLINE       oradb2
ora.ons
               ONLINE  ONLINE       oradb1
               ONLINE  ONLINE       oradb2
ora.registry.acfs
               ONLINE  ONLINE       oradb1
               ONLINE  ONLINE       oradb2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       oradb2
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       oradb1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       oradb1
ora.cvu
      1        ONLINE  ONLINE       oradb1
ora.bablive.db
      1        ONLINE  INTERMEDIATE oradb1                   CHECKED TIMED OUT
      2        ONLINE  ONLINE       oradb2                   Open
ora.otslive.db
      1        ONLINE  INTERMEDIATE oradb1                   CHECKED TIMED OUT
      2        ONLINE  ONLINE       oradb2                   Open
ora.oc4j
      1        ONLINE  ONLINE       oradb1
ora.oradb1.vip
      1        ONLINE  ONLINE       oradb1
ora.oradb2.vip
      1        ONLINE  ONLINE       oradb2
ora.scan1.vip
      1        ONLINE  ONLINE       oradb2
ora.scan2.vip
      1        ONLINE  ONLINE       oradb1
ora.scan3.vip
      1        ONLINE  ONLINE       oradb1


Steps:-
------

1. First we announce the down time for the users for 1 hour.

2. Shutdown the application.

3. Started Bouncing our Cluster

4. Use srvctl utility to stop and start the database in RAC.

   > Srvctl stop database -d otslive

   > Srvctl stop database -d bablive


   > Node1 : crsctl stop crs
   > Node2 : crsctl stop crs


   > Node1 : crsctl start crs
   > Node2 : crsctl stop  crs


Now check the crs services are online or not....

   > Node1 : crsctl check crs
   > Node2 : crsctl check crs


If all the CRS services are online then you can check the status.

D:\app\oracle>srvctl status database -d otslive

   Instance OTSLIVE1 is running on node oradb1
   Instance OTSLIVE2 is running on node oradb2

D:\app\oracle>srvctl status database -d bablive

   Instance BABLIVE1 is running on node oradb1
   Instance BABLIVE2 is running on node oradb2

Happy Learning ....