一个顽固的ORA-15137处理过程

19C的RAC两个节点patch level不一致,导致无法操作asm的diskgroup,包括增加、删除dg时报错如下:
ORA-15039: diskgroup not dropped
ORA-15137: The ASM cluster is in rolling patch state.

■查看软件版本
crsctl query crs softwareversion
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [19.0.0.0.0]

■查看patch level
[root@zzjk01:0 ~]$ crsctl query crs softwarepatch zzjk01
Oracle Clusterware patch level on node zzjk01 is [441346801].
[root@zzjk01:0 ~]$ crsctl query crs softwarepatch zzjk02
Oracle Clusterware patch level on node zzjk02 is [1487907289].

■按照网搜包括oracle官方的方法,如下操作也不行
crsctl stop rollingpatch

■如下试过也不行【这个是乱试哈】
/u01/app/12.2.0/grid/crs/install/rootcrs.sh -prepatch
/u01/app/oracle/product/12.2.0/db_1/rdbms/install/rootadd_rdbms.sh
/u01/app/12.2.0/grid/crs/install/rootcrs.sh -postpatch

■■决定回滚重打补丁,如下操作
export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
opatchauto rollback /u01/soft/patch/31305339

■回滚后patchlevel仍然不一致,以下多个途径验证
[root@zzjk01:0 ~]$ crsctl query crs softwarepatch zzjk01
Oracle Clusterware patch level on node zzjk01 is [724960844].
[root@zzjk01:0 ~]$ crsctl query crs softwarepatch zzjk02
Oracle Clusterware patch level on node zzjk02 is [3203133993].

[root@zzjk01:0 ~]$ kfod op=patches
29401763
29517242
29517247
29585399
[root@zzjk02:0 ~]$ kfod op=patches
29401763
29517242
29517247
29585399
31281355
31304218

[grid@zzjk01:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)

[grid@zzjk02:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)

[oracle@zzjk01:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)

[oracle@zzjk02:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)

■按照之前的操作方式,试着rollback节点2的两个补丁
export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
opatchauto rollback /u01/soft/patch/31305339/31281355
opatchauto rollback /u01/soft/patch/31305339/31304218

■顺利回滚后node2的相关信息已经与node1完全一致!如下
[root@zzjk02:1 ~]$ kfod op=patches
29401763
29517242
29517247
29585399
[grid@zzjk02:1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)

[root@zzjk02:1 ~]$ crsctl query crs softwarepatch zzjk01
Oracle Clusterware patch level on node zzjk01 is [724960844].
[root@zzjk02:1 ~]$ crsctl query crs softwarepatch zzjk02
Oracle Clusterware patch level on node zzjk02 is [724960844].

■此时由于手动删除了data磁盘组,导致数据库实例无法启动,报错如下
LRM-00109: could not open parameter file '/u01/app/oracle/product/12.2.0/db_1/dbs/initzzjk1.ora'
导致无法使用dbca删库,鼓捣一个init.ora配置文件临时起个实例,用dbca删干净配置信息。
另一个节点也使用dbca确认是否删掉相关的db配置信息。

最后重新建库时遇见创建数据字典表时报错,原因是前期修改了glogin.sql导致,屏蔽即可。

《一个顽固的ORA-15137处理过程》有3个想法

  1. 再次打补丁31305339到连个节点成功,两个节点输出日志完全一致,如下列出一个节点的输出,留作参考。
    注:节点1用时20分钟,节点2用时25分钟。

    [root@zzjk01 soft]# export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
    [root@zzjk01 soft]# opatchauto apply /u01/soft/patch/31305339

    OPatchauto session is initiated at Thu Oct 15 14:29:46 2020

    System initialization log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-15_02-29-48PM.log.

    Session log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-15_02-30-40PM.log
    The id for this session is KD2A

    Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.2.0/db_1

    Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0/grid
    Patch applicability verified successfully on home /u01/app/oracle/product/12.2.0/db_1

    Patch applicability verified successfully on home /u01/app/12.2.0/grid

    Verifying SQL patch applicability on home /u01/app/oracle/product/12.2.0/db_1
    SQL patch applicability verified successfully on home /u01/app/oracle/product/12.2.0/db_1

    Preparing to bring down database service on home /u01/app/oracle/product/12.2.0/db_1
    Successfully prepared home /u01/app/oracle/product/12.2.0/db_1 to bring down database service

    Bringing down CRS service on home /u01/app/12.2.0/grid
    CRS service brought down successfully on home /u01/app/12.2.0/grid

    Performing prepatch operation on home /u01/app/oracle/product/12.2.0/db_1
    Perpatch operation completed successfully on home /u01/app/oracle/product/12.2.0/db_1

    Start applying binary patch on home /u01/app/oracle/product/12.2.0/db_1
    Binary patch applied successfully on home /u01/app/oracle/product/12.2.0/db_1

    Performing postpatch operation on home /u01/app/oracle/product/12.2.0/db_1
    Postpatch operation completed successfully on home /u01/app/oracle/product/12.2.0/db_1

    Start applying binary patch on home /u01/app/12.2.0/grid
    Binary patch applied successfully on home /u01/app/12.2.0/grid

    Starting CRS service on home /u01/app/12.2.0/grid
    CRS service started successfully on home /u01/app/12.2.0/grid

    Preparing home /u01/app/oracle/product/12.2.0/db_1 after database service restarted
    No step execution required………

    Trying to apply SQL patch on home /u01/app/oracle/product/12.2.0/db_1
    SQL patch applied successfully on home /u01/app/oracle/product/12.2.0/db_1

    OPatchAuto successful.

    ——————————–Summary——————————–

    Patching is completed successfully. Please find the summary as follows:

    Host:zzjk01
    RAC Home:/u01/app/oracle/product/12.2.0/db_1
    Version:19.0.0.0.0
    Summary:

    ==Following patches were SKIPPED:

    Patch: /u01/soft/patch/31305339/31304218
    Reason: This patch is not applicable to this specified target type – “rac_database”

    Patch: /u01/soft/patch/31305339/31335188
    Reason: This patch is not applicable to this specified target type – “rac_database”

    ==Following patches were SUCCESSFULLY applied:

    Patch: /u01/soft/patch/31305339/31281355
    Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-33-29PM_1.log

    Patch: /u01/soft/patch/31305339/31305087
    Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-33-29PM_1.log

    Host:zzjk01
    CRS Home:/u01/app/12.2.0/grid
    Version:19.0.0.0.0
    Summary:

    ==Following patches were SUCCESSFULLY applied:

    Patch: /u01/soft/patch/31305339/31281355
    Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-37-38PM_1.log

    Patch: /u01/soft/patch/31305339/31304218
    Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-37-38PM_1.log

    Patch: /u01/soft/patch/31305339/31305087
    Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-37-38PM_1.log

    Patch: /u01/soft/patch/31305339/31335188
    Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-37-38PM_1.log

    OPatchauto session completed at Thu Oct 15 14:50:04 2020
    Time taken to complete the session 20 minutes, 19 seconds

  2. 节点2输出如下:

    [root@zzjk02 ~]# export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
    [root@zzjk02 ~]# opatchauto apply /u01/soft/patch/31305339

    OPatchauto session is initiated at Thu Oct 15 14:55:10 2020

    System initialization log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-15_02-55-13PM.log.

    Session log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-15_02-56-05PM.log
    The id for this session is EST4

    Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0/grid

    Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.2.0/db_1
    Patch applicability verified successfully on home /u01/app/oracle/product/12.2.0/db_1

    Patch applicability verified successfully on home /u01/app/12.2.0/grid

    Verifying SQL patch applicability on home /u01/app/oracle/product/12.2.0/db_1
    SQL patch applicability verified successfully on home /u01/app/oracle/product/12.2.0/db_1

    Preparing to bring down database service on home /u01/app/oracle/product/12.2.0/db_1
    Successfully prepared home /u01/app/oracle/product/12.2.0/db_1 to bring down database service

    Bringing down CRS service on home /u01/app/12.2.0/grid
    CRS service brought down successfully on home /u01/app/12.2.0/grid

    Performing prepatch operation on home /u01/app/oracle/product/12.2.0/db_1
    Perpatch operation completed successfully on home /u01/app/oracle/product/12.2.0/db_1

    Start applying binary patch on home /u01/app/oracle/product/12.2.0/db_1
    Binary patch applied successfully on home /u01/app/oracle/product/12.2.0/db_1

    Performing postpatch operation on home /u01/app/oracle/product/12.2.0/db_1
    Postpatch operation completed successfully on home /u01/app/oracle/product/12.2.0/db_1

    Start applying binary patch on home /u01/app/12.2.0/grid
    Binary patch applied successfully on home /u01/app/12.2.0/grid

    Starting CRS service on home /u01/app/12.2.0/grid
    CRS service started successfully on home /u01/app/12.2.0/grid

    Preparing home /u01/app/oracle/product/12.2.0/db_1 after database service restarted
    No step execution required………

    Trying to apply SQL patch on home /u01/app/oracle/product/12.2.0/db_1
    SQL patch applied successfully on home /u01/app/oracle/product/12.2.0/db_1

    OPatchAuto successful.

    ——————————–Summary——————————–

    Patching is completed successfully. Please find the summary as follows:

    Host:zzjk02
    RAC Home:/u01/app/oracle/product/12.2.0/db_1
    Version:19.0.0.0.0
    Summary:

    ==Following patches were SKIPPED:

    Patch: /u01/soft/patch/31305339/31304218
    Reason: This patch is not applicable to this specified target type – “rac_database”

    Patch: /u01/soft/patch/31305339/31335188
    Reason: This patch is not applicable to this specified target type – “rac_database”

    ==Following patches were SUCCESSFULLY applied:

    Patch: /u01/soft/patch/31305339/31281355
    Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-59-03PM_1.log

    Patch: /u01/soft/patch/31305339/31305087
    Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-59-03PM_1.log

    Host:zzjk02
    CRS Home:/u01/app/12.2.0/grid
    Version:19.0.0.0.0
    Summary:

    ==Following patches were SUCCESSFULLY applied:

    Patch: /u01/soft/patch/31305339/31281355
    Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_15-04-00PM_1.log

    Patch: /u01/soft/patch/31305339/31304218
    Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_15-04-00PM_1.log

    Patch: /u01/soft/patch/31305339/31305087
    Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_15-04-00PM_1.log

    Patch: /u01/soft/patch/31305339/31335188
    Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_15-04-00PM_1.log

    OPatchauto session completed at Thu Oct 15 15:20:37 2020
    Time taken to complete the session 25 minutes, 27 seconds

  3. 此时查询两节点补丁状态如下,完全一致:
    [root@zzjk01:0 ~]$ crsctl query crs softwarepatch zzjk01
    Oracle Clusterware patch level on node zzjk01 is [441346801].
    [root@zzjk01:0 ~]$ crsctl query crs softwarepatch zzjk02
    Oracle Clusterware patch level on node zzjk02 is [441346801].

    [root@zzjk01:0 ~]$ kfod op=patches
    —————
    List of Patches
    ===============
    31281355
    31304218
    31305087
    31335188

    [grid@zzjk01:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
    31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
    31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
    31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
    31281355;Database Release Update : 19.8.0.0.200714 (31281355)

    OPatch succeeded.

    [oracle@zzjk01:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
    31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
    31281355;Database Release Update : 19.8.0.0.200714 (31281355)

    OPatch succeeded.

    [root@zzjk02:0 /u01/soft/patch/31305339]$ kfod op=patches
    —————
    List of Patches
    ===============
    31281355
    31304218
    31305087
    31335188

    [grid@zzjk02:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
    31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
    31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
    31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
    31281355;Database Release Update : 19.8.0.0.200714 (31281355)

    OPatch succeeded.

    [oracle@zzjk02:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
    31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
    31281355;Database Release Update : 19.8.0.0.200714 (31281355)

    OPatch succeeded.

发表评论

电子邮件地址不会被公开。 必填项已用*标注