一个顽固的ORA-15137处理过程

2020年10月14日 2252点热度 0人点赞 3条评论

19C的RAC两个节点patch level不一致，导致无法操作asm的diskgroup，包括增加、删除dg时报错如下：
ORA-15039: diskgroup not dropped
ORA-15137: The ASM cluster is in rolling patch state.

■查看软件版本
crsctl query crs softwareversion
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [19.0.0.0.0]

■查看patch level
[root@zzjk01:0 ~]$ crsctl query crs softwarepatch zzjk01
Oracle Clusterware patch level on node zzjk01 is [441346801].
[root@zzjk01:0 ~]$ crsctl query crs softwarepatch zzjk02
Oracle Clusterware patch level on node zzjk02 is [1487907289].

■按照网搜包括oracle官方的方法，如下操作也不行
crsctl stop rollingpatch

■如下试过也不行【这个是乱试哈】
/u01/app/12.2.0/grid/crs/install/rootcrs.sh -prepatch
/u01/app/oracle/product/12.2.0/db_1/rdbms/install/rootadd_rdbms.sh
/u01/app/12.2.0/grid/crs/install/rootcrs.sh -postpatch

■■决定回滚重打补丁，如下操作
export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
opatchauto rollback /u01/soft/patch/31305339

■回滚后patchlevel仍然不一致，以下多个途径验证
[root@zzjk01:0 ~]$ crsctl query crs softwarepatch zzjk01
Oracle Clusterware patch level on node zzjk01 is [724960844].
[root@zzjk01:0 ~]$ crsctl query crs softwarepatch zzjk02
Oracle Clusterware patch level on node zzjk02 is [3203133993].

[root@zzjk01:0 ~]$ kfod op=patches
29401763
29517242
29517247
29585399
[root@zzjk02:0 ~]$ kfod op=patches
29401763
29517242
29517247
29585399
31281355
31304218

[grid@zzjk01:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)

[grid@zzjk02:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)

[oracle@zzjk01:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)

[oracle@zzjk02:0 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)

■按照之前的操作方式，试着rollback节点2的两个补丁
export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
opatchauto rollback /u01/soft/patch/31305339/31281355
opatchauto rollback /u01/soft/patch/31305339/31304218

■顺利回滚后node2的相关信息已经与node1完全一致！如下
[root@zzjk02:1 ~]$ kfod op=patches
29401763
29517242
29517247
29585399
[grid@zzjk02:1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)

[root@zzjk02:1 ~]$ crsctl query crs softwarepatch zzjk01
Oracle Clusterware patch level on node zzjk01 is [724960844].
[root@zzjk02:1 ~]$ crsctl query crs softwarepatch zzjk02
Oracle Clusterware patch level on node zzjk02 is [724960844].

■此时由于手动删除了data磁盘组，导致数据库实例无法启动，报错如下
LRM-00109: could not open parameter file '/u01/app/oracle/product/12.2.0/db_1/dbs/initzzjk1.ora'
导致无法使用dbca删库，鼓捣一个init.ora配置文件临时起个实例，用dbca删干净配置信息。
另一个节点也使用dbca确认是否删掉相关的db配置信息。

最后重新建库时遇见创建数据字典表时报错，原因是前期修改了glogin.sql导致，屏蔽即可。

文章评论

liking

再次打补丁31305339到连个节点成功，两个节点输出日志完全一致，如下列出一个节点的输出，留作参考。
注：节点1用时20分钟，节点2用时25分钟。

[root@zzjk01 soft]# export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
[root@zzjk01 soft]# opatchauto apply /u01/soft/patch/31305339

OPatchauto session is initiated at Thu Oct 15 14:29:46 2020

System initialization log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-15_02-29-48PM.log.

Session log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-15_02-30-40PM.log
The id for this session is KD2A

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.2.0/db_1

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0/grid
Patch applicability verified successfully on home /u01/app/oracle/product/12.2.0/db_1

Patch applicability verified successfully on home /u01/app/12.2.0/grid

Verifying SQL patch applicability on home /u01/app/oracle/product/12.2.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/12.2.0/db_1

Preparing to bring down database service on home /u01/app/oracle/product/12.2.0/db_1
Successfully prepared home /u01/app/oracle/product/12.2.0/db_1 to bring down database service

Bringing down CRS service on home /u01/app/12.2.0/grid
CRS service brought down successfully on home /u01/app/12.2.0/grid

Performing prepatch operation on home /u01/app/oracle/product/12.2.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/12.2.0/db_1

Start applying binary patch on home /u01/app/oracle/product/12.2.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/12.2.0/db_1

Performing postpatch operation on home /u01/app/oracle/product/12.2.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/12.2.0/db_1

Start applying binary patch on home /u01/app/12.2.0/grid
Binary patch applied successfully on home /u01/app/12.2.0/grid

Starting CRS service on home /u01/app/12.2.0/grid
CRS service started successfully on home /u01/app/12.2.0/grid

Preparing home /u01/app/oracle/product/12.2.0/db_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/12.2.0/db_1
SQL patch applied successfully on home /u01/app/oracle/product/12.2.0/db_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:zzjk01
RAC Home:/u01/app/oracle/product/12.2.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/soft/patch/31305339/31304218
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/soft/patch/31305339/31335188
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /u01/soft/patch/31305339/31281355
Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-33-29PM_1.log

Patch: /u01/soft/patch/31305339/31305087
Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-33-29PM_1.log

Host:zzjk01
CRS Home:/u01/app/12.2.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/soft/patch/31305339/31281355
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-37-38PM_1.log

Patch: /u01/soft/patch/31305339/31304218
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-37-38PM_1.log

Patch: /u01/soft/patch/31305339/31305087
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-37-38PM_1.log

Patch: /u01/soft/patch/31305339/31335188
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-15_14-37-38PM_1.log

OPatchauto session completed at Thu Oct 15 14:50:04 2020
Time taken to complete the session 20 minutes, 19 seconds

2020年10月15日

liking

节点2输出如下：

[root@zzjk02 ~]# export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
[root@zzjk02 ~]# opatchauto apply /u01/soft/patch/31305339

OPatchauto session is initiated at Thu Oct 15 14:55:10 2020

System initialization log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-15_02-55-13PM.log.

Session log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-15_02-56-05PM.log
The id for this session is EST4

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.2.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/12.2.0/db_1