19c用opatchauto打19.9补丁后无法操作ASM，操作磁盘组时，提示错误：
ORA-15137: The ASM cluster is in rolling patch state
以下为一次耗时几天的操作记录，留作参考。
■grid/oracle用户lspatches，完全一致
[grid@wydb01:1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31780966;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31780966)
31773437;ACFS RELEASE UPDATE 19.9.0.0.0 (31773437)
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)
[oracle@wydb01:1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)
■crs softwarepatch，不一致
crsctl query crs softwarepatch wydb01/wydb02
Oracle Clusterware patch level on node wydb01 is [376483838].
Oracle Clusterware patch level on node wydb02 is [2640357783].
■kfod op=patches，不一致
[grid@wydb01:1 ~]$ kfod op=patches
List of Patches
31771877
31772784
31773437
31780966
[grid@wydb02:1 ~]$ kfod op=patches
List of Patches
31281355
31771877
31772784
31773437
31780966
■在19c打补丁数次出现类似如上问题，导致无法操作ASM磁盘，说明是19c补丁或补丁操作脚本本身有问题。
■看上去节点1缺少补丁 31281355, 因此单独打这个补丁：
export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
opatchauto apply /u01/soft/patch/31305339/31281355
提示成功，但是31281355 skipped！
■既然不需要这个 31281355 ，那为什么节点2有呢？干脆在节点2卸载这个补丁：
export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
opatchauto rollback /u01/soft/patch/31305339/31281355
提示成功，但是31281355 skipped，问题来了，节点2并没有这个 31281355 ！
■再次直接在节点2手动卸载31281355
/u01/app/12.2.0/grid/OPatch/opatchauto rollback -id 31281355
同时卸载另三个补丁31281355,31304218,31305087,31335188
/u01/app/12.2.0/grid/OPatch/opatchauto rollback -id 31281355,31304218,31305087,31335188
还是不行。
■怀疑当初打19.9补丁时节点1出的问题导致，卸载试一试！
/u01/app/12.2.0/grid/OPatch/opatchauto rollback /u01/soft/patch/31750108
[Dec 16, 2020 8:57:25 PM] [INFO] Following active executables are not used by opatch process :
Following active executables are used by opatch process :
/u01/app/oracle/product/12.2.0/db_1/lib/libclntsh.so.19.1
/u01/app/oracle/product/12.2.0/db_1/lib/libsqlplus.so
[Dec 16, 2020 8:57:25 PM] [INFO] Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Following active executables are not used by opatch process :
Following active executables are used by opatch process :
/u01/app/oracle/product/12.2.0/db_1/lib/libclntsh.so.19.1
/u01/app/oracle/product/12.2.0/db_1/lib/libsqlplus.so
[Dec 16, 2020 8:57:25 PM] [SEVERE] OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
这个错误是因为有sqlplus之类的进程在占用相关欲patch的资源，kill掉相关进程再次执行rollback，输出如下：
--------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:wydb01 RAC Home:/u01/app/oracle/product/12.2.0/db_1 Version:19.0.0.0.0 Summary: ==Following patches were SKIPPED: Patch: /u01/soft/patch/31750108/31772784 Reason: Patch /u01/soft/patch/31750108/31772784 is not applied as part of bundle patch 31750108 Patch: /u01/soft/patch/31750108/31773437 Reason: This patch is not applicable to this specified target type - "rac_database" Patch: /u01/soft/patch/31750108/31780966 Reason: This patch is not applicable to this specified target type - "rac_database" ==Following patches were SUCCESSFULLY rolled back: Patch: /u01/soft/patch/31750108/31771877 Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-16_21-17-04PM_1.log Host:wydb01 CRS Home:/u01/app/12.2.0/grid Version:19.0.0.0.0 Summary: ==Following patches were SUCCESSFULLY rolled back: Patch: /u01/soft/patch/31750108/31772784 Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-16_21-22-15PM_1.log Patch: /u01/soft/patch/31750108/31773437 Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-16_21-22-15PM_1.log Patch: /u01/soft/patch/31750108/31780966 Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-16_21-22-15PM_1.log Patch: /u01/soft/patch/31750108/31771877 Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-16_21-22-15PM_1.log OPatchauto session completed at Wed Dec 16 21:37:32 2020 Time taken to complete the session 24 minutes, 32 seconds
■在节点2继续执行rollback，一直报错如下：
==Following patches FAILED in rollback:
Patch: /u01/soft/patch/31750108/31771877
Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-17_11-14-38AM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Re-link fails on target "ioracle".
---
[Dec 17, 2020 11:20:18 AM] [INFO] --------------------------------------------------------------------------------
Failed to run make commands. Please contact Oracle Support.
[Dec 17, 2020 11:20:18 AM] [SEVERE] OUI-67115:OPatch failed to restore OH '/u01/app/oracle/product/12.2.0/db_1'. Consult OPatch document to restore the home manually
before proceeding.
[Dec 17, 2020 11:20:18 AM] [WARNING]OUI-67124:
NRollback was not able to restore the home. Please invoke the following scripts:
- restore.[sh,bat]
- make.txt (Unix only)
to restore the ORACLE_HOME. They are located under
"/u01/app/oracle/product/12.2.0/db_1/.patch_storage/NRollback/2020-12-17_11-15-42AM"
[Dec 17, 2020 11:20:18 AM] [SEVERE] OUI-67073:UtilSession failed: Re-link fails on target "ioracle".
[Dec 17, 2020 11:20:18 AM] [INFO] --------------------------------------------------------------------------------
[Dec 17, 2020 11:20:18 AM] [INFO] The following warnings have occurred during OPatch execution:
[Dec 17, 2020 11:20:18 AM] [INFO] 1) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/12.2.0/db_1"....'/u0
1/app/oracle/product/12.2.0/db_1/lib//libserver19.a(ksl.o): In function `kslges':
ksl.c:(.text+0x41d8): undefined reference to `sslss_async_call_cnt_'
,,,
[Dec 17, 2020 11:20:18 AM] [INFO] 2) OUI-67124:Re-link fails on target "ioracle".
[Dec 17, 2020 11:20:18 AM] [INFO] 3) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/12.2.0/db_1"....'/u0
1/app/oracle/product/12.2.0/db_1/lib//libserver19.a(ksl.o): In function `kslges':
ksl.c:(.text+0x41d8): undefined reference to `sslss_async_call_cnt_'
,,,
[Dec 17, 2020 11:20:18 AM] [INFO] 4) OUI-67124:
NRollback was not able to restore the home. Please invoke the following scripts:
- restore.[sh,bat]
- make.txt (Unix only)
to restore the ORACLE_HOME. They are located under
"/u01/app/oracle/product/12.2.0/db_1/.patch_storage/NRollback/2020-12-17_11-15-42AM"
■这个是中间错误的修复操作也记录一下：
mv /u01/app/oracle/product/12.2.0/db_1/lib/libclntsh.so.19.1 /u01/app/oracle/product/12.2.0/db_1/lib/libclntsh.so.19.1.old mv /u01/app/oracle/product/12.2.0/db_1/lib/libclntshcore.so.19.1 /u01/app/oracle/product/12.2.0/db_1/lib/libclntshcore.so.19.1.old
■后面的操作也是一直报错如下：
==Following patches FAILED in rollback: Patch: /u01/soft/patch/31750108/31771877 Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-17_13-49-51PM_1.log Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Re-link fails on target "ioracle".
■根源在于这一句无法make：
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=$ORACLE_HOME
■耗时一天没有解决，决定节点1打回最新19.9补丁:
/u01/app/12.2.0/grid/OPatch/opatchauto apply /u01/soft/patch/31750108
成功打回19.9
■然后重建节点2：
第一步，根据节点1重建oracle软件部分，重建后无效，两个节点的补丁状态依旧。
第二步，根据节点1重建 grid软件部分，重建后已经可以正常操作ASM。
通过这次节点软件重建，发现grid软件重建要改的东西真是很多！几乎需要检查每一个路径及文件。
注：采用tar包方式可以保留文件属性加参数p即可。
■不幸的是，由于主机硬件方面之前因为测试，出现了多达9块磁盘offline，如下：
dd if=/dev/zero of=/dev/asmdisks/su001_lun07 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su002_lun02 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su002_lun04 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su002_lun10 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su002_lun13 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su003_lun12 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su003_lun13 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su003_lun14 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su002_lvvote bs=1M count=100
■后来虽然正常恢复了vote磁盘组、FRA磁盘组，但是data磁盘组状态已经不正常(exhausted)，无法恢复，估计只能重建磁盘组了，借机也改变一下冗余策略。
文章评论