19c用opatchauto打19.9补丁后无法操作ASM

2020年12月19日 893点热度 0人点赞 0条评论

19c用opatchauto打19.9补丁后无法操作ASM,操作磁盘组时,提示错误:
ORA-15137: The ASM cluster is in rolling patch state

以下为一次耗时几天的操作记录,留作参考。

■grid/oracle用户lspatches,完全一致
[grid@wydb01:1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31780966;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31780966)
31773437;ACFS RELEASE UPDATE 19.9.0.0.0 (31773437)
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)

[oracle@wydb01:1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)

■crs softwarepatch,不一致
crsctl query crs softwarepatch wydb01/wydb02
Oracle Clusterware patch level on node wydb01 is [376483838].
Oracle Clusterware patch level on node wydb02 is [2640357783].

■kfod op=patches,不一致
[grid@wydb01:1 ~]$ kfod op=patches
List of Patches
31771877
31772784
31773437
31780966
[grid@wydb02:1 ~]$ kfod op=patches
List of Patches
31281355
31771877
31772784
31773437
31780966

■在19c打补丁数次出现类似如上问题,导致无法操作ASM磁盘,说明是19c补丁或补丁操作脚本本身有问题。

■看上去节点1缺少补丁 31281355, 因此单独打这个补丁:
export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
opatchauto apply /u01/soft/patch/31305339/31281355
提示成功,但是31281355 skipped!
■既然不需要这个 31281355 ,那为什么节点2有呢?干脆在节点2卸载这个补丁:
export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
opatchauto rollback /u01/soft/patch/31305339/31281355
提示成功,但是31281355 skipped,问题来了,节点2并没有这个 31281355 !
■再次直接在节点2手动卸载31281355
/u01/app/12.2.0/grid/OPatch/opatchauto rollback -id 31281355
同时卸载另三个补丁31281355,31304218,31305087,31335188
/u01/app/12.2.0/grid/OPatch/opatchauto rollback -id 31281355,31304218,31305087,31335188
还是不行。

■怀疑当初打19.9补丁时节点1出的问题导致,卸载试一试!

/u01/app/12.2.0/grid/OPatch/opatchauto rollback /u01/soft/patch/31750108

[Dec 16, 2020 8:57:25 PM] [INFO]    Following active executables are not used by opatch process :

                                    Following active executables are used by opatch process :
                                    /u01/app/oracle/product/12.2.0/db_1/lib/libclntsh.so.19.1
                                    /u01/app/oracle/product/12.2.0/db_1/lib/libsqlplus.so
[Dec 16, 2020 8:57:25 PM] [INFO]    Prerequisite check "CheckActiveFilesAndExecutables" failed.
                                    The details are:

                                    Following active executables are not used by opatch process :

                                    Following active executables are used by opatch process :
                                    /u01/app/oracle/product/12.2.0/db_1/lib/libclntsh.so.19.1
                                    /u01/app/oracle/product/12.2.0/db_1/lib/libsqlplus.so
[Dec 16, 2020 8:57:25 PM] [SEVERE]  OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

这个错误是因为有sqlplus之类的进程在占用相关欲patch的资源,kill掉相关进程再次执行rollback,输出如下:

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:wydb01
RAC Home:/u01/app/oracle/product/12.2.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/soft/patch/31750108/31772784
Reason: Patch /u01/soft/patch/31750108/31772784 is not applied as part of bundle patch 31750108

Patch: /u01/soft/patch/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/soft/patch/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/soft/patch/31750108/31771877
Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-16_21-17-04PM_1.log

Host:wydb01
CRS Home:/u01/app/12.2.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/soft/patch/31750108/31772784
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-16_21-22-15PM_1.log

Patch: /u01/soft/patch/31750108/31773437
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-16_21-22-15PM_1.log

Patch: /u01/soft/patch/31750108/31780966
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-16_21-22-15PM_1.log

Patch: /u01/soft/patch/31750108/31771877
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-16_21-22-15PM_1.log

OPatchauto session completed at Wed Dec 16 21:37:32 2020
Time taken to complete the session 24 minutes, 32 seconds

■在节点2继续执行rollback,一直报错如下:

==Following patches FAILED in rollback:

Patch: /u01/soft/patch/31750108/31771877
Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-17_11-14-38AM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Re-link fails on target "ioracle". 
---
[Dec 17, 2020 11:20:18 AM] [INFO]   --------------------------------------------------------------------------------
                                    Failed to run make commands. Please contact Oracle Support.
[Dec 17, 2020 11:20:18 AM] [SEVERE] OUI-67115:OPatch failed to restore OH '/u01/app/oracle/product/12.2.0/db_1'. Consult OPatch document to restore the home manually
before proceeding.
[Dec 17, 2020 11:20:18 AM] [WARNING]OUI-67124:
                                    NRollback was not able to restore the home.  Please invoke the following scripts:
                                      - restore.[sh,bat]
                                      - make.txt (Unix only)
                                    to restore the ORACLE_HOME.  They are located under
                                    "/u01/app/oracle/product/12.2.0/db_1/.patch_storage/NRollback/2020-12-17_11-15-42AM"
[Dec 17, 2020 11:20:18 AM] [SEVERE] OUI-67073:UtilSession failed: Re-link fails on target "ioracle".
[Dec 17, 2020 11:20:18 AM] [INFO]   --------------------------------------------------------------------------------
[Dec 17, 2020 11:20:18 AM] [INFO]   The following warnings have occurred during OPatch execution:
[Dec 17, 2020 11:20:18 AM] [INFO]   1) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/12.2.0/db_1"....'/u0
1/app/oracle/product/12.2.0/db_1/lib//libserver19.a(ksl.o): In function `kslges':
                                    ksl.c:(.text+0x41d8): undefined reference to `sslss_async_call_cnt_'
,,,
[Dec 17, 2020 11:20:18 AM] [INFO]   2) OUI-67124:Re-link fails on target "ioracle".
[Dec 17, 2020 11:20:18 AM] [INFO]   3) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/12.2.0/db_1"....'/u0
1/app/oracle/product/12.2.0/db_1/lib//libserver19.a(ksl.o): In function `kslges':
                                    ksl.c:(.text+0x41d8): undefined reference to `sslss_async_call_cnt_'
,,,
[Dec 17, 2020 11:20:18 AM] [INFO]   4) OUI-67124:
                                    NRollback was not able to restore the home.  Please invoke the following scripts:
                                      - restore.[sh,bat]
                                      - make.txt (Unix only)
                                    to restore the ORACLE_HOME.  They are located under
                                    "/u01/app/oracle/product/12.2.0/db_1/.patch_storage/NRollback/2020-12-17_11-15-42AM"

■这个是中间错误的修复操作也记录一下:

mv /u01/app/oracle/product/12.2.0/db_1/lib/libclntsh.so.19.1 /u01/app/oracle/product/12.2.0/db_1/lib/libclntsh.so.19.1.old
mv /u01/app/oracle/product/12.2.0/db_1/lib/libclntshcore.so.19.1 /u01/app/oracle/product/12.2.0/db_1/lib/libclntshcore.so.19.1.old

■后面的操作也是一直报错如下:

==Following patches FAILED in rollback:

Patch: /u01/soft/patch/31750108/31771877
Log: /u01/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-17_13-49-51PM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Re-link fails on target "ioracle". 

■根源在于这一句无法make:
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=$ORACLE_HOME

■耗时一天没有解决,决定节点1打回最新19.9补丁:
/u01/app/12.2.0/grid/OPatch/opatchauto apply /u01/soft/patch/31750108
成功打回19.9

■然后重建节点2:
第一步,根据节点1重建oracle软件部分,重建后无效,两个节点的补丁状态依旧。
第二步,根据节点1重建 grid软件部分,重建后已经可以正常操作ASM。
通过这次节点软件重建,发现grid软件重建要改的东西真是很多!几乎需要检查每一个路径及文件。
注:采用tar包方式可以保留文件属性加参数p即可。

■不幸的是,由于主机硬件方面之前因为测试,出现了多达9块磁盘offline,如下:
dd if=/dev/zero of=/dev/asmdisks/su001_lun07 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su002_lun02 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su002_lun04 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su002_lun10 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su002_lun13 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su003_lun12 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su003_lun13 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su003_lun14 bs=1M count=100
dd if=/dev/zero of=/dev/asmdisks/su002_lvvote bs=1M count=100

■后来虽然正常恢复了vote磁盘组、FRA磁盘组,但是data磁盘组状态已经不正常(exhausted),无法恢复,估计只能重建磁盘组了,借机也改变一下冗余策略。

liking

我是雪人

文章评论