1. 首页
  2. Oracle
  3. 正文

linux 6.5 11g RAC 扩展asm磁盘

2017年12月27日 2044点热度 0人点赞 1条评论

linux6.5，3个11gRAC节点。

SQL> !cat /etc/redhat-release
CentOS release 6.5 (Final)

在存储层面要求厂家映射了3个1T的RAID5虚拟磁盘，如何在不重启系统情况下正确识别新加的磁盘？

首先确认存在几个HBA卡：

[root@jkrac3:~]dmesg|grep scsi
megaraid_sas 0000:03:00.0: [scsi0]: FW supports<16> MSIX vector,Online CPUs: <64>,Current MSIX <16>
scsi0 : LSI SAS based MegaRAID driver
scsi 0:0:0:0: Enclosure         LSI 12G  SAS Expander     0402 PQ: 0 ANSI: 5
scsi 0:0:1:0: Direct-Access     LSI      LSI              3.40 PQ: 0 ANSI: 5
scsi 0:2:0:0: Direct-Access     LSI      LSI              3.40 PQ: 0 ANSI: 5
scsi 0:2:1:0: Direct-Access     LSI      LSI              3.40 PQ: 0 ANSI: 5
scsi1 : qla2xxx
scsi 1:0:0:0: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:0:1: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:0:2: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:0:3: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:0:4: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:0:5: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:0:6: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:1:0: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:1:1: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:1:2: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:1:3: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:1:4: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:1:5: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:1:6: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:2:0: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:2:1: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:2:2: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:2:3: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:2:4: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:2:5: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:2:6: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:3:0: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:3:1: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:3:2: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:3:3: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:3:4: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:3:5: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 1:0:3:6: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi2 : qla2xxx
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
scsi6 : ahci
scsi7 : ahci
scsi8 : ahci
scsi 2:0:0:0: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:0:1: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:0:2: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:0:3: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:0:4: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:0:5: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:0:6: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:1:0: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:1:1: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:1:2: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:1:3: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:1:4: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:1:5: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:1:6: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:2:0: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:2:1: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:2:2: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:2:3: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:2:4: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:2:5: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:2:6: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:3:0: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:3:1: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:3:2: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:3:3: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:3:4: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:3:5: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6
scsi 2:0:3:6: Direct-Access     IBM      2145             0000 PQ: 0 ANSI: 6

如上可以看出存在两个scsi host，即scsi 1、scsi 2，对应host1、host2，使用如下命令主动触发扫描host1、2下面的所有channel、target、lun。【3个节点操作】

cd /sys/class/scsi_host/host1
echo '- - -' > scan
cd /sys/class/scsi_host/host2
echo '- - -' > scan
multipath -ll

[root@jkrac3:~]multipath -ll
mpathl (3600507680c80013398000000000000c5) dm-15 IBM,2145
size=1.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 1:0:2:9 sdbo 68:32  active ready running
| |- 2:0:3:9 sdcd 69:16  active ready running
| |- 1:0:3:9 sdbr 68:80  active ready running
| `- 2:0:2:9 sdca 68:224 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:0:9 sdbi 67:192 active ready running
  |- 1:0:1:9 sdbl 67:240 active ready running
  |- 2:0:0:9 sdbu 68:128 active ready running
  `- 2:0:1:9 sdbx 68:176 active ready running
mpathk (3600507680c80013398000000000000c3) dm-14 IBM,2145
size=1.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 1:0:2:7 sdbm 68:0   active ready running
| |- 2:0:3:7 sdcb 68:240 active ready running
| |- 1:0:3:7 sdbp 68:48  active ready running
| `- 2:0:2:7 sdby 68:192 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:0:7 sdbg 67:160 active ready running
  |- 2:0:0:7 sdbs 68:96  active ready running
  |- 1:0:1:7 sdbj 67:208 active ready running
  `- 2:0:1:7 sdbv 68:144 active ready running
mpathj (3600507680c80013398000000000000c4) dm-13 IBM,2145
size=1.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 1:0:0:8 sdbh 67:176 active ready running
| |- 1:0:1:8 sdbk 67:224 active ready running
| |- 2:0:0:8 sdbt 68:112 active ready running
| `- 2:0:1:8 sdbw 68:160 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:2:8 sdbn 68:16  active ready running
  |- 2:0:3:8 sdcc 69:0   active ready running
  |- 2:0:2:8 sdbz 68:208 active ready running
  `- 1:0:3:8 sdbq 68:64  active ready running

可以看到新增的3个lun已经可见了。

此时在/dev/下可见dm-*盘符，在/dev/mapper/下可见mpath*盘符，理论上，配置udev时都可以使用，此处还是使用本人常用的dm-*盘符配置udev吧。

此处需要提一下下udev。由于在操作系统级别的磁盘盘符，在系统重启后可能不确定的，至少在集群配置共享磁盘的多个节点之间盘符是不确定的，这样会导致asm磁盘配置的混乱，此时udev就排上用场了，使用它给对应的共享lun配置名字、属性，使得磁盘在不同节点保持一致。

此时查看3个节点的/dev/dm-*盘符，看见新增的dm-13、dm-14、dm-15，在不同节点实际是不同的磁盘，这个很要命啊，直接使用dm-*盘符配置asm磁盘显然完全不可以，如下。

1号节点：

[root@jkrac1:/etc/udev/rules.d]ll /dev/dm-*
brw-rw---- 1 root disk 253,  0 Dec 12 23:06 /dev/dm-0
brw-rw---- 1 root disk 253,  1 Dec 12 23:06 /dev/dm-1
brw-rw---- 1 root disk 253, 10 Dec 12 23:06 /dev/dm-10
brw-rw---- 1 root disk 253, 11 Dec 12 23:06 /dev/dm-11
brw-rw---- 1 root disk 253, 12 Dec 12 23:06 /dev/dm-12
brw-rw---- 1 root disk 253, 13 Dec 21 10:08 /dev/dm-13
brw-rw---- 1 root disk 253, 14 Dec 21 10:08 /dev/dm-14
brw-rw---- 1 root disk 253, 15 Dec 21 10:08 /dev/dm-15
brw-rw---- 1 root disk 253,  2 Dec 12 23:06 /dev/dm-2
brw-rw---- 1 root disk 253,  3 Dec 12 23:07 /dev/dm-3
brw-rw---- 1 root disk 253,  4 Dec 12 23:07 /dev/dm-4
brw-rw---- 1 root disk 253,  5 Dec 12 23:07 /dev/dm-5
brw-rw---- 1 root disk 253,  6 Dec 12 23:07 /dev/dm-6
brw-rw---- 1 root disk 253,  7 Dec 12 23:07 /dev/dm-7
brw-rw---- 1 root disk 253,  8 Dec 12 23:07 /dev/dm-8
brw-rw---- 1 root disk 253,  9 Dec 12 23:07 /dev/dm-9
[root@jkrac1:/etc/udev/rules.d]/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-13
3600507680c80013398000000000000c4
[root@jkrac1:/etc/udev/rules.d]/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-14
3600507680c80013398000000000000c5
[root@jkrac1:/etc/udev/rules.d]/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-15
3600507680c80013398000000000000c3

2号节点：

[root@jkrac2:/dev]ll dm-*
brw-rw---- 1 root disk 253,  0 Dec 12 22:31 dm-0
brw-rw---- 1 root disk 253,  1 Dec 12 22:31 dm-1
brw-rw---- 1 root disk 253, 10 Dec 21 12:38 dm-10
brw-rw---- 1 root disk 253, 11 Dec 12 22:31 dm-11
brw-rw---- 1 root disk 253, 12 Dec 12 22:31 dm-12
brw-rw---- 1 root disk 253, 13 Dec 21 09:59 dm-13
brw-rw---- 1 root disk 253, 14 Dec 21 09:59 dm-14
brw-rw---- 1 root disk 253, 15 Dec 21 09:59 dm-15
brw-rw---- 1 root disk 253,  2 Dec 21 09:54 dm-2
brw-rw---- 1 root disk 253,  3 Dec 21 10:11 dm-3
brw-rw---- 1 root disk 253,  6 Dec 21 10:11 dm-6
brw-rw---- 1 root disk 253,  7 Dec 21 09:56 dm-7
brw-rw---- 1 root disk 253,  8 Dec 21 09:56 dm-8
brw-rw---- 1 root disk 253,  9 Dec 21 09:54 dm-9
[root@jkrac2:/dev]/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-13
3600507680c80013398000000000000c5
[root@jkrac2:/dev]/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-14
3600507680c80013398000000000000c3
[root@jkrac2:/dev]/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-15
3600507680c80013398000000000000c4

3号节点：

[root@jkrac3:/dev]ll dm-*
brw-rw---- 1 root disk 253,  0 Dec 13 10:13 dm-0
brw-rw---- 1 root disk 253,  1 Dec 13 10:13 dm-1
brw-rw---- 1 root disk 253, 10 Dec 13 10:13 dm-10
brw-rw---- 1 root disk 253, 11 Dec 13 10:13 dm-11
brw-rw---- 1 root disk 253, 12 Dec 13 15:15 dm-12
brw-rw---- 1 root disk 253, 13 Dec 21 10:08 dm-13
brw-rw---- 1 root disk 253, 14 Dec 21 10:08 dm-14
brw-rw---- 1 root disk 253, 15 Dec 21 10:08 dm-15
brw-rw---- 1 root disk 253,  2 Dec 13 10:13 dm-2
brw-rw---- 1 root disk 253,  4 Dec 13 16:51 dm-4
brw-rw---- 1 root disk 253,  5 Dec 13 10:59 dm-5
brw-rw---- 1 root disk 253,  6 Dec 13 11:14 dm-6
brw-rw---- 1 root disk 253,  7 Dec 13 11:14 dm-7
brw-rw---- 1 root disk 253,  8 Dec 13 10:59 dm-8
[root@jkrac3:/dev]/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-13
3600507680c80013398000000000000c4
[root@jkrac3:/dev]/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-14
3600507680c80013398000000000000c3
[root@jkrac3:/dev]/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-15
3600507680c80013398000000000000c5

盘符不同，但是唯有一个scsi id可以唯一标识一块磁盘，即如上所示的“3600507680c80013398000000000000c3”标识，此处使用udev的设备定义规则，定义对应的asm磁盘，如下修改udev规则配置文件/etc/udev/rules.d/。【3个节点操作】

[root@jkrac1:/etc/udev/rules.d]vi 99-oracle-asmdevices.rules
KERNEL=="dm-*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$n
ame", RESULT=="3600507680c80013398000000000000c3", NAME="asm5", OWNER="grid", GROUP="asmadmin", MODE="0660"

如上我新增了一个asm5盘符，用于asm磁盘配置，使得新增的设备名字及属性生效，如下所示：【3个节点操作】

[root@jkrac1:/dev]ll asm?
brw-rw---- 1 grid asmadmin 253, 5 Dec 27 21:14 asm1
brw-rw---- 1 grid asmadmin 253, 8 Dec 27 18:56 asm2
brw-rw---- 1 grid asmadmin 253, 9 Dec 27 21:14 asm3
brw-rw---- 1 grid asmadmin 253, 7 Dec 27 18:56 asm4
[root@jkrac1:/dev]start_udev
Starting udev:                                             [  OK  ]
[root@jkrac1:/dev]ll asm?
brw-rw---- 1 grid asmadmin 253,  5 Dec 27 21:15 asm1
brw-rw---- 1 grid asmadmin 253,  8 Dec 27 21:15 asm2
brw-rw---- 1 grid asmadmin 253,  9 Dec 27 21:15 asm3
brw-rw---- 1 grid asmadmin 253,  7 Dec 27 21:15 asm4
brw-rw---- 1 grid asmadmin 253, 15 Dec 27 21:15 asm5

接下来可以在grid用户下修改asm磁盘组，将asm5磁盘增加到fra磁盘组中。【1个节点操作】

[grid@jkrac1:~]rlwrap sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Wed Dec 27 21:22:00 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup fra add disk '/dev/asm5';

Diskgroup altered.

SQL> select name from v$asm_disk;

NAME
------------------------------------------------------------
DATA_0001
FRA_0000
FRA_0001
OCR_0002
DATA_0000
OCR_0001
OCR_0000
FRA_0002

8 rows selected.

此时对应的fra磁盘组已经扩容了，由于本次操作的目的是确保归档空间，需要修改数据库参数到合适的大小，如下：【1个节点操作】

SYS@jkdb1> show parameter recovery

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      +FRA
db_recovery_file_dest_size           big integer 1600G
recovery_parallelism                 integer     0
SYS@jkdb1> alter system set db_recovery_file_dest_size=2600G scope=both sid='*';

System altered.

SYS@jkdb1> show parameter recovery

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      +FRA
db_recovery_file_dest_size           big integer 2600G
recovery_parallelism                 integer     0

写作此文，供同事参考。

标签：
最后更新：2017年12月27日

liking

这个人很懒，什么都没留下

文章评论

  • liking

    补充，FRA_0002对应新增的asm5磁盘。

    2017年12月27日
    回复

    • COPYRIGHT © 2024 Liking's Blog. ALL RIGHTS RESERVED.

    Theme Kratos Made By Seaton Jiang