Playing with ASM disks on linux, part 1

ASM (Oracle’s Automatic Storage Management) uses disk devices in Linux.
These devices are typically native SCSI devices (/dev/sd*), powerpath (/dev/emcpower*), etc.

The devices are used as raw devices (formerly using the /dev/raw/raw* meta-devices, nowadays directly on the devices itself), or as ASMlib devices.

What makes a device an ASMlib device, instead of a raw device?

Let’s create a diskgroup which uses a raw device on /dev/sdb1:


# chown oracle /dev/sdb1
# su - oracle
$ export ORACLE_HOME=/oracle/db/11.1.0.6
$ export ORACLE_SID=+ASM
$ /oracle/db/11.1.0.6/bin/sqlplus / as sysasm

SQL*Plus: Release 11.1.0.6.0 - Production on Tue Feb 12 15:22:52 2008

Copyright (c) 1982, 2007, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> alter system set asm_diskstring = '/dev/sd*' scope=both;

System altered.

SQL> col path format a30
SQL> select mount_status, header_status, state, path from v$asm_disk;
MOUNT_S HEADER_STATU STATE PATH
------- ------------ -------- ------------------------------
CLOSED CANDIDATE NORMAL /dev/sdb1

SQL> SQL> create diskgroup rawtest external redundancy disk '/dev/sdb1';

Diskgroup created.

State of the disk is altered in ASM:


SQL> select mount_status, header_status, state, path from v$asm_disk;

MOUNT_S HEADER_STATU STATE PATH
------- ------------ -------- ------------------------------
CACHED MEMBER NORMAL /dev/sdb1

Because this disk now belongs to a diskgroup, and has gotten a name “RAWTEST_0000”:


select g.name, d.name from v$asm_diskgroup g, v$asm_disk d where d.group_number=g.group_number;

NAME NAME
------------------------------ ------------------------------
RAWTEST RAWTEST_0000

How does this visualise at the operating system level?


$ od -a -x -A x /dev/sdb1 | less

000000 soh stx soh soh nul nul nul nul nul nul nul nul u # stx D
8201 0101 0000 0000 0000 8000 23f5 4482
000010 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
0000 0000 0000 0000 0000 0000 0000 0000
000020 O R C L D I S K nul nul nul nul nul nul nul nul
524f 4c43 4944 4b53 0000 0000 0000 0000
000030 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
0000 0000 0000 0000 0000 0000 0000 0000
000040 nul nul dle nl nul nul soh etx R A W T E S T _
0000 0a10 0000 0301 4152 5457 5345 5f54
000050 0 0 0 0 nul nul nul nul nul nul nul nul nul nul nul nul
3030 3030 0000 0000 0000 0000 0000 0000
000060 nul nul nul nul nul nul nul nul R A W T E S T nul
0000 0000 0000 0000 4152 5457 5345 0054
000070 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
0000 0000 0000 0000 0000 0000 0000 0000
000080 nul nul nul nul nul nul nul nul R A W T E S T _
0000 0000 0000 0000 4152 5457 5345 5f54
000090 0 0 0 0 nul nul nul nul nul nul nul nul nul nul nul nul
3030 3030 0000 0000 0000 0000 0000 0000
0000a0 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
0000 0000 0000 0000 0000 0000 0000 0000
*

Well, what do we see?
– ORCLDISK at offset 0x20
This means it’s an ASM disk.
– RAWTEST_0000 at offset 0x48
This is the disk
– RAWTEST at offset 0x68
This is the diskgroup name.
– RAWTEST_0000 at offset 0x88
This is the failgroup name (not used with external redundancy; nowhere to fail to)

Now let’s drop the diskgroup:


SQL> drop diskgroup rawtest;

Diskgroup dropped.

Now the diskgroup is dropped, but the disk still exists:

SQL> select group_number, mount_status, header_status, mode_status, state, name from v$asm_disk;

GROUP_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE
------------ ------- ------------ ------- --------
NAME
------------------------------
0 CLOSED FORMER ONLINE NORMAL

It’s header status is ‘FORMER’. This obviously means it used to be part of a diskgroup. But that’s not all:

When looking at the disk again using the ‘od’ utility (“octal dump”) it shows the diskheader is not updated at all.
This is probably how the alter diskgroup x undrop disk y features is implemented: the dictionary of the ASM instance has recorded the dropping of the disk, but the disk itself is untouched.

What difference makes an ASMlib disk?

First, make the disk disappear in ASM:

# chown root /dev/sdb1

Check in the ASM instance:

SQL> select * from v$asm_disk;

no rows selected

Now mark the disk as ASMlib:


# /etc/init.d/oracleasm createdisk SDB1 /dev/sdb1
Marking disk "/dev/sdb1" as an ASM disk: asmtool: Device "/dev/sdb1" is already labeled for ASM disk ""
[FAILED]

As I’ve written above, the diskheader is touched. As we see, the oracleasm utility (actually the asmtool command) checks the header and sees a valid ASM disk. But the oracleasm utility has some undocumented commands which let us do that:


# /etc/init.d/oracleasm force-renamedisk /dev/sdb1 SDB1
Renaming disk "/dev/sdb1" to "SDB1": [ OK ]

That should do the trick. Lets see if it did:

# /etc/init.d/oracleasm listdisks
SDB1

Let’s see what difference an ASMlib disk makes using the ‘od’ utility again:


od -a -x -A x /dev/sdb1 | less

000000 soh stx soh soh nul nul nul nul nul nul nul nul % C Z >
8201 0101 0000 0000 0000 8000 c3a5 be5a
000010 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
0000 0000 0000 0000 0000 0000 0000 0000
000020 O R C L D I S K S D B 1 nul nul nul nul
524f 4c43 4944 4b53 4453 3142 0000 0000
000030 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
0000 0000 0000 0000 0000 0000 0000 0000
000040 nul nul dle nl nul nul soh eot R A W T E S T _
0000 0a10 0000 0401 4152 5457 5345 5f54
000050 0 0 0 0 nul nul nul nul nul nul nul nul nul nul nul nul
3030 3030 0000 0000 0000 0000 0000 0000
000060 nul nul nul nul nul nul nul nul R A W T E S T nul
0000 0000 0000 0000 4152 5457 5345 0054
000070 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
0000 0000 0000 0000 0000 0000 0000 0000
000080 nul nul nul nul nul nul nul nul R A W T E S T _
0000 0000 0000 0000 4152 5457 5345 5f54
000090 0 0 0 0 nul nul nul nul nul nul nul nul nul nul nul nul
3030 3030 0000 0000 0000 0000 0000 0000
0000a0 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
0000 0000 0000 0000 0000 0000 0000 0000
*

Well, an ASMlib disk looks (not very surprisingly) quite much like a normal ASM disk.
We got the usual stuff at 0x20 (ORCLDISK; the ASM disk marker), 0x48 (RAWTEST_0000, the diskname), 0x68 (RAWTEST, the diskgroup). 0x88 (RAWTEST_000, the failgroup) Only visual difference is the ASMlib disk name at 0x28: SDB1!

How does that look like in the ASM instance:

SQL> alter system set asm_diskstring = '' scope=both;
SQL> col path format a30
SQL> select mount_status, header_status, state, path from v$asm_disk;
MOUNT_S HEADER_STATU STATE PATH
------- ------------ -------- ------------------------------
CLOSED PROVISIONED NORMAL
ORCL:SDB1

That’s it for this part. Next part we’ll investigate somewhat more of the inner working of both ASMlib and ASM.

Advertisements
1 comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: