Watching the “CopyBack” progress of a new disk on an Exadata compute node

This is just a very small post on how to watch the progress of the “CopyBack” state of a freshly inserted disk in an Exadata “Computing” (database) node. A disk failed in the (LSI Hardware) RAID5 set, and the hotspare disk was automatically used. The failed disk was replaced, and we are now awaiting the intermediate “CopyBack” phase.

The current state of the disks is visible using the following command:

# /opt/MegaRAID/MegaCli/MegaCli64 -pdlist -a0 | grep -iE "slot|firmware"
Slot Number: 0Firmware state: Copyback
Device Firmware Level: 0D70
Slot Number: 1
Firmware state: Online, Spun Up
Device Firmware Level: 0B70
Slot Number: 2
Firmware state: Online, Spun Up
Device Firmware Level: 0B70
Slot Number: 3
Firmware state: Online, Spun Up
Device Firmware Level: 0B70

But what is the progress? Some googling came up with (almost) the answer (http://trac.camsentry.com/wordpress/tag/ldinfo/), I modified it a tiny bit to make it refresh:

while $(true); do /opt/MegaRAID/MegaCli/MegaCli64 adpeventlog getlatest 200 -f ~/adpeventlog.txt a0; awk '/^Time/{TIME=$0};/Seconds/{SECS=$5}/^Event Desc/{printf("%25.25s %5.5s %s\n",TIME,SECS,$0);TIME=" ";SECS=""}' ~/adpeventlog.txt|grep -v fan|tac; sleep 5; done

This wil refresh every 5 seconds and show the progress of the CopyBack, and the state changes after it once it gets to that.

About these ads
5 comments
  1. Freek said:

    On X3-2 it seems that the compute nodes don’t have a hot spare anymore:

    X3-2:
    ====

    [root@XXXXXXX ~]# /opt/MegaRAID/MegaCli/MegaCli64 -pdlist -a0 | grep -iE “slot|firmware”
    Slot Number: 0
    Firmware state: Online, Spun Up
    Device Firmware Level: A2B0
    Slot Number: 1
    Firmware state: Online, Spun Up
    Device Firmware Level: A2B0
    Slot Number: 2
    Firmware state: Online, Spun Up
    Device Firmware Level: A2B0
    Slot Number: 3
    Firmware state: Online, Spun Up
    Device Firmware Level: A2B0

    X2-2
    ====

    [root@XXXXXXXXX ~]# /opt/MegaRAID/MegaCli/MegaCli64 -pdlist -a0 | grep -iE “slot|firmware”
    Slot Number: 0
    Firmware state: Online, Spun Up
    Device Firmware Level: A2B0
    Slot Number: 1
    Firmware state: Online, Spun Up
    Device Firmware Level: A2B0
    Slot Number: 2
    Firmware state: Online, Spun Up
    Device Firmware Level: A2B0
    Slot Number: 3
    Firmware state: Hotspare, Spun down
    Device Firmware Level: A2B0

    • Hi Freek,

      This is known and it started from version 11.2.3.2.0. You can check known issue#5 hotspare removed for compute nodes’ for the same.

      However, Frits already mentioned the same in his blog.

      Regards,
      Sunil Bhola

  2. Jee said:

    Good Stuff. Also couple of reasons for not completing resync after systeem disk replaced, mentioned in 1279288.1 1328727.1 1316829.1.

  3. Martin Bach said:

    Frits, there’s a nifty tool out there that does something similar, named watch. watch -n 5 would have achieved similar to what you coded. $ man watch for all the detail, it’s pretty cool

  4. Howdy just wanted to give you a quick heads
    up. The words in your content seem to be running off the screen in Opera.
    I’m not sure if this is a formatting issue or something to do with internet browser compatibility but I thought I’d
    post to let you know. The style and design look great though!
    Hope you get the issue fixed soon. Kudos

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 2,136 other followers

%d bloggers like this: