How to monitor hardware RAID in Linux (PSBM or Xen)

In this KB several utilities are introduced that can be used to monitor the status of RAID from within OS.

  1. Login to the server by SSH or console and run the following command

    # lspci | grep SAS
    02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

    The ouput shows the model of RAID which on this case is LSI SAS2008. As far as internal servers are concerned the M600 and M605 use SAS1068E which is SAS standard. C6100 servers use SAS2008 which is SAS2 standard.

  2. For SAS1068E we use a utility called “lsiutil” . It can be downloaded from here:
    For Xen server we use the 32 bit version called “lsiutil” and for PSBM we use the 64 bit version called “lsiutil.x86_64”
    Assuming that is downloaded to /root folder.

    # chmod a+x lsiutil   or  chmod 755 lsiutil

  3. Run the utility by following command

    # ./lsiutil  or ./lsiutil.x86_64

    LSI Logic MPT Configuration Utility, Version 1.63, June 4, 2009

    1 MPT Port found

    Port Name         Chip Vendor/Type/Rev    MPT Rev  Firmware Rev  IOC
    1.  /proc/mpt/ioc0    LSI Logic SAS1068E B3     105      00192f00     0

  4. When you are peompted with “Select a device:  [1-1 or 0 to quit]” type 1 which is the Port number that we obtained in previous step. The following menu will be shown.

     1.  Identify firmware, BIOS, and/or FCode
    2.  Download firmware (update the FLASH)
    4.  Download/erase BIOS and/or FCode (update the FLASH)
    8.  Scan for devices
    10.  Change IOC settings (interrupt coalescing)
    13.  Change SAS IO Unit settings
    16.  Display attached devices
    20.  Diagnostics
    21.  RAID actions
    22.  Reset bus
    23.  Reset target
    42.  Display operating system names for devices
    45.  Concatenate SAS firmware and NVDATA files
    59.  Dump PCI config space
    60.  Show non-default settings
    61.  Restore default settings
    66.  Show SAS discovery errors
    69.  Show board manufacturing information
    97.  Reset SAS link, HARD RESET
    98.  Reset SAS link
    99.  Reset port
    e   Enable expert mode in menus
    p   Enable paged mode
    w   Enable logging

  5. When prompted with “Main menu, select an option:  [1-99 or e/p/w or 0 to quit]” select 21 which is “RAID actions” in the menu shown in last step.

     1.  Show volumes
    2.  Show physical disks
    3.  Get volume state
    4.  Wait for volume resync to complete
    23.  Replace physical disk
    26.  Disable drive firmware update mode
    27.  Enable drive firmware update mode
    30.  Create volume
    31.  Delete volume
    32.  Change volume settings
    33.  Change volume name
    50.  Create hot spare
    51.  Delete hot spare
    99.  Reset port
    e   Enable expert mode in menus
    p   Enable paged mode
    w   Enable logging

  6. When propmted with “RAID actions menu, select an option:  [1-99 or e/p/w or 0 to quit] ”
    select 1 to show the status of volume and select 2 to view the status of physical disks. If you select 2 the output is like the following:

    1 volume is active, 2 physical disks are active

    PhysDisk 0 is Bus 0 Target 1
    PhysDisk State:  online
    PhysDisk Size 70007 MB, Inquiry Data:  FUJITSU  MBB2073RC        D407
    Warning, page length in reply does not match page length in buffer
    (9 vs. 15)

    PhysDisk 1 is Bus 0 Target 9
    PhysDisk State:  online
    PhysDisk Size 70007 MB, Inquiry Data:  FUJITSU  MBB2073RC        D407
    Warning, page length in reply does not match page length in buffer
    (9 vs. 15)
    RAID actions menu, select an option:  [1-99 or e/p/w or 0 to quit]

  7. For SAS2008 we use sas2ircu utility. After downloading and extracting proper version

    # chmod a+x sas2ircu or chmod 755 sas2ircu

  8. First we obtain the list of RAID controllers by running the following command:

    # ./sas2ircu list

    LSI Corporation SAS2 IR Configuration Utility.
    Version 15.00.00.00 (2012.11.08)
    Copyright (c) 2009-2012 LSI Corporation. All rights reserved.

    Adapter      Vendor  Device                       SubSys  SubSys
    Index    Type          ID      ID    Pci Address          Ven ID  Dev ID
    —–  ————  ——  ——  —————–    ——  ——
    0     SAS2008     1000h    72h   00h:02h:00h:00h      1170h   6019h
    SAS2IRCU: Utility Completed Successfully.

  9. Use the index number obtained in previous step to display the status of that controller

    # ./sas2ircu 0 display

    LSI Corporation SAS2 IR Configuration Utility.
    Version 15.00.00.00 (2012.11.08)
    Copyright (c) 2009-2012 LSI Corporation. All rights reserved.

    Read configuration has been initiated for controller 0
    ————————————————————————
    Controller information
    ————————————————————————
    Controller type                         : SAS2008
    BIOS version                            : 7.07.00.00
    Firmware version                        : 6.00.00.00
    Channel description                     : 1 Serial Attached SCSI
    Initiator ID                            : 0
    Maximum physical devices                : 125
    Concurrent commands supported           : 2607
    Slot                                    : Unknown
    Segment                                 : 0
    Bus                                     : 2
    Device                                  : 0
    Function                                : 0
    RAID Support                            : Yes
    ————————————————————————
    IR Volume information
    ————————————————————————
    IR volume 1
    Volume ID                               : 172
    Status of volume                        : Okay (OKY)
    Volume wwid                             : 02a87be379b1217d
    RAID level                              : RAID1
    Size (in MB)                            : 237464
    Physical hard disks                     :
    PHY[0] Enclosure#/Slot#                 : 1:0
    PHY[1] Enclosure#/Slot#                 : 1:1
    ————————————————————————
    Physical device information
    ————————————————————————
    Initiator at ID #0

    Device is a Hard disk
    Enclosure #                             : 1
    Slot #                                  : 0
    SAS Address                             : 4433221-1-0000-0000
    State                                   : Optimal (OPT)
    Size (in MB)/(in sectors)               : 238475/488397167
    Manufacturer                            : ATA
    Model Number                            : GB0250EAFYK
    Firmware Revision                       : HPG1
    Serial No                               : WCAT1E165748
    GUID                                    : 50014ee157685c2e
    Protocol                                : SATA
    Drive Type                              : SATA_HDD

    Device is a Hard disk
    Enclosure #                             : 1
    Slot #                                  : 1
    SAS Address                             : 4433221-1-0100-0000
    State                                   : Optimal (OPT)
    Size (in MB)/(in sectors)               : 238475/488397167
    Manufacturer                            : ATA
    Model Number                            : GB0250EAFYK
    Firmware Revision                       : HPG1
    Serial No                               : WCAT1E256749
    GUID                                    : 50014ee102151ae5
    Protocol                                : SATA
    Drive Type                              : SATA_HDD
    ————————————————————————
    Enclosure information
    ————————————————————————
    Enclosure#                              : 1
    Logical ID                              : 500a0d10:00520c60
    Numslots                                : 8
    StartSlot                               : 0
    ————————————————————————
    SAS2IRCU: Command DISPLAY Completed Successfully.
    SAS2IRCU: Utility Completed Successfully.

  10. Dell PE R410 server uses MegaRAID

    # lspci | grep SAS

    RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)

  11. We use MegaCli tool for this controller. This has to be installed. Thr following command is used to view the status of virtual disk.

    # /opt/MegaRAID/MegaCli/MegaCli -LDInfo -Lall -aALL

    ( or in case of 64 bit Linux

    # /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aALL )

    Adapter 0 — Virtual Drive Information:
    Virtual Drive: 0 (Target Id: 0)
    Name                :RAID 5
    RAID Level          : Primary-5, Secondary-0, RAID Level Qualifier-3
    Size                : 1.818 TB
    Sector Size         : 512
    Parity Size         : 931.0 GB
    State               : Optimal
    Strip Size          : 64 KB
    Number Of Drives    : 3
    Span Depth          : 1
    Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
    Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
    Default Access Policy: Read/Write
    Current Access Policy: Read/Write
    Disk Cache Policy   : Disk’s Default
    Encryption Type     : None
    Is VD Cached: No

    Exit Code: 0x00

  12. The following command is used to view the status of physical disks.

    # /opt/MegaRAID/MegaCli/MegaCli -PDList -aALL

    Adapter #0

    Enclosure Device ID: N/A
    Slot Number: 0
    Drive’s position: DiskGroup: 0, Span: 0, Arm: 0
    Enclosure position: N/A
    Device Id: 0
    WWN:
    Sequence Number: 2
    Media Error Count: 0
    Other Error Count: 0
    Predictive Failure Count: 0
    Last Predictive Failure Event Seq Number: 0
    PD Type: SATA

    Raw Size: 931.512 GB [0x74706db0 Sectors]
    Non Coerced Size: 931.012 GB [0x74606db0 Sectors]
    Coerced Size: 931.0 GB [0x74600000 Sectors]
    Sector Size:  0
    Firmware state: Online, Spun Up
    Device Firmware Level: CC37
    Shield Counter: 0
    Successful diagnostics completion on :  N/A
    SAS Address(0): 0x1221000000000000
    Connected Port Number: 0(path0)
    Inquiry Data:             5VP1V1Q3ST31000528AS                            CC37
    FDE Capable: Not Capable
    FDE Enable: Disable
    Secured: Unsecured
    Locked: Unlocked
    Needs EKM Attention: No
    Foreign State: None
    Device Speed: Unknown
    Link Speed: Unknown
    Media Type: Hard Disk Device
    Drive Temperature : N/A
    PI Eligibility:  No
    Drive is formatted for PI information:  No
    PI: No PI
    Port-0 :
    Port status: Active
    Port’s Linkspeed: Unknown
    Drive has flagged a S.M.A.R.T alert : No

    Enclosure Device ID: N/A
    Slot Number: 1
    Drive’s position: DiskGroup: 0, Span: 0, Arm: 1
    Enclosure position: N/A
    Device Id: 1
    WWN:
    Sequence Number: 2
    Media Error Count: 0
    Other Error Count: 0
    Predictive Failure Count: 0
    Last Predictive Failure Event Seq Number: 0
    PD Type: SATA

    Raw Size: 931.512 GB [0x74706db0 Sectors]
    Non Coerced Size: 931.012 GB [0x74606db0 Sectors]
    Coerced Size: 931.0 GB [0x74600000 Sectors]
    Sector Size:  0
    Firmware state: Online, Spun Up
    Device Firmware Level: CC38
    Shield Counter: 0
    Successful diagnostics completion on :  N/A
    SAS Address(0): 0x1221000001000000
    Connected Port Number: 1(path0)
    Inquiry Data:             6VP4E9NFST31000528AS                            CC38
    FDE Capable: Not Capable
    FDE Enable: Disable
    Secured: Unsecured
    Locked: Unlocked
    Needs EKM Attention: No
    Foreign State: None
    Device Speed: Unknown
    Link Speed: Unknown
    Media Type: Hard Disk Device
    Drive Temperature : N/A
    PI Eligibility:  No
    Drive is formatted for PI information:  No
    PI: No PI
    Port-0 :
    Port status: Active
    Port’s Linkspeed: Unknown
    Drive has flagged a S.M.A.R.T alert : No

    Enclosure Device ID: N/A
    Slot Number: 2
    Drive’s position: DiskGroup: 0, Span: 0, Arm: 2
    Enclosure position: N/A
    Device Id: 2
    WWN:
    Sequence Number: 2
    Media Error Count: 0
    Other Error Count: 0
    Predictive Failure Count: 0
    Last Predictive Failure Event Seq Number: 0
    PD Type: SATA

    Raw Size: 931.512 GB [0x74706db0 Sectors]
    Non Coerced Size: 931.012 GB [0x74606db0 Sectors]
    Coerced Size: 931.0 GB [0x74600000 Sectors]
    Sector Size:  0
    Firmware state: Online, Spun Up
    Device Firmware Level: CC38
    Shield Counter: 0
    Successful diagnostics completion on :  N/A
    SAS Address(0): 0x1221000002000000
    Connected Port Number: 2(path0)
    Inquiry Data:             9VP9LMHJST31000528AS                            CC38
    FDE Capable: Not Capable
    FDE Enable: Disable
    Secured: Unsecured
    Locked: Unlocked
    Needs EKM Attention: No
    Foreign State: None
    Device Speed: Unknown
    Link Speed: Unknown
    Media Type: Hard Disk Device
    Drive Temperature : N/A
    PI Eligibility:  No
    Drive is formatted for PI information:  No
    PI: No PI
    Port-0 :
    Port status: Active
    Port’s Linkspeed: Unknown
    Drive has flagged a S.M.A.R.T alert : No

    Exit Code: 0x00

    ( or in case of 64 bit Linux

    # /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL )

    The following is also a useful command that provides information about virtual disk and physical disks at once.

    # /opt/MegaRAID/MegaCli/MegaCli64 -CfgDsply -a0