Raid on Linux

Motivation

While they have gotten significantly more reliable in the last years, hard-disk are still the parts that most often fail in computers and they contain all your data. Thus you should always have a good and recent Backup. But if your server crashes because of a defective hard-drive this is still bad as it means the server is not available and you might need a significant amount of time to restore your data. What we want is a system that can tolerate the failure of a disk and still work. This is what RAID is for.

What is a RAID

In a RAID system you store your data on more then one disk when you write and when you read you can take any of the disks that contain a copy of your data and read from there. When one of the disks fails your data is still available. Once your replace the damaged disk the system can sync the data back and restore the redundancy.

On a RAID1 system you have 2 disks and the data is mirrored on both disks.

On a RAID4 and RAID5 you have 3 or more disks and one of the disks is a party disk (on RAID5 the parity is distributed) disk: E.g. on a RAID5 with 4 disks of 1TB you have a capacity of 3TB and one of the disks can fail.

On a RAID6 you have 2 party disks. This means that 2 disks can fail and you still have access to your data. In a RAID5 when after one disk fails and the disk is replaced the data on the party must be reconstructed. The strain that this puts on the disks can cause an additonal disk to fail so it makes sense to have a 2nd party disk for such cases.

On a RAID0 you do NOT have redundancy but the data is striped between the 2 disks and if one of them fails you loose all your data. People use RAID0 if they need higher performance and if they do not care about data loss.

RAID10 consists of 2 RAID1 systems that are combined as a RAID0 so you have additional performance while still some degree of redundancy. Similar: RAID60 which is a stripe set of 2 RAID6.

Hot Spare Disks are disks that are already in the system but not used until a disk fails and then used to immediately replace the failed disk.

A RAID needs to be monitored!

Since the system runs uninterrupted when a disk fails it can easily happen that you completely miss that event and for month you run the system without redundancy. When another disk fails your data is lost! Thus you always need to monitor your RAID system for failed disks so that you can replace the failed parts in a timely manner. This applies to all redundant components in your system (e.g. redundant power supply).

Software vs. Hardware RAID

Hardware RAID

If you buy a server you usually get them with a Hardware RAID controller: That is the controller card takes care of the RAID. If multiple disks are connected and your controller is configured the right way it presents the RAID devices as a normal disk, independent of the operating system you are using. Often this configuration can be done via a BIOS menu that is available while booting the server.

Now, since you need to monitor your RAID and since on an important server you will also want the ability to change the configuration of your RAID while the system is running you need some vendor specific tools that can talk to your RAID controller. While Linux is the most important server operating system today and those tools are widely available the downside is still that they are different for each vendor.

If you have a RAID with more then 2 disks there might be an issue in the case of a loss of power: When some disks have written a sector and the parity is still missing the sector is in an inconsistent state. To increase performance and to avoid such cases hardware RAID controllers often come with a battery backed RAM that can cache writes. This is less of an issue with RAID1. Also the processor on the RAID card can take off the load that is necessary to compute the parity (which is not much load for modern processors).

Software RAID

On a software RAID the RAID is done by the kernel. In Linux this can be with the LVM Subsystem but usually one creates RAID devices with the mdadm tool.

Advantages of Software RAID over Hardware RAID:

• Tools to create and monitor your RAID are independent of the hardware vendor.
• You can create RAIDs between disks on different controllers or even including network disks
• You can create RAIDs on certain partitions and leave other parts of the disk for other purposes.

Disadvantages:

• No battery cache
• A little bit more CPU
• Setting up the boot configuration on RAID is a bit more difficult

Software RAID with mdadm on Linux

# create a raid device md0 with RAID1:
mdadm create /dev/md0 -l1 -n2 /dev/sdb7 /dev/sdc7
#
to see the status of your raid

cat /proc/mdstat 

# now we can create a filesystem on this device:

mkfs.ext4 /dev/md0

mount /dev/md0 /mnt/data

#After a disk failure: add a replacement: 

mdadm --manage /dev/md0 --add /dev/sde5

The devices should be the same size (they do not need to match exactly).

To see which partitions contain RAID signatures you can use:

mdadm --examine --scan

The list of UUIDs and ARRAYs produced above should be configured in the config file:

/etc/mdadm/mdadm.conf

There you should also configure an email address that receives a warning when the RAID is in a degraded state. This of course only works if you have properly setup your machine so that it can send emails. Which you should always do.

Exercises

• Compare the advantages and disadvantages of Hardware vs. Software RAID
• If you have a system where you have 2 spare partitions you could try to setup a RAID1 with mdadm there.