Fixing a broken mdadm array – failed drive

Sometimes, even with the best intentions things can go wrong with a RAID array. A drive may fail, or the array may become ‘dirty’ for any number of reasons.

Here, we will go through some simple steps to repair a damaged array.

In our example case, a drive has failed. By running the following command in a terminal, we can get a status update on our array:

sudo mdadm --detail /dev/md0 # Displays detail about /dev/md0

The output:


You can see the state is listed as “clean, degraded” this means a drive is missing from the array. Also note that device 1 has been “removed”.

Before we do anything, we need to unmount our array (in this case, /dev/md0)

sudo umount /dev/md0 # Unmounts /dev/md0

If you recieve a ‘device is busy’ warning, you can find out what process is using the array with the following command:

fuser -m /dev/md0 # Shows what process number is using /dev/md0
/dev/sdc1: 538
ps auxw|grep 538
# Shows what process number 538 refers to
damian 538 0.4 2.7 219212 56792 ? SLl Feb11 11:25 rhythmbox

So in this case, it is rhythmbox that is using the drive. Close this, and umount again. If it is Samba, then issue the following command then umount:

sudo /etc/init.d/samba stop # Stops the Samba process

Failed Drive has been re-added

In this instance, we can try to re-add the lost device. In this case I believe that restarting your computer is a good first step. On reboot, open a terminal and run su to become root. Then run the following commands:

sudo mdadm --detail /dev/md0   # Just to check nothing has changed
sudo mdadm --add /dev/md0 /dev/sdc1  # To re-add the faulty (now working) HDD

You should recieve mdadm: re-added /dev/sdc1. If so, run the following:

sudo mdadm --detail /dev/md0   # To ensure the drive has readded successfully

The output should look like this:


Notice “clean, degraded, recovering” this is a good sign – as is “spare rebuilding” these messages mean that the array is rebuilding successfully (so far).

To monitor further – run the following command:

sudo watch cat /proc/mdstat

This command will display the status of mdadm, and refresh every 2 seconds. When you are done watching, you can press CTRL+C to escape back to the command line, or you can simply close the terminal window

The eventual result of another sudo mdadm --detail /dev/md0 should show the array as “clean”

MDADM can be tricky – Jaytag can help. Give us a call on 0845 310 2750 about software and hardware RAID arrays, and we can give advice about choosing the right option for you.

read more
DamianFixing a broken mdadm array – failed drive

Creating a RAID 5 array in Ubuntu with MDADM – Walkthrough

Software RAID-5 is a cheap and easy way to create a virtual single drive from many to store your files. Software RAID in Linux, via mdadm, offers lots of advanced features that are only normally available on harware RAID controller cards. A big one is the ability to ‘grow’ the array of disks when you run out of space! Consider also that software RAID can move with you – so if you decide to change motherboards or your RAID controller fails it is won’t mean the end of the world.

This guide details setting up software RAID 5 on Hardy Heron (8.04) Ubuntu using mdadm after you have a running Ubuntu install. It does not cover everything you need to know about RAID and the knowledge in this document is by no means extensive – please check out the further reading link for more information at the end of this article.

There is an expectation that you have a basic understanding on how to use the command line (terminal) It’s recommended that you are at least familiar with it before following this walkthrough.

Before we get started, remember that no RAID solution is a viable replacement for regularly backing up your data! If your data is mission critical, make sure it’s backed either on removable media or on another device. We will cover this in another article.

One: Prepare the Disks

Open up a terminal window and run:

sudo fdisk -l

This will bring up a list of available disks.

I will be using 4, 250gb Hard Disks (sdb1, sdc1, sdd1 and sde1) that I wish to combine into a RAID-5 volume. The disks are all currently unformatted and unpartitioned.

We firstly need to format the drives (as ext3) and set the RAID flag. This can be easily achieved with gPartEd. If you do not have “Partition Editor” on your System->Administration menu, install it with this command:

sudo apt-get install gparted

In the Partition Editor application, select each disk in turn and format it to ext3. CAUTION! you will lose all data on the device you format.

Once the format is complete, right click on each new “volume” you have created and select “Manage Flags” – tick the “RAID” flag to indicate the disk will become part of a RAID set.

Once you have your three disks set up, running fdisk -l (as root) should now indicate that those disks are correctly prepared to be created into the RAID 5 array.

Two: Creating the RAID Device
If you don’t already have mdadm (Mirrored Device Admin) installed, run the following:

sudo apt-get install mdadm

You can now create your RAID volume (md0, if this is to be your first) by running:

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /sde1

To further explain, --verbose will display more information on the screen as the RAID is created (which is useful in case of problems), /dev/md0 is the RAID device to be created, --level=5 dictates that we want to create a RAID 5 volume, and --raid-devices=4 dictates that there are to be four drives in our raid array. Following this is a list of the disks you wish to assign to the array. For more information, you can run: mdadm --help-options

When mdadm --create is run, your RAID device will be created

Once the build is under-way, you can monitor its progress by running:

sudo watch cat /proc/mdstat

This command will display the status of mdadm, and refresh every 2 seconds. When you are done watching, you can press CTRL+C to escape back to the command line, or you can simply close the terminal window.

The last stage before you create the file system on your new raid disk is to create your mdadm.conf file. This file contains information about your RAID array that the mdm daemon will need on boot to “re-assemble” your array ready for mounting. To create your mdadm.conf file, run the following commands:

sudo echo "DEVICE partitions" > /etc/mdadm/mdadm.conf
sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Three: Creating the Filesystem and Mount Point
Now that you have created a RAID 5 volume, it needs to be formatted before data can be stored on it. In this case, we have opted for ext3. The following command will format the RAID volume to ext3. Note that you can format the disk whilst the array is still synchronising).

sudo mke2fs -j /dev/md0

This may take some time – especially if the command is run while you are synchronising the array.

Once the format has completed you need to edit your fstab to make the RAID-5 volume automatically mount on the next boot. In this case, we have used /var/media. However, you may use any mount point that you wish. Add the mount point to the /etc/fstab file:

/dev/md0 /var/media auto defaults 0 3

You can now mount the array (even while it is still syncing!) by issuing the following command as root:

mount /dev/md0

Some further reading:

This short guide is by no means everything there is to know on the subject of RAID and mdadm. Once your RAID array is up and running there are still plenty more tweaks and obstacles to overcome in the future. Review the following links to find out more:

read more
DamianCreating a RAID 5 array in Ubuntu with MDADM – Walkthrough