Increase the speed of Linux Software RAID reconstruction

If you are in a situation where you sit in front of the console (or on a remote ssh connection) waiting for a Linux software RAID to finish rebuilding (either you added a new drive, or you replaced a failed one, etc.) then you might be frustrated by how slow this process is running. You are running cat on /proc/mdstat repeatedly (you should really use watch in this case ;) ), and this seems to never finish… Obviously that there is a logical reason for this ‘slowness‘ and on a production system you should leave it running with the defaults. But in case you want to speed up this process here is how you can do it. This will place a much higher load on the system so you should use it with care.

To see your Linux kernel speed limits imposed on the RAID reconstruction use:

cat /proc/sys/dev/raid/speed_limit_max
200000
cat /proc/sys/dev/raid/speed_limit_min
1000

In the system logs you can see something similar to:
md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.

This means that the minimum guaranteed speed of the rebuild of the array is approx 1MB/s. The actual speed will be higher and will depend on the system load and what other processes are running at that time.
In case you want to increase this minimum speed you need to enter a higher value in speed_limit_min. For example to set this to approx 50 megabytes per second as minimum use:

echo 50000 >/proc/sys/dev/raid/speed_limit_min

The results are instant… you can return to the watch window to see it running, and hope that this will finish a little faster (this will really depend on the system you are running, the HDDs, controllers, etc.):

watch cat /proc/mdstat

The first thing you must learn about RAID technologies in Linux is that they have nothing in common with HP-UX, and I mean nothing! Yes there is LVM but that’s all, the mirror of a volume group for example is not done through LVM commands, in fact you are not going to mirror a volume group but the block device/s where the volume group resides.

There are two tools to manage RAID in Linux.

* dmraid

* mdadm

Dmraid is used to discover and activate software (ATA)RAID arrays, commonly known as fakeRAID, and mdadm is used to manage Linux Software RAID devices.

dmraid

Dmraid, uses libdevmapper and the device-mapper kernel driver to perform all the tasks.

The device-mapper is a component of the Linux Kernel. This the way the Linux Kernel do all the block device managment. It maps a block device onto another and forms the base of volume management (LVM2 and EVMS) and software raid. Multipathing support is also provided through the device-mapper. Device-mapper support is present in 2.6 kernels although there are patches for the most recent versions of 2.4 kernel version.

dmraid supports several array types.

[root@caladan ~]# dmraid -l

asr : Adaptec HostRAID ASR (0,1,10)

ddf1 : SNIA DDF1 (0,1,4,5,linear)

hpt37x : Highpoint HPT37X (S,0,1,10,01)

hpt45x : Highpoint HPT45X (S,0,1,10)

isw : Intel Software RAID (0,1)

jmicron : JMicron ATARAID (S,0,1)

lsi : LSI Logic MegaRAID (0,1,10)

nvidia : NVidia RAID (S,0,1,10,5)

pdc : Promise FastTrack (S,0,1,10)

sil : Silicon Image(tm) Medley(tm) (0,1,10)

via : VIA Software RAID (S,0,1,10)

dos : DOS partitions on SW RAIDs

[root@caladan ~]#

Following are a couple of examples to show dmraid operation.

* Array discovering

[root@caladan ~]# dmraid -r

/dev/dm-46: hpt45x, « hpt45x_chidjhaiaa-0″, striped, ok, 320172928 sectors, data@ 0

/dev/dm-50: hpt45x, « hpt45x_chidjhaiaa-0″, striped, ok, 320172928 sectors, data@ 0

/dev/dm-54: hpt45x, « hpt45x_chidjhaiaa-1″, striped, ok, 320172928 sectors, data@ 0

/dev/dm-58: hpt45x, « hpt45x_chidjhaiaa-1″, striped, ok, 320172928 sectors, data@ 0

[root@caladan ~]#

* Activate all discovered arrays

[root@caladan ~]# dmraid -ay

* Deactivate all discovered arrays

[root@caladan ~]# dmraid -an

mdadm

mdadm, is a tool to manage the Linux software RAID arrays. This tool has nothing to do with the device-mapper, in fact the device-mapper is not aware of the RAID arrays created with mdadm.

To illustrate this take a look at the screenshot below. I created a RAID1 device, /dev/md0, I then show its configuration with mdadm –detail. Later with dmsetup ls I list all the block devices seen by the device-mapper, as you can see there is no reference to /dev/md0.

Instead mdadm uses the MD (Multiple Devices) device driver, this driver provides virtual devices created from another independent devices. Currently the MD driver supports the following RAID levels and configurations

* RAID1

* RAID4

* RAID5

* RAID6

* RAID0

* LINEAR (a concatenated array)

* MULTIPATH

* FAULTY (an special failed array type for testing purposes)

The configuration of the MD devices is contained in the /etc/mdadm.conf file.

[root@caladan ~]# cat mdadm.conf

ARRAY /dev/md1 level=raid5 num-devices=3 spares=1 UUID=5c9d6a69:4a0f120b:f6b02789:3bbc8698

ARRAY /dev/md0 level=raid1 num-devices=2 UUID=b36f1b1c:87cf9497:73b81e8c:79ee3c44

[root@caladan ~]#

The mdadm tool has seven operation modes.

1. Assemble

2. Build

3. Create

4. Manage

5. Misc

6. Follow or Monitor

7. Grow

A more detailed description of every major operation mode is provided in the mdadm man page.

Finally below are examples of some of the more common operations with mdadm.

* Create a RAID1 array

[root@caladan ~]# mdadm –create /dev/md1 –verbose –level raid1 –raid-devices 2 /dev/sd[de]1

mdadm: size set to 1044096K

mdadm: array /dev/md1 started.

[root@caladan ~]#

* Get detailed configuration of the array

[root@caladan ~]# mdadm –query –detail /dev/md1

/dev/md1:

Version : 00.90.01

Creation Time : Tue Nov 23 22:37:05 2010

Raid Level : raid1

Array Size : 1044096 (1019.80 MiB 1069.15 MB)

Device Size : 1044096 (1019.80 MiB 1069.15 MB)

Raid Devices : 2

Total Devices : 2

Preferred Minor : 1

Persistence : Superblock is persistent

Update Time : Tue Nov 23 22:37:11 2010

State : clean

Active Devices : 2

Working Devices : 2

Failed Devices : 0

Spare Devices : 0

UUID : c1893118:c1327582:7dc3a667:aa87dfeb

Events : 0.2

Number Major Minor RaidDevice State

0 8 49 0 active sync /dev/sdd1

1 8 65 1 active sync /dev/sde1

[root@caladan ~]#

* Destroy the array

[root@caladan ~]# mdadm –remove /dev/md1

[root@caladan ~]# mdadm –stop /dev/md1

[root@caladan ~]# mdadm –detail /dev/md1

mdadm: md device /dev/md1 does not appear to be active.

[root@caladan ~]#

* Create a RAID5 array with an spare device

[root@caladan ~]# mdadm –create /dev/md1 –verbose –level raid5 –raid-devices 3 –spare-devices 1 /dev/sd[def]1 /dev/sdg1

mdadm: array /dev/md1 started

[root@caladan ~]#

* Check for the status of a task into the /proc/mdstat file.

[root@caladan ~]# cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]

md0 : active raid6 sdi1[7] sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]

226467456 blocks level 6, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

[=========>...........] resync = 49.1% (18552320/37744576) finish=11.4min speed=27963K/sec

unused devices: <none>

[root@caladan ~]#

* Generate the mdadm.conf file from the current active devices.

[root@caladan ~]# mdadm –detail –scan

ARRAY /dev/md1 level=raid5 num-devices=3 spares=1 UUID=5c9d6a69:4a0f120b:f6b02789:3bbc8698

ARRAY /dev/md0 level=raid1 num-devices=2 UUID=b36f1b1c:87cf9497:73b81e8c:79ee3c44

[root@caladan ~]# mdadm –detail –scan >> mdadm.conf

As a final thought, my recommendation is that if there is hardware RAID controller available, like the HP Smart Array P400 for example, go hard-RAID five by five and if not always use mdadm even if there is an onboard RAID controller.

Juanma.

via2010 November 24 « Juanma’s Blog.

Creating a 4-disk RAID10 using mdadm

Since I can’t seem to find instructions on how to do this (yet)…

I’m going to create a 4-disk RAID10 array using Linux Software RAID and mdadm. The old way is to create individual RAID1 volumes and then stripe a RAID0 volume over the RAID1 arrays. That requires creating extra /dev/mdN nodes which can be confusing to the admin that follows you.

1) Create the /dev/mdN node for the new RAID10 array. In my case, I already have /dev/md0 to /dev/md4 so I’m going to create /dev/md5 (note that « 5″ appears twice in the command).

# mknod /dev/md5 b 9 5

2) Use fdisk on the (4) drives, create a single primary partition of type « fd » (Linux raid autodetect). Note that I have *nothing* on these brand new drives, so I don’t care if it wipes out data.

3) Create the mdadm RAID set using 4 devices and a level of RAID10.

# mdadm –create /dev/md5 -v –raid-devices=4 –chunk=32 –level=raid10 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1

Which will result in the following output:

mdadm: layout defaults to n1

mdadm: size set to 732571904K

mdadm: array /dev/md5 started.

# cat /proc/mdstat

Personalities : [raid1] [raid10]

md5 : active raid10 sdf1[3] sde1[2] sdd1[1] sdc1[0]

1465143808 blocks 32K chunks 2 near-copies [4/4] [UUUU]

[>....................] resync = 0.2% (3058848/1465143808) finish=159.3min speed=152942K/sec

As you can see, we get around 150MB/s from the RAID10 array. The regular RAID1 arrays only have about 75MB/s throughput (same as a single 750GB drive).

A final note. My mdadm.conf file is completely empty on this system. That works well for simple systems, but you’ll want to create a configuration file in more complex setups.

Updates:

Most of the arrays that I’ve built have been based on 7200 RPM SATA drives. For small arrays (4 disks w/ a hot spare), often you can find enough ports on the motherboard. For larger arrays, you’ll need to look for PCIe SATA controllers. I’ve used Promise and 3ware SATA RAID cards. Basically any card that allows the SATA drives to be seen and is supported directly in the Linux kernel are good bets (going forward we’re going to switch to Areca at work).

Example of creating a 6-drive RAID10 array:

# mdadm –create /dev/md5 –raid-devices=6 –spare-devices=1 –layout=n2 –level=raid10 /dev/sd[a-g]1

In this case, we’re setting up a 6-drive RAID10 array along with 1 hot-spare. Disks sda to sdg all have a single partition on them, tagged as « fd » Linux RAID in fdisk.

« n2″ is the default RAID10 layout for mdadm and is a good default that provides balanced performance for reads and writes.

« f2″ is an optional layout that has better read performance, but worse write performance.

viatgharold.com: Tech Blog: Creating a 4-disk RAID10 using mdadm.

So, you created an array and you messed up something? Now you would like to delete it?

Please note: all data will be lost with this operation!

1. Stop the array

Code: Select all

# mdadm -S /dev/md0

2. Remove the disks from the array (in my example md0 is a raid5 array with 3 disks)

Code: Select all

# mdadm /dev/md0 -r /dev/sda

# mdadm /dev/md0 -r /dev/sdb

# mdadm /dev/md0 -r /dev/sdc

3. Destroy the array configuration from each disk (this will destroy all the data from your disks)

Code: Select all

# dd if=/dev/zero of=/dev/sda bs=1M count=1024

# dd if=/dev/zero of=/dev/sdb bs=1M count=1024

# dd if=/dev/zero of=/dev/sdc bs=1M count=1024

4. Remove the mdadm.conf file (if any)

Code: Select all

# rm /etc/mdadm/mdadm.conf

Now you can try to create a new array as you wish or as you were suposed to…

viaMdadm – How can i destroy or delete an array : Memory, Storage, Backup and Filesystems.

Installation RAID1 Logiciel sous Linux Debian Lenny x64
Installation.
Installer un système Debian avec deux disques dur, laisser le deuxième disque (sdb) sans
partitions, voici une imprime d’écran des partitions faites pour ce document :
1. Installez les paquets nécessaires :
apt-get install initramfs-tools mdadm
Ensembles MD requis par le système de fichiers racine : all
2. Chargez les modules nécessaires du noyau :
modprobe md
modprobe linear
modprobe multipath
modprobe raid0
modprobe raid1
modprobe raid5
modprobe raid6
modprobe raid10
3. Partitionner le deuxième disque (sdb) avec les mêmes partitions sda :
sfdisk -d /dev/sda | sfdisk /dev/sdb
4. Changer les type de partitions sdb à « Linux raid autodetect » :
fdisk /dev/sdb
Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): fd
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): fd
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): fd
Command (m for help): w
5. Assurez-vous qu’il n’y a pas de restes d’autre RAID sur le disque dur :
mdadm –zero-superblock /dev/sdb1
mdadm –zero-superblock /dev/sdb2
mdadm –zero-superblock /dev/sdb3
6. Créer le RAID (md) et ajouter chaque partition du sdb (sdb1 -> md0, sdb2 -> md1 et
sdb3 -> md2) :
mdadm –create /dev/md0 –level=1 –raid-disks=2 missing /dev/sdb1
mdadm –create /dev/md1 –level=1 –raid-disks=2 missing /dev/sdb2
mdadm –create /dev/md2 –level=1 –raid-disks=2 missing /dev/sdb3
continue creating array? Y
7. Formater le RAID (md) :
mkswap /dev/md0
mkfs.ext3 /dev/md1
mkfs.ext3 /dev/md2
8. reconfigurer le fichier mdadm.conf :
cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf_orig
mdadm –examine –scan >> /etc/mdadm/mdadm.conf
9. Monter le RAID md1 et md2 :
mkdir /mnt/md1
mkdir /mnt/md2
mount /dev/md1 /mnt/md1
mount /dev/md2 /mnt/md2
10. Modifier le fstab et mtab pour démarrer à partir du RAID :
sed -i ‘s:/dev/sda2:/dev/md1:g’ /etc/fstab
sed -i ‘s:/dev/sda3:/dev/md2:g’ /etc/fstab
sed -i ‘s:/dev/sda1:/dev/md0:g’ /etc/fstab
sed -i ‘s:/dev/sda3:/dev/md2:g’ /etc/mtab
sed -i ‘s:/dev/sda2:/dev/md1:g’ /etc/mtab
11. Modifier le grub sorte que si le démarrage échoue avec un disque, essaie l’autre :
nano /boot/grub/menu.lst
En dessous de la ligne default 0 ajouter : fallback 1
12. Modifier GRUB pour démarrer à partir du RAID, la liste de démarrage devrait
ressembler à :
nano /boot/grub/menu.lst
## ## End Default Options ##
title Debian GNU/Linux, kernel 2.6.26-2-amd64 (RAID1 HD1)
root (hd1,1)
kernel /boot/vmlinuz-2.6.26-2-amd64 root=/dev/md1 ro quiet
initrd /boot/initrd.img-2.6.26-2-amd64
title Debian GNU/Linux, kernel 2.6.26-2-amd64 (NO RAID)
root (hd0,1)
kernel /boot/vmlinuz-2.6.26-2-amd64 root=/dev/sda2 ro quiet
initrd /boot/initrd.img-2.6.26-2-amd64
title Debian GNU/Linux, kernel 2.6.26-2-amd64 (single-user mode)
root (hd0,1)
kernel /boot/vmlinuz-2.6.26-2-amd64 root=/dev/sda2 ro single
initrd /boot/initrd.img-2.6.26-2-amd64
### END DEBIAN AUTOMAGIC KERNELS LIST
13. Mettre à jour le ramdisk:
update-initramfs -u
14. Copiez le contenu de / vers /mnt/md1 :
cp -dpRxv / /mnt/md1/
15. Préparer GRUB :
grub
root (hd0,1)
setup (hd0)
root (hd1,1)
setup (hd1)
quit
16. Redémarrez le système pour démarrer sur le nouveau RAID 1 :
reboot
17. Changer le type de partition sda «Linux raid autodetect » :
fdisk /dev/sda
Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): fd
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): fd
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): fd
Command (m for help): w
18. Ajoutez le disque sda au RAID 1 :
mdadm –add /dev/md0 /dev/sda1
mdadm –add /dev/md1 /dev/sda2
mdadm –add /dev/md2 /dev/sda3
19. Attendez que le nouveau disque soit duplique :
watch cat /proc/mdstat
Una vez terminado el proceso presione Ctrl + C
20. reconfigurer le fichier mdadm.conf :
cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf_orig
mdadm –examine –scan >> /etc/mdadm/mdadm.conf
21. Modifier GRUB pour démarrer à partir du RAID, la liste de démarrage devrait
ressembler à :
nano /boot/grub/menu.lst
## ## End Default Options ##
title Debian GNU/Linux, kernel 2.6.26-2-amd64 (RAID1 HD0)
root (hd0,1)
kernel /boot/vmlinuz-2.6.26-2-amd64 root=/dev/md1 ro quiet
initrd /boot/initrd.img-2.6.26-2-amd64
title Debian GNU/Linux, kernel 2.6.26-2-amd64 (RAID1 HD1)
root (hd1,1)
kernel /boot/vmlinuz-2.6.26-2-amd64 root=/dev/md1 ro quiet
initrd /boot/initrd.img-2.6.26-2-amd64
#title Debian GNU/Linux, kernel 2.6.26-2-amd64 (NO RAID)
#root (hd0,1)
#kernel /boot/vmlinuz-2.6.26-2-amd64 root=/dev/sda2 ro quiet
#initrd /boot/initrd.img-2.6.26-2-amd64
#title Debian GNU/Linux, kernel 2.6.26-2-amd64 (single-user mode)
#root (hd0,1)
#kernel /boot/vmlinuz-2.6.26-2-amd64 root=/dev/sda2 ro single
#initrd /boot/initrd.img-2.6.26-2-amd64
### END DEBIAN AUTOMAGIC KERNELS LIST
22. Modifiez la ligne kopt :
sed -i ’63s:/dev/sda2:/dev/md1:g’ /boot/grub/menu.lst
23. Mettre à jour le ramdisk:
update-initramfs -u
24. Éditer le fichier /etc/mdadm.conf et définir l’option MAILADDR pour envoyer des
alertes :
MAILADDR user@mail.com
25. Restart :
reboot
Tests.
1. Simuler la perte de l’un des disques durs (dans ce cas hdb) :
mdadm –manage /dev/md0 –fail /dev/sdb1
mdadm –manage /dev/md1 –fail /dev/sdb2
mdadm –manage /dev/md2 –fail /dev/sdb3
mdadm –manage /dev/md0 –remove /dev/sdb1
mdadm –manage /dev/md1 –remove /dev/sdb2
mdadm –manage /dev/md2 –remove /dev/sdb3
2. Maintenant, vous pouvez éteindre le système et remplacer le disque dur sdb :
3. Maintenant, vous pouvez afficher l’état du RAID, remarquez qu’il manque de
disques :
cat /proc/mdstat
Reconstruire un disque dur.
1. Après avoir remplacé le disque hdb, créer la table de partitions :
sfdisk -d /dev/sda | sfdisk –force /dev/sdb
2. Assurez-vous qu’il n’y a pas de restes d’autre RAID sur le disque dur :
mdadm –zero-superblock /dev/sdb1
mdadm –zero-superblock /dev/sdb2
mdadm –zero-superblock /dev/sdb3
3. Ajoutez le disque sdb au RAID 1 :
mdadm –add /dev/md0 /dev/sdb1
mdadm –add /dev/md1 /dev/sdb2
mdadm –add /dev/md2 /dev/sdb3
4. Attendez que le nouveau disque soit duplique :
watch cat /proc/mdstat
Una vez terminado el proceso presione Ctrl + C
5. Réinstaller grub sur les deux disques :
grub
root (hd0,1)
setup (hd0)
root (hd1,1)
setup (hd1)
quit
Notes pour l’installation du Kernel OpenVZ.
1. Après l’installation du noyau OpenVZ, la liste de démarrage devrait ressembler à :
nano /boot/grub/menu.lst
## ## End Default Options ##
title Debian GNU/Linux, kernel 2.6.18-20-ovz-amd64 (RAID1 HD0)
root (hd0,1)
kernel /boot/vmlinuz-2.6.18-20-ovz-amd64 root=/dev/md1 ro quiet
initrd /boot/initrd.img-2.6.18-20-ovz-amd64
title Debian GNU/Linux, kernel 2.6.18-20-ovz-amd64 (RAID1 HD1)
root (hd1,1)
kernel /boot/vmlinuz-2.6.18-20-ovz-amd64 root=/dev/md1 ro quiet
initrd /boot/initrd.img-2.6.18-20-ovz-amd64
title Debian GNU/Linux, kernel 2.6.26-2-amd64 (RAID1 HD0)
root (hd0,1)
kernel /boot/vmlinuz-2.6.26-2-amd64 root=/dev/md1 ro quiet
initrd
/boot/initrd.img-2.6.26-2-amd64
title Debian GNU/Linux, kernel 2.6.26-2-amd64 (RAID1 HD1)
root (hd1,1)
kernel /boot/vmlinuz-2.6.26-2-amd64 root=/dev/md1 ro quiet
initrd /boot/initrd.img-2.6.26-2-amd64
#title Debian GNU/Linux, kernel 2.6.26-2-amd64 (single-user mode)
#root (hd0,1)
#kernel /boot/vmlinuz-2.6.26-2-amd64 root=/dev/md1 ro single
#initrd /boot/initrd.img-2.6.26-2-amd64
#title Debian GNU/Linux, kernel 2.6.18-20-ovz-amd64 (single-user mode)
#root (hd0,1)
#kernel /boot/vmlinuz-2.6.18-20-ovz-amd64 root=/dev/md1 ro single
#initrd /boot/initrd.img-2.6.18-20-ovz-amd64
### END DEBIAN AUTOMAGIC KERNELS LIST
2. Mettre à jour le ramdisk:
update-initramfs -u