12.4 RAID
12.4.1 Software RAID
12.4.1.1 Concatenated Disk Driver (CCD) Configuration
Original work by Christopher Shumway. Revised by Jim Brown.When choosing a mass storage solution the most important factors to consider are speed, reliability, and cost. It is rare to have all three in balance; normally a fast, reliable mass storage device is expensive, and to cut back on cost either speed or reliability must be sacrificed.
In designing the system described below, cost was chosen as the most important factor, followed by speed, then reliability. Data transfer speed for this system is ultimately constrained by the network. And while reliability is very important, the CCD drive described below serves online data that is already fully backed up on CD-R's and can easily be replaced.
Defining your own requirements is the first step in choosing a mass storage solution. If your requirements prefer speed or reliability over cost, your solution will differ from the system described in this section.
12.4.1.1.1 Installing the Hardware
In addition to the IDE system disk, three Western Digital 30GB, 5400 RPM IDE disks form the core of the CCD disk described below providing approximately 90GB of online storage. Ideally, each IDE disk would have its own IDE controller and cable, but to minimize cost, additional IDE controllers were not used. Instead the disks were configured with jumpers so that each IDE controller has one master, and one slave.
Upon reboot, the system BIOS was configured to automatically detect the disks attached. More importantly, DragonFly detected them on reboot:
ad0: 19574MB <WDC WD205BA> [39770/16/63] at ata0-master UDMA33 ad1: 29333MB <WDC WD307AA> [59598/16/63] at ata0-slave UDMA33 ad2: 29333MB <WDC WD307AA> [59598/16/63] at ata1-master UDMA33 ad3: 29333MB <WDC WD307AA> [59598/16/63] at ata1-slave UDMA33
Note: If DragonFly does not detect all the disks, ensure that you have jumpered them correctly. Most IDE drives also have a ``Cable Select'' jumper. This is not the jumper for the master/slave relationship. Consult the drive documentation for help in identifying the correct jumper.
Next, consider how to attach them as part of the file system. You should research both vinum(8) (Chapter 13) and ccd(4). In this particular configuration, ccd(4) was chosen.
12.4.1.1.2 Setting Up the CCD
The driver ccd(4) allows you to take several identical disks and concatenate them into one logical file system. In order to use ccd(4), you need a kernel with ccd(4) support built in. Add this line to your kernel configuration file, rebuild, and reinstall the kernel:
pseudo-device ccd 4
The ccd(4) support can also be loaded as a kernel loadable module.
To set up ccd(4), you must first use disklabel(8) to label the disks:
disklabel -r -w ad1 auto disklabel -r -w ad2 auto disklabel -r -w ad3 auto
This creates a disklabel for ad1c, ad2c and ad3c that spans the entire disk.
The next step is to change the disk label type. You can use disklabel(8) to edit the disks:
disklabel -e ad1 disklabel -e ad2 disklabel -e ad3
This opens up the current disk label on each disk with the editor specified by the EDITOR environment variable, typically vi(1).
An unmodified disk label will look something like this:
16 partitions: # size offset fstype [fsize bsize bps/cpg] c: 60074784 0 unused 0 0 0 # (Cyl. 0 - 59597)
Add a new e partition for ccd(4) to use. This
can usually be copied from the c partition,
but the fstype must
be 4.2BSD. The disk label should
now look something like this:
16 partitions: # size offset fstype [fsize bsize bps/cpg] c: 60074784 0 unused 0 0 0 # (Cyl. 0 - 59597) e: 60074784 0 4.2BSD 0 0 0 # (Cyl. 0 - 59597)
12.4.1.1.3 Building the File System
The device node for ccd0c may not exist yet, so to create it, perform the following commands:
cd /dev sh MAKEDEV ccd0
Now that you have all of the disks labeled, you must build the ccd(4). To do that, use ccdconfig(8), with options similar to the following:
ccdconfig ccd032
0
/dev/ad1e
/dev/ad2e /dev/ad3e
The use and meaning of each option is shown below:
- The first argument is the device to configure, in this case, /dev/ccd0c. The /dev/ portion is optional.
- The interleave for the file system. The interleave defines the size of a stripe in disk blocks, each normally 512 bytes. So, an interleave of 32 would be 16,384 bytes.
- Flags for ccdconfig(8). If you want to enable drive mirroring, you can specify a flag here. This configuration does not provide mirroring for ccd(4), so it is set at 0 (zero).
- The final arguments to ccdconfig(8) are the devices to place into the array. Use the complete pathname for each device.
After running ccdconfig(8) the ccd(4) is configured. A file system can be installed. Refer to newfs(8) for options, or simply run:
newfs /dev/ccd0c
12.4.1.1.4 Making it All Automatic
Generally, you will want to mount the ccd(4) upon each reboot. To do this, you must configure it first. Write out your current configuration to /etc/ccd.conf using the following command:
ccdconfig -g > /etc/ccd.conf
During reboot, the script /etc/rc runs ccdconfig -C if /etc/ccd.conf exists. This automatically configures the ccd(4) so it can be mounted.
Note: If you are booting into single user mode, before you can mount(8) the ccd(4), you need to issue the following command to configure the array:
ccdconfig -C
To automatically mount the ccd(4), place an entry for the ccd(4) in /etc/fstab so it will be mounted at boot time:
/dev/ccd0c /media ufs rw 2 2
12.4.1.2 The Vinum Volume Manager
The Vinum Volume Manager is a block device driver which implements virtual disk drives. It isolates disk hardware from the block device interface and maps data in ways which result in an increase in flexibility, performance and reliability compared to the traditional slice view of disk storage. vinum(8) implements the RAID-0, RAID-1 and RAID-5 models, both individually and in combination.
See Chapter 13 for more information about vinum(8).
12.4.2 Hardware RAID
DragonFly also supports a variety of hardware RAID controllers. These devices control a RAID subsystem without the need for DragonFly specific software to manage the array.
Using an on-card BIOS, the card controls most of the disk operations itself. The following is a brief setup description using a Promise IDE RAID controller. When this card is installed and the system is started up, it displays a prompt requesting information. Follow the instructions to enter the card's setup screen. From here, you have the ability to combine all the attached drives. After doing so, the disk(s) will look like a single drive to DragonFly. Other RAID levels can be set up accordingly.
12.4.3 Rebuilding ATA RAID1 Arrays
DragonFly allows you to hot-replace a failed disk in an array. This requires that you catch it before you reboot.
You will probably see something like the following in /var/log/messages or in the dmesg(8) output:
ad6 on monster1 suffered a hard error. ad6: READ command timeout tag=0 serv=0 - resetting ad6: trying fallback to PIO mode ata3: resetting devices .. done ad6: hard error reading fsbn 1116119 of 0-7 (ad6 bn 1116119; cn 1107 tn 4 sn 11) status=59 error=40 ar0: WARNING - mirror lost
Using atacontrol(8), check for further information:
# atacontrol list ATA channel 0: Master: no device present Slave: acd0 <HL-DT-ST CD-ROM GCR-8520B/1.00> ATA/ATAPI rev 0 ATA channel 1: Master: no device present Slave: no device present ATA channel 2: Master: ad4 <MAXTOR 6L080J4/A93.0500> ATA/ATAPI rev 5 Slave: no device present ATA channel 3: Master: ad6 <MAXTOR 6L080J4/A93.0500> ATA/ATAPI rev 5 Slave: no device present # atacontrol status ar0 ar0: ATA RAID1 subdisks: ad4 ad6 status: DEGRADED
You will first need to detach the disk from the array so that you can safely remove it:
# atacontrol detach 3
Replace the disk.
Reattach the disk as a spare:
# atacontrol attach 3 Master: ad6 <MAXTOR 6L080J4/A93.0500> ATA/ATAPI rev 5 Slave: no device present
Rebuild the array:
# atacontrol rebuild ar0
The rebuild command hangs until complete. However, it is possible to open another terminal (using Alt+Fn) and check on the progress by issuing the following command:
# dmesg | tail -10 [output removed] ad6: removed from configuration ad6: deleted from ar0 disk1 ad6: inserted into ar0 disk1 as spare # atacontrol status ar0 ar0: ATA RAID1 subdisks: ad4 ad6 status: REBUILDING 0% completed
Wait until this operation completes.