4. Failure Scenarios

This section contains descriptions of various failure scenarios. For each scenario, there is a subsection on how to configure your server for degraded mode operation, how to recover from the failure, how to exit degraded mode, and how to simulate the failure.

Tip: Make a hard copy of these instructions and leave them inside the CPU case, being careful not to interfere with ventilation.

4.1. Root filesystem on ad0 unusable, rest of drive ok

Note: We assume here that the boot blocks and disk label on /dev/ad0 are ok. If your BIOS can boot from a drive other than C:, you may be able to get around this limitation.

4.1.1. Configure Server for Degraded Mode

  1. Use BootMgr to load kernel from /dev/ad2s1a.

    1. Hit F5 in BootMgr to select Drive 1.

    2. Hit F1 to select FreeBSD.

  2. After the kernel is loaded, hit any key but enter to interrupt the boot sequence. Boot into single-user mode and allow explicit entry of a root filesystem.

    Hit [Enter] to boot immediately, or any other key for command prompt.
    Booting [kernel] in 8 seconds...
    
    Type '?' for a list of commands, 'help' for more detailed help.
    ok boot -as
    
  3. Select /rootback as your root filesystem.

    Manual root filesystem specification:
        <fstype>:<device>  Mount <device> using filesystem <fstype>
                 e.g. ufs:/dev/da0s1a
        ?                  List valid disk boot devices
        <empty line>       Abort manual input
    
      mountroot> ufs:/dev/ad2s1a
    
  4. Now that you are in single-user mode, change /etc/fstab to avoid the bad root filesystem.

    Tip: If you used the bootvinum Perl script from Appendix A below, then these commands should configure your server for degraded mode.

    # fsck -p /
    # mount /
    # cd /etc
    # mv fstab fstab.bak
    # cp fstab_ad0s1_root_bad fstab
    # cd /
    # mount -o ro /
    # vinum start
    # fsck -p
    # ^D
    

4.1.2. Recovery

  1. Restore /dev/ad0s1a from backups or copy /rootback to it with these commands:

    # umount /rootbad
    # newfs /dev/ad0s1a
    # tunefs -n enable /dev/ad0s1a
    # mount /rootbad
    # cd /rootbad
    # dump 0f - / | restore rf -
    # rm restoresymtable
    

4.1.3. Exiting Degraded Mode

  1. Enter single-user mode.

    # shutdown now
    
  2. Put /etc/fstab back to normal and reboot.

    # cd /rootbad/etc
    # rm fstab
    # mv fstab.bak fstab
    # reboot
    
  3. Reboot and hit F1 to boot from /dev/ad0 when prompted by BootMgr.

4.1.4. Simulation

This kind of failure can be simulated by shutting down to single-user mode and then booting as shown above in Section 4.1.1.

4.2. Drive ad2 Fails

This section deals with the total failure of /dev/ad2.

4.2.1. Configure Server for Degraded Mode

  1. After the kernel is loaded, hit any key but Enter to interrupt the boot sequence. Boot into single-user mode.

    Hit [Enter] to boot immediately, or any other key for command prompt.
    Booting [kernel] in 8 seconds...
    
    Type '?' for a list of commands, 'help' for more detailed help.
    ok boot -s
    
  2. Change /etc/fstab to avoid the bad drive. If you used the bootvinum Perl script from Appendix A below, then these commands should configure your server for degraded mode.

    # fsck -p /
    # mount /
    # cd /etc
    # mv fstab fstab.bak
    # cp fstab_only_have_ad0s1 fstab
    # cd /
    # mount -o ro /
    # vinum start
    # fsck -p
    # ^D
    

    If you do not have modified versions of /etc/fstab that are ready for use, then you can use ed to make one. Alternatively, you can fsck and mount /usr and then use your favorite editor.

4.2.2. Recovery

We assume here that your server is up and running multi-user in degraded mode on just /dev/ad0 and that you have a new spindle now on /dev/ad2 ready to go.

You will need a new spindle with enough room to hold root and swap partitions plus a Vinum partition large enough to hold /home and /usr.

  1. Create a BIOS partition (slice) on the new spindle.

    # /stand/sysinstall
    
    1. Select Custom.

    2. Select Partition.

    3. Select ad2.

    4. Create a FreeBSD (type 165) slice large enough to hold everything mentioned above.

    5. Write changes.

    6. Yes, you are absolutely sure.

    7. Select BootMgr.

    8. Quit Partitioning.

    9. Exit /stand/sysinstall.

  2. Create disk label partitioning based on current /dev/ad0 partitioning.

    # disklabel ad0 > /tmp/ad0
    # disklabel -e ad2
    

    This will drop you into your favorite editor.

    1. Copy the lines for the a and b partitions from /tmp/ad0 to the ad2 disklabel.

    2. Add the size of the a and b partitions to find the proper offset for the h partition.

    3. Subtract this offset from the size of the c partition to find the proper size for the h partition.

    4. Define an h partition with the size and offset calculated above.

    5. Set the fstype column to vinum.

    6. Save the file and quit your editor.

  3. Tell Vinum about the new drive.

    1. Ask Vinum to start an editor with a copy of the current configuration.

      # vinum create
      
    2. Uncomment the drive line referring to drive UpWindow and set device to /dev/ad2s1h.

    3. Save the file and quit your editor.

  4. Now that Vinum has two spindles again, revive the mirrors.

    # vinum start -w usr.p1.s0
    # vinum start -w home.p1.s0
    
  5. Now we need to restore /rootback to a current copy of the root filesystem. These commands will accomplish this.

    # newfs /dev/ad2s1a
    # tunefs -n enable /dev/ad2s1a
    # mount /dev/ad2s1a /mnt
    # cd /mnt
    # dump 0f - / | restore rf -
    # rm restoresymtable
    # cd /
    # umount /mnt
    

4.2.3. Exiting Degraded Mode

  1. Enter single-user mode.

    # shutdown now
    
  2. Return /etc/fstab to its normal state and reboot.

    # cd /etc
    # rm fstab
    # mv fstab.bak fstab
    # reboot
    

4.2.4. Simulation

You can simulate this kind of failure by unplugging /dev/ad2, write-protecting it, or by this procedure:

  1. Shutdown to single-user mode.

  2. Unmount all non-root filesystems.

  3. Clobber any existing Vinum configuration and partitioning on /dev/ad2.

    # vinum stop
    # dd if=/dev/zero of=/dev/ad2s1h count=512
    # dd if=/dev/zero of=/dev/ad2 count=512
    

4.3. Drive ad0 Fails

Some BIOSes can boot from drive 1 or drive 2 (often called C: or D:), while others can boot only from drive 1. If your BIOS can boot from either, the fastest road to recovery might be to boot directly from /dev/ad2 in single-user mode and install /etc/fstab_only_have_ad2s1 as /etc/fstab. You would then have to adapt the /dev/ad2 failure recovery instructions from Section 4.2.2 above.

If your BIOS can only boot from drive one, then you will have to unplug drive YouCrazy from the controller for /dev/ad2 and plug it into the controller for /dev/ad0. Then continue with the instructions for /dev/ad2 failure recovery in Section 4.2.2 above.