How to re-mirror rootvg after a migration installation on hdisk0

After confirmation from the Application and DBA teams that the application and databases are down the UNIX Administrators need to perform the following steps.
1) Remove the alternate i.e.old copy of rootvg altinst_rootvg to free hdisk1
2) Add hdisk1 to rootvg
3) Mirror the rootvg's hdisk0 with hdisk1
4) Create the boot device
5) Rebuild the bootlist
6) Add hdisk1 and hdisk0 to bootlist and reboot the server to disable the quorum
7) Check the status of server and check the boot log for any errors
8) Rearrange the bootlist with hdisk0 and hdisk1
9) Reboot with hdisk0
First complete a status check

Hostname:/:/> lspv
hdisk0          005db36d2f1e67e5                    rootvg          active
hdisk1          005db36d8792541c                    altinst_rootvg    active

Hostname:/> lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            542         337         109..08..03..108..109

Hostname:/:/> bootinfo -b
hdisk0
Hostname:/:/> lsvg
rootvg
altinst_rootvg
issvg
datavg

Now proceed with the activity:

Hostname:/:/> alt_rootvg_op -X altinst_rootvg   #This removes altinst_rootvg   ; hdisk1 is now accessable

Bootlist is set to the boot disk: hdisk0

Hostname:/:/> lsvg
rootvg
issvg
datavg

Hostname:/:/> lsvg -o
datavg
issvg
rootvg

Hostname:/:/> lspv
hdisk0          005db36d2f1e67e5                    rootvg          active
hdisk1          005db36d8792541c                    None

Hostname:/> extendvg -f rootvg hdisk1   #Add hdisk1 to rootvg; I had to use the -f force option

Hostname:/> lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            542         337         109..08..03..108..109
hdisk1            active            542         542         109..108..108..108..109

Hostname:/> mirrorvg rootvg
0516-1124 mirrorvg: Quorum requirement turned off, reboot system for this
       to take effect for rootvg.
0516-1126 mirrorvg: rootvg successfully mirrored, user should perform
       bosboot of system to initialize boot records.  Then, user must modify
       bootlist to include:  hdisk0 hdisk1.
You have mail in /usr/spool/mail/root

Hostname:/> lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            542         337         109..08..03..108..109
hdisk1            active            542         353         109..24..03..108..109

Hostname:/> lsvg -l rootvg
rootvg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
hd5                 boot       1     2     2    closed/syncd  N/A
hd6                 paging     32    64    2    open/syncd    N/A
hd8                 jfslog     1     2     2    open/syncd    N/A
hd4                 jfs        1     2     2    open/syncd    /
hd2                 jfs        101   202   2    open/syncd    /usr
hd9var              jfs        12    24    2    open/syncd    /var
hd3                 jfs        24    48    2    open/syncd    /tmp
hd1                 jfs        14    28    2    open/syncd    /home
hd10opt             jfs        2     4     2    open/syncd    /opt
dump_disk0          sysdump    16    16    1    open/syncd    N/A
lv01                jfs        1     2     2    open/syncd    /var/adm/perfmgr

Hostname:/> bosboot -a
Hostname: Boot image is 30577 512 byte blocks.

Hostname:/>  bootlist -m normal hdisk0 hdisk1

Hostname> bootlist -m normal hdisk1 hdisk0

Hostname> reboot

Hostname> bootinfo -b
hdisk1

Hostname:/> alog -o -t boot | less

Search for your disk (hdisk1 for me) in the boot log:

Hostname>  bootlist -m normal hdisk0 hdisk1

Hostname> reboot

Hostname> bootinfo -b
hdisk0

Configuration of AIX 5.3 on an IBM p5 55A server after installation

This document sets out the procedure to configure AIX 5.3 on an IBM P5 55A server connected to a DS4700 after installation
After AIX 5.3 is installed (see the AIX installation guide under the AIX and installation tabs on this website), there will be some configuration tasks necessary. The tasks documented here are not a complete and exhaustive list but some of the tasks I have needed to complete whilst installing servers.

Check if all necessary filesets are installed

If the new server is replacing an old server run lslpp -l | more on the old server and compare with the new server to ensure that all necessary filesets are installed

Check that rootvg is mirrored to the second internal hard drive

To do this type lsvg -l rootvg. You should see a list of the logical volumes in the volume group and twice the number of PPs to LPs, if this is the case then the LVs are mirrored. If there are the same number of PPs to LPs then there is no software mirroring. Also type lsvg -p rootvg to view the physical disks associated with that volume group.
There may be mirroring at the hardware/controller level, check if so before proceeding with configuring of software mirroring.

How to mirror the root volume group rootvg

Type mirrorvg rootvg
Once mirroring has finished type lsvg -l rootvg and you should see twice as many PPs as there are LPs
If for any reason you need to unmirror rootvg type rmlvcopy hd6 1 hdisk1 to remove the copy PPs on hdisk1 for LV hd6. Do this for each LV in rootvg

Increase the size of the rootvg logical volumes/filesystems if required

For instance to increase the size of /var to 800M type chfs -a size=800M /var

Increasing the size of the paging space

Paging space should normally be one to two times the size of memory depending on the application or database running on the server and also how much memory is installed in the server
To view the size and location of paging space type lsps -a
To view how many LPs are allocated to paging space type lslv hd6, hd6 with probably be the paging space LV
To increase the size of paging space by 64 logical partitions type chps -s64 hd6

Increasing the number of AIO servers

If oracle is going to be installed on the server (or another database that uses AIO) then you may need to increase the number of asynchronous I/O (AIO) server processes
To view the max and min number of AIO servers, type lsattr -El aio0
Oracle recommends minservers to be equal to the number of CPUs on the system or 10, whichever is lower
If you are using JFS/JFS2 Oracle recommends maxservers to be 10 times the number of logical disks divided by the number of CPUs.
For maxreqs set the initial value to 4 times the number of logical disks multiplied by the queue depth. You can determine the queue depth by running the following command: lsattr -E -l hdiskxx
To change the number of max and min servers type chdev -l aio0 -a maxservers='m' -a minservers='n'
To change the number of maxreqs type chdev -l aio0 -a maxreqs='n'

Setting the maximum number of processes per user (maxuproc)

To view the value of maxuproc type lsattr -D -l sys0 -a maxuproc
To change the value of maxuproc type chdev -l sys0 -a maxuproc='n', the new maxproc value takes effect after the next reboot

Creating files greater than 2GB

To be able to create files greater than 2GB on JFS filesystems (this option does not exist on JFS2) do the following
Start smit as root then select:
1. System storage management
2. File systems
3. Add/change/show/delete file systems
4. Journaled file systems
5. Change/show characteristics of a journaled file system (select File System where DB2 data resides)
6. Large file enabled TRUE
Also by default the value of fsize (file size) is set to 1048575 blocks (for JFS and JFS2)
To change this value to unlimited edit /etc/security/limits and change the value of fsize to -1. This can be done system wide by modifying the default value or per user by adding fsize = -1 to an individual user directive.
The value will take effect the next time you login

Configuration of RDAC (Redundant Dual Active Controller) multipathing

Ensure you have dual fibre channel HBA cards fitted in your server with a fibre cable from each card to a separate controller on your disk array. I used a DS4700 array for this configuration.
Run cfgmgr then type fget_config -A, you should get something similiar to the following if the correct filesets are installed
---dar0---
dac0 ACTIVE dac1 ACTIVE
hdisk7 dac1
hdisk8 dac1
hdisk9 dac0
hdisk10 dac1
hdisk11 dac0
Including a -v as follows will include LUN information
fget_config -Av
Ensure that LVs and filesystems exist on the hdisks mapped out by the array before testing the multipathing failover. Once the filesystems are created and mounted test by unplugging one of the fibre cables from one of the HBA adapters. If filesystems don't exist on the hdisks then it will fail and the output of fget_config produces errors. If the failover works you should be able to cd into each filesystem and the output of fget_config should show only one dac in use, in this case either dac0 or dac1. Also on the DS4700 the LUNs that are on the controller with the disconnect fibre cable should move to the other controller, so all LUNs will be on either controller A or controller B

Configuring etherchannel bonding of ethernet ports

Before bonding the ethernet ports check that the characteristics of the ports to be bonded are correct, such as speed 10, 100 or 1000Mbs, full or half duplex and auto negotiation on or off.
To check the settings on en0 type entstat -d en0 or lsattr -R -l ent0 -a media_speed
To change the settings on en0 you will need to ensure that en0 is down and detached, to do this type
chdev -l en0 -a state=down then chdev -l en0 -a state=detach
Then to change the settings on en0 to 100 full duplex type chdev -l ent0 -a media_speed=100_Full_Duplex
NOTE you cannot do the following, chdev -l ent0 -a media_speed=1000_Full_Duplex, to change to 1000Mbps, this can only be set through auto negotiation as follows chdev -l ent0 -a media_speed="Auto_Negotiation" .
Then bring en0 back up as follows chdev -l en0 -a state=up
Once the characteristics are correct we can proceed to etherchannel bonding the ports together
Type smitty commodev, then move to EtherChannel / IEEE 802.3ad Link Aggregation
and then Add An EtherChannels / Link Aggregation
Select two ethernet adapters in this case ent0 and ent1 and hit return to create bonded adapter en2
To change the configuration to Active/Passive, go to the previous menu then go to
Change / Show Characteristics of an EtherChannel / Link Aggregation
Then change EtherChannel / link aggregation adpaters to ent0 only
And Backup adapter to ent1. Leave the rest as default and press return
Configure the bonded ethernet adapter en2 with an ip-address etc, as follows smitty mktcpip

Creating a RAID 10, mirrored stripe across 6 hdisks using LVM

Create a Volume Group VG and include all 6 disks, type smitty mkvg
Create one or more logical volumes in the volume group, type smitty mklv
Fill out all options as normal with the exception of the following parameters
Physical volume names choose all 6 hdisks
Range of physical volumes maximum
Maximum number of physical volumes to use for allocation should equal 6
Number of copies of each logical partition should be set to 2
To test if the correct LPs have been placed on the correct hdisks type lspv -p hdiskx where x should be replaced with each hdisk number

How to create and remove a linux logical volume and filesystem

Gather information

In this example we would like to add a 10G filesystem to the system volume group, so we need to determine if the volume group (system) has enough free space.
Check the status of the mounted filesystems
df -h

Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/system-root  10G  3.2G  6.9G  32% /                        
udev                    474M  184K  474M   1% /dev
/dev/sda1               102M   40M   63M  39% /boot
/dev/mapper/system-home_lv 2.0G  872M  1.2G  43% /home

Use the vgdisplay command to list the volume groups (VGs) and check the available space in the chosen VG.
vgdisplay

--- Volume group ---
  VG Name               system
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  16
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               4
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               233.66 GB
  PE Size               4.00 MB
  Total PE              59816
  Alloc PE / Size       5428 / 21.20 GB
  Free  PE / Size       54388 / 212.45 GB
  VG UUID               efvr7g-grgf5g-V0SP-45gdf-Xber-VKzj-fg5hf

The "Free PE / Size 54388 / 212.45 GB" line shows that we have 212.45 GB free in this volume group.

Create the logical volume

To create the logical volume use the lvcreate command as follows.
lvcreate -L 10G -n optlv system

Logical volume "optlv" created

This creates a 10GB logical volume named optlv in the volume group called system.

Create a reiserfs filesystem

If you require a reiserfs filesystem then follow this step else skip to the next step to create a ext3 filesystem.
(Reiser filesystems are often used on Suse linux).
mkfs.reiserfs /dev/system/optlv

mkfs.reiserfs 3.6.19 (2003 www.namesys.com)

A pair of credits:
BigStorage  (www.bigstorage.com)  contributes to our general fund  every month,
and has done so for quite a long time.

The  Defense  Advanced  Research  Projects Agency (DARPA, www.darpa.mil) is the
primary sponsor of Reiser4.  DARPA  does  not  endorse  this project; it merely
sponsors it.

Guessing about desired format.. Kernel 2.6.16.21-0.25-smp is running.
Format 3.6 with standard journal
Count of blocks on the device: 2621440
Number of blocks consumed by mkreiserfs formatting process: 8291
Blocksize: 4096
Hash function used to sort names: "r5"
Journal Size 8193 blocks (first block 18)
Journal Max transaction length 1024
inode generation number: 0
UUID: 571e5090-af78-4732-a058-c96596d829bc
ATTENTION: YOU SHOULD REBOOT AFTER FDISK!
        ALL DATA WILL BE LOST ON '/dev/system/optlv'!
Continue (y/n):y
Initializing journal - 0%....20%....40%....60%....80%....100%
Syncing..ok
ReiserFS is successfully created on /dev/system/optlv.

Create an ext3 filesystem

Use the mkfs.ext3 command to make an ext3 filesystem.
mkfs.ext3 /dev/system/optlv

mke2fs 1.38 (30-Jun-2005)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1310720 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
First data block=0
80 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 24 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Mounting the filesystem

Create the directory to mount the new filesystem.
mkdir /opt
Then mount the directory onto the new mount point.
mount /dev/system/optlv /opt
Check the status of the new filesystem.
df -h

Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/system-root  10G  3.2G  6.9G  32% /                         
udev                    474M  188K  474M   1% /dev
/dev/sda1               102M   40M   63M  39% /boot
/dev/mapper/system-home_lv 2.0G  872M  1.2G  43% /home                        
/dev/mapper/system-optlv 10G   33M   10G   1% /opt

Edit /etc/fstab to ensure that the new filesystem mounted on /opt is mounted at system boot.
This example is for a reiserfs. Change reiserfs in the table below to ext3 for an ext3 filesystem.

/dev/system/root     /                    reiserfs   acl,user_xattr        1 1
/dev/sda1            /boot                reiserfs   acl,user_xattr        1 2
/dev/system/swap     swap                 swap       defaults              0 0
/dev/system/home_lv /home                 reiserfs   acl,user_xattr        1 2
proc                 /proc                proc       defaults              0 0
sysfs                /sys                 sysfs      noauto                0 0
usbfs                /proc/bus/usb        usbfs      noauto                0 0
devpts               /dev/pts             devpts     mode=0620,gid=5       0 0
/dev/system/optlv    /opt           reiserfs   acl,user_xattr        1 2

Removing a filesystem and logical volume

Unmount the filesystem.
umount /opt
Then remove the logical volume. Removing the logical volume will also remove the filesystem
lvremove /dev/system/optlv

Do you really want to remove active logical volume "optlv"? [y/n]: y
  Logical volume "optlv" successfully removed

Then edit and remove the entry from /etc/fstab.
Then remove the mount directory.
rmdir /opt

How to resize a logical volume and a reiserfs or ext3 filesystem

Gather information

Type df -h to list the filesystem we would like to increase.

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LVROOT 4.0G  3.4G  480M  90% /

We would like to increase the root filesystem by 500M so we need to determine if the volume group (VolGroup00) listed above has enough free space.
To do this type vgdisplay

--- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  7
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                5
  Open LV               5
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               136.42 GB
  PE Size               4.00 MB
  Total PE              34924
  Alloc PE / Size       5760 / 22.50 GB
  Free  PE / Size       29164 / 113.92 GB
  VG UUID               efvr7g-grgf5g-V0SP-45gdf-Xber-VKzj-fg5hf

The "Free PE / Size 29164 / 113.92 GB" line shows that we have 113.92 GB free in this volume group.
Type lvdisplay to show additional information about the logical volumes on the server.

--- Logical volume ---
  LV Name                /dev/VolGroup00/LVROOT
  VG Name                VolGroup00
  LV UUID                fdsff5-Ma8z-8rr3-5ytgb-fht7-N37h-465yhh
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                4.00 GB
  Current LE             1152
  Segments               2
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:0

Increase logical volume size

In this case we have identified /dev/VolGroup00/LVROOT as being the logical volume to be modified
The current size of the /dev/VolGroup00/LVROOT logical volume is 4G and we need to increase the size to 4.5G
The first step is to increase the logical volume and the second step to increase the filesystem
To increase the logical volume size use the lvextend command as follows
lvextend -L 4.5G /dev/VolGroup00/LVROOT

Increase the filesystem size

Then to increase the size of the filesystem to equal the new size of the LV, do one of the following depending on whether you have an reiserfs or ext3 filesystem.
To check what filesystems you are using type mount

/dev/mapper/VolGroup00-LVROOT on / type ext3 (rw)

We can see in this case we are using the ext3 filesystem

Reiserfs

resize_reiserfs /dev/VolGroup00/LVROOT

ext3

ext2online -d -v /dev/VolGroup00/LVROOT
or
resize2fs -p /dev/VolGroup00/LVROOT

Type df -k and lvdisplay to check that all is as expected

Decrease the filesystem size

ext3

We would like to decrease the size of the /home filesystem by 2GB.
First check the current size.

[root@linux1 /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_linux1-lv_root
                       50G  4.8G   42G  11% /
tmpfs                1002M   88K 1002M   1% /dev/shm
/dev/sda1             485M   29M  431M   7% /boot
/dev/mapper/vg_linux1-lv_home
                       20G  217M   19G   2% /home

Then resize the filesystem before resizing the LV to prevent data loss, however it is still important to backup the fileystem first.
Then try to reduce the size of the /home filesystem as follows.

[root@linux1 /]# resize2fs -p /dev/vg_linux1/lv_home 18G
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/vg_linux1/lv_home is mounted on /home; on-line resizing required
On-line shrinking from 5267456 to 4718592 not supported.

This shows that the filesystem needs to be unmounted first.

[root@linux1 /]# umount /home

Ok lets try again.

[root@linux1 /]# resize2fs -p /dev/vg_linux1/lv_home 18G
resize2fs 1.41.12 (17-May-2010)
Please run 'e2fsck -f /dev/vg_linux1/lv_home' first.

We are still unable to shrink the filesystem until we run the filesystem checker e2fsck

[root@linux1 /]# e2fsck -f /dev/vg_linux1/lv_home
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vg_linux1/lv_home: 212/1318912 files (2.8% non-contiguous), 138179/5267456 blocks

Ok lets have another go.

[root@linux1 /]# resize2fs -p /dev/vg_linux1/lv_home 18G
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/vg_linux1/lv_home to 4718592 (4k) blocks.
Begin pass 3 (max = 161)
Scanning inode table          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The filesystem on /dev/vg_linux1/lv_home is now 4718592 blocks long.

Success!
Lets take a look.

[root@linux1 /]# mount /home
[root@linux1 /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_linux1-lv_root
                       50G  4.8G   42G  11% /
tmpfs                1002M   88K 1002M   1% /dev/shm
/dev/sda1             485M   29M  431M   7% /boot
/dev/mapper/vg_linux1-lv_home
                       18G  217M   17G   2% /home

reiserfs

The following procedure will shrink the reiserfs filesystem mounted on /logs from 20G to 10G

[root@linux1 /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/system-logs
                       20G  33M   20G  1% /logs

[root@linux1 /]# mount
/dev/mapper/system-logs on /logs type reiserfs (rw,acl,user_xattr)
[root@linux1 /]# umount /logs
[root@linux1 /]# resize_reiserfs -s -10G /dev/system/logs
[root@linux1 /]# mount /dev/system/logs /logs

Decrease the LV size

Ok we just need to reduce the size of the LV now.

[root@linux1 /]# lvreduce -L 18G /dev/vg_linux1/lv_home
  WARNING: Reducing active logical volume to 18.00 GiB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce lv_home? [y/n]: y
  Reducing logical volume lv_home to 18.00 GiB
  Logical volume lv_home successfully resized

How to create LVM Snapshot

Read only
Firstly an LVM snapshot volume is created. This snapshot volume collects changes to blocks in the original volume. The blocks that are updated are recorded in an exception table. When a backup is initiated the unchanged blocks are copied from the original LV, any changes to the original LV are recorded in the exception table and the blocks placed into the snapshot volume. The backup then finds these changed blocks in the snapshot volume via the exception table, therefore getting a consistent backup. The snapshot LV needs to be removed after the backup to prevent it from filling up due to continued changes.
The commands to complete a snapshot of an LV are shown below, the VG and LV will need to be changed to suit your environment.

lvcreate -L100M -s -n datalv_snap /dev/datavg/datalv
mkdir /mnt/datalv_snap
mount /dev/datavg/datalv_snap /mnt/datalv_snap
cp or rsync from /mnt/datalv_snap
umount /mnt/datalv_snap
lvremove /dev/datavg/datalv_snap

How to create an LVM2 Logical Volume for swap

To add a swap logical volume /dev/rootvg/swap1lv

1. Create the LVM2 logical volume of size 8GB:
lvcreate rootvg -n swap1lv -L 8G
2. Format the new swap space:
mkswap /dev/rootvg/swap1lv
3. Add the following entry to the /etc/fstab file:
/dev/rootvg/swap1lv swap swap defaults 0 0
4. Enable the extended logical volume:
swapon -v /dev/rootvg/swap1lv
To test if the swap logical volume was successfully created, use
cat /proc/swaps
or
free
or
swapon -s
to inspect the swap space.

Preferred disk layout for Linux servers

With age comes wisdom? After working with SoftwareRAID and Linux servers for a while, I've changed my preferred disk system design and layout.

RAID

Under the old system, I was running a (2) disk RAID1 (mirror) with a hot-spare disk setup and ready for action. But if you're going to have a hot-spare dedicated to the RAID1 array, why not use it as an active array member? That way, if a disk fails, you still have two good disks. Unfortunately, when a RAID element fails, the load from the rebuild process can often kill the one of the remaining disks in the array.

Is it a likely scenario? Probably not. But Linux's Software RAID handles a triple-active RAID1 mirror without any slowdown, so there's not much reason *not* to implement it that way. Plus it's a useful trick to know for situations where you really *do* need to be that paranoid.

(I'm not sure whether any hardware RAID cards provide for a triple-active mirroring RAID1 configuration.)

Partitions

I've also simplified how many partitions I like to have on the disk. My current disk layouts typically look like:

/dev/sdX1 - /dev/md0 - 250MB - /boot
/dev/sdX2 - /dev/md1 - 12GB - / (primary root)
/dev/sdX3 - /dev/md2 - 12GB - / (backup root)
/dev/sdX5 - /dev/md3 - 32GB - /var/log
/dev/sdX6 - /dev/md4 - 2GB - swap
/dev/sdX7 - /dev/md5 - 64GB - /backup/system
/dev/sdX8 - /dev/md6 - (remainder) - LVM area

During normal operations, we boot and run /dev/md1 as our / (root) partition. The /dev/md2 partition is kept offline and is never mounted. Periodically, after validating that the server is in good health, we will copy the contents of /dev/md1 to /dev/md2, make adjustments to /etc/fstab and the hostname. This requires some server downtime (long enough to setup the 2nd root partition).

In the case where the primary OS is hosed, we can boot from the backup OS partition and get back up and running quickly. That gives us the luxury to continue operations until we can schedule downtime to fix the primary OS partition.

Notice that I've broken /var/log out to its own partition. I do this so that an overflowing set of logs won't take the server box down. Plus, by putting the log files in their own physical partition, it's easy to use a boot CD or USB key to gain access to the logs in case of severe issues.

The other physical partition that I consider necessary is /backup/system. This partition is used to hold images of the boot and root partitions, along with information about the partition layout and images of the MBRs. Basically, it's used to store disaster recovery backups. You should not have this partition mounted during normal operations. Taking the contents of this partition offsite is also a good idea. A basic text file of how the backups were created along with information for how to restore these backups is recommended.

Summary

This setup tries to walk the fine line between keeping it simple, but having enough flexibility to deal with a large set of potential failures. Anything from a two-disk failure, to the primary OS being hosed, to both OS partitions having problems all the way up to boot records or the /boot partition being killed.