J.O.A.T.: 06/01/2007

We were given some additional hard drive space for our servers to help mitigate some space issues we were having. I wanted to keep the drive integration as simple as possible so I attempted to utilize the extra space to expand our current drives rather than adding new drives/mount points.

We have two different server environments, Windows 2003 and SUSE 10. Naturally the two systems require a different procedure. The end result, however, is pretty similar ... a drive mounted into the existing file system.

Steps on Windows 2003
Here are the steps I took when adding the new drive in Windows.

initialized the drive as a dynamic disk
formatted the drive using NTFS
assigned a drive letter

The final step was to mount the drive into one of the data-heavy folders. This required a few steps:

stop all programs/services that were accessing the folder in question
rename the folder (I just use a .bak extension)
create a new empty folder using the name of the folder you just renamed
mount the drive into the folder
copy all the files to the now-full folder; if you need to copy permissions use something like scopy or xcopy

Steps on SUSE 10
Our linux server presented a bit more of a challenge. We already had a number of mount points and our new drive was larger than any of our current mount points. I wanted to format the drive as a single partition to use in our most data-heavy folder as well as free up some currently used space to use in resizing the root some space-limited partitions. In all this would require assigning some partitions to different mount points, copying data between different partitions as their mount points change, and even moving some partitions. Here are the general step required:

shut down any programs accessing the partition
go to the partitioner in YaST (system->partitioner)
assign the partition to a new mount point (such as changing /inet to /inet.bak; be sure to leave enough space for a temporary root partition if you need to expand the new root partition ... you can reclaim this space later)
create a new partition on the new drive and assign it to the original mount point
apply the changes and exit YaST
unmount the partition as it will still occupy the old mount point; sudo umount /inet
remount all the partitions; sudo mount -a
duplicate the contents of /inet.bak into /inet; sudo cp --recursive --preserve=all /inet.old/* /inet
restart the system and test for points of failure (particularly with system services)

When you're creating mount points, don't forget to create an empty directory in which to mount it. The only exception being the root partition. Also, if you have partitions mounted as subdirectories you want to make sure you don't copy the contents of those partition. The easiest way to do this is to create a new mount point for the partition you're copying. The sub-mounts will then be absent.

One of the reasons for going through this process was to free up space on the drive that contained the root partition. It has been close to running out of space for a while now and expanding that partition is important. Though it would probably be best to leave the root partition as it is and create a new mount point for the one directory taking up the most space (/usr), we really don't have a lot of flexibility since the system has already been set up and I wanted to dedicate more space to the directory containing our web apps and databases.

Unfortunately, our root partition was created at the end of the drive, meaning it was impossible to expand it. So in order to expand the partition I had to move the root partition so that it was located in the empty space in the middle of the drive. Then I could expand it to use the empty space at the end of the drive.

The biggest problem with this plan is that there is not enough empty space on the drive to create a new partition large enough to hold the content of the root partition. And in order to avoid serious problems I want to keep the current root partition available as a backup in case something goes wrong with the process of creating the new root. Also, you can't resize an active partition, so I also had to use some space on one of the other partitions to create a temporary root partition. So here's the steps I took:

create a subdirectory of our largest partition (/inet) to hold a copy of the root partition
using the free space on the new partition
mount the root partition to a new mount point for copying (/mnt/tmp)
copy the content of /mnt/tmp to /inet/rootdir
go to the partitioner in YaST
assign the empty space on the drive containing the root partition to a new partition and mount it to a temporary location (/mnt/rootnew)
apply your changes and exit YaST
copy the content of /inet/rootdir to the new drive, excluding /usr since it contains too much data
create a symbolic link on /mnt/rootnew that points usr to /inet/rootdir/usr; sudo ln -s /inet/rootdir/usr /usr
go to /mnt/rootnew/etc and sudo edit fstab; change the / mount to /mnt/rootold; change the /mnt/rootnew mount to /
go back into YaST and go into the boot loader (System -> Boot Loader)
create a copy of the primary boot entry; point the original boot entry to point to the new root partition
reboot and cross your fingers

After this I waited a week to ensure that all was well, then completed the process. I neglected to ensure that I had some space available for a temporary root partition, so I had to take a few extra steps. Otherwise the process is as follows:

set up the temporary root partition and boot to it (make sure to edit the fstab to point the the / mount to the temporary root partition and to comment out the old root mount from fstab; also might as well set the default init to 1 in inittab)
from within YaST expand the root partition to occupy the available space on its drive and exit YaST
check the newly expanded file system for errors; e.g. reiserfsck --check /dev/hda6
mount /dev/hda6 (i.e. to /mnt/tmp)
delete the symbolic link for /usr in /mnt/tmp
copy /inet/rootdir/usr to /mnt/tmp/usr
reboot
enter single-user mode: sudo /sbin/init 1
comment out the /inet partition in fstab
unmount /inet
from within YaST delete the temporary root partition and expand /inet to take over the rest of the drive
run a file system check on the /inet partition
reboot and perform some system checks to see if any errors are cropping up (e.g. hit the web server applications)

Post Notes
I found a few commands the helped immensely when attempting to sort out directory sizes when trying to determine how to handle moving the root partition.

df shows partition usage
du shows directory usage; du --max-depth=1 let me see the size of the directories immediately below my current location

The problem with how to handle the root directory due to the lack of drive space wasn't my only source of frustration. Not all went well with the process since I was still trying to figure out what to do and how I was going to do it.

Since I wasn't too familiar with partitioning and mounting in linux I had to go through a bit of trial and error. My biggest mistake came when I was trying to set up the new root. I initially went ahead and tried to re-assign everything through the YaST partitioner, but the process errored out before finishing resulting in a partial update to the system. The fstab file was only partially updated, the boot loader was changed to reflect the new root partition. I'm pretty sure that the reason I ran into problems is because I neglected to set up host directories for the mounts.

Following the errors I made things worse by trying to clear up the issues by rebooting the system rather than finishing the work and making sure everything was set up correctly. And I rebooted before copying any files at all to the new root partition. After reboot the boot loader was looking for system files on the new partition, and since they didn't exist the system wouldn't boot. Once I determined that the boot loader was pointing to the incorrect partition, I used the grub editing capabilities to point the boot command to the correct partition.

Upon getting the system up and running again I ran into a new problem, the filesystem was being loaded as read-only, meaning I was unable to make any changes to the system. My guess is that there was a conflict caused by both the old root partition and new one being mounted at the same time. I was telling grub that the root partition was in one location, but the system was looking for it in another location. The conflict led to the system setting the mount to read-only mode.

Eventually I was able to unlock the root partition by executing sudo mount -o remount,rw /. Once this was done I manually edited the fstab file to mount the partitions correctly and was able to get the system running again. At this point I essentially followed the process described to get the new partition running as the default.

References:
Partitioning & Mounting