Wednesday, March 2, 2011

Recovering data from a broken hard disk

Some time ago, my computer's hard disk broke and I didn't had a recent backup copy which increased my pain.
I hope no computer dependent ever faces such crappy situation. Just like a burnt child dreads fire all life, I have made it a habit to backup my system data at the end of each day (backintime is a super fast tool) and monitor the healthiness (SMART status) of my hard disk's regularly!

Here I describe how I recovered my data in that situation. Most of the commands mentioned here will require root privileges.

It happened one fine day when I started my computer and after making some changes on a set of files, tried to save them for testing. System refused and reported that my file system was read-only. Without thinking much I tried to restart my system and it never opened up in GUI. I ended up with unmountable partitions-

mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory

Target filesystem doesn't have requested /sbin/init.
No init found. Try passing init= bootarg...

and then it asks me to enter commands in Busybox.

On much googling I found this somewhat useful.
I decided to use my Ubuntu live CD to recover the data, check the drive's status and reinstall the system if possible. The disk utility reported two bad sectors and when that happens, there are more to follow.
Broken disk was not really bad for my PC was still under warranty but I was worried about my data.
In panick I decided to use the dd command to create an image of the partition on a blank external drive.
I got this error:
ubuntu@ubuntu:/$ sudo dd if=/dev/sda3 of=/media/2c0adbca-f37e-4f53-b975-8c7a99ee2757/sda3.imgdd: reading `/dev/sda3': Input/output error5341584+0 records in5341584+0 records out2734891008 bytes (2.7 GB) copied, 162.043 s, 16.9 MB/s
Then Danni suggested me to use ddrescue as a tool to copy my disk off. ddrescue is like dd, just that it retries, skipping bad bits.
So I tried to install ddrescue onto the live CD but each time apt-get kept getting stuck while unpacking, in D-state (ps aux | grep dpkg showed the unpacking process' status  Ds+ which means uninterrupted sleep.) Processes in D-state cannot be killed until system is rebooted and are termed as zombie processes.

Dismounting the corrupted partition didn't help either, I'm not sure why.
The syslogs kept displaying IO errors on device sr0 which is the cd-rom usually, so I instead burnt Ubuntu rescue remix (which has many rescue tools including ddrescue pre-installed).
Then in hurry and without knowing the consequences of writing a large file onto a MS-DOS file system (which was what this external drive was formatted with), I started creating the corrupted file system's image with the rescue remix.

My disk was spacious enough but the process halted reporting disk to be out of space.

My bad, I doubted this was because of mismatching file system on my external drive and thought of  formatting my external drive with two partitions (for the first time):  ext4 for the image and NTFS for the rest. For this I studied the fdisk and parted commands extensively. I eventually could make a large enough ext4 partition but something went wrong with magic numbering when fiddling with fdisk. The image was ready within a second which smelled of something wrong! :-/

The reason for image creation faliure turned out to be the disk's file system. MS-DOS restricts files sizes from being more than 4GB[?] in size. Danni told me I could save the image (which is nothing but a byte-wise copy) to any file system with larger file sizes permitted (not necessarily ext*), so I deleted all partitions and made a single fully stretched one (using fdisk utility). I formatted it with NTFS whose data I would be able to see using any OS like this:
mkntfs /dev/sdc
where sdc was the external hard disk (mount -l command shows currently mounted partitions or df -T could also be used to check this).

Then, with all other partitions unmounted (umount command) and only the external disk mounted, I ran the ddrescue command. The steps were:
  • Mounting the external hard disk file system I was writing the image to (sdc1 in my case) if not yet mounted:
    mkdir /mnt/target
    mount /dev/sdb1 /mnt/target
  • Writing an image of corrupted partition (sda3 in my case) to the file system mounted:
    ddrescue /dev/sda3 /mnt/target/ddrescue.img /mnt/target/ddrescue.log
This ways, I was able to create my image with corrupted data skipped.
Checking the image using e2fsck couldn't succeed and I needed to run fsck manually:
fsck -y /mnt/target/ddrescue.img
FWIW, my image on check (without repairing it) reported:
error: Inodes that were part of a corrupted orphan linked list found
Now, rerunning check reported the file-system's copy to be clean and (hopefully) repaired and ready to mount.

I tried it using mount loop (which is just a way to mount a file as a file system) as:
mount -o loop /mnt/target/ddrescue.img /mnt/recovered_sda3
where /mnt/target/ddrescue.img was treated as device and /mnt/recovered_sda3 was the desired mount point directory. Now I was able to use  /mnt/recovered_sda3  just like root directory with all my data in /mnt/recovered_sda3/home! :D
I tried to format my corrupted partition and even to delete the entire partition on the faulty disk but could not do it.
Finally, I installed Ubuntu on a new hard disk afresh, mounted my external drive onto it, mounted the recovered partition's image and copied my /mnt/recovered_sda3/home onto /home :
 rsync -auvxz --progress /mnt/recovered_sda3/home/ /home/
(keeping the trailing '/' intact)

With that, had my computer back to what it was before the snap!

Had I had faced this situation on a healthy hard disk (as sometimes happens if the computer suddenly shuts down while booting) and if direct checking and repairing the file system using e2fsck/fsck failed, I could have copied the repaired image of the partition bytewise back to the same device like this:
dd if=/mnt/target/ddrescue.img of=/dev/sda3
Thank you Danni for guiding me through this and saving me a lot of  time and Google searches.