Backup your photographs

You never know, the accident is closer than you might think. A few days ago my workdisk started having funny noises and thrashing about. I quickly checked that backups was running as they should and just for the heck of it made sure I got a full backup of the whole thing before it went dead.

Then I went out and bought myself a new USB attached disk, 1 TB of capacity is a lot but I am sure me and my camera will fill it soon enough. So I started transferring data and then the old disk just died. Flat out died. Lucky to have a backup on my other disk I started transferring the backup onto the new disk. A few percent in it started sounding bad and then the disk failed.

Not knowing what to do I quickly dismantled my backup disk, another external disk although attached by firewire instead of USB. The disk was so hot I could not touch it so I quickly realized the problem was overheating, it wasn’t really designed for extensive data transfers like this. So I let the disk cool down a bit and rest, then I disassembled it completely so it could run in free air, I used the cabinet fan to cool the hotest circuit, I put ice clamps around the back side of it and the sides and right now it is running fine, I have extracted all my photos up to 2005 so far.

This just goes to show that you can’t be too careful. Keeping your disks cool are important but keeping more than one recent backup is even more important. I have also considered getting a disk to swap with a friend to put backups on to it and I will push and see if she is interested. Then we can back up and meet up bi-weekly or monthly or so and exchange the disk. I am scared of what would happen to my collection of photographs if there was a mishap, burnout, smal fire, accident, electic overload, lightning strike… well there are many things that may potentially damage more than one disk at a time.

We are talking several hundreds of GB here people. CD’s are not an option, a full backup would mean close to 600 CD’s to burn. Just the sheer time of it is incomprehensible, but let’s say a full data CD takes 8 minutes to burn top speed. That’s 80 hours. Two full working weeks.

DVD’s are not much better either, they have a magnitude up on the CD’s or so but it is definitely not an option either, a full backup there would be 150 DVD’s or so, they take longer to burn than CD’s so I bet the time necessary to do that is about the same.

That’s why I use hard disks to store everything and I then schedule backups from one to the other. The whole idea is that a single point of failure should never mean a loss of data. My worries after tonight is that if there is a second point of failure I might be toast. I don’t like that.

Yeah, I got much stuff on my Flickr account, but not the negatives, the RAW files, the working copies, the drafts the ORIGINALS and more important my own notes and stuff. I need a solution for this soon. The broadband ISP has a solution with online storage but only 10 GB. I need at least 60 times that in order to feel safe.

It’s not an easy solution, disks do go bad, the only thing you can do to make sure your data stays around is to copy it and copy it again and keep it current. Digital data screams to be copied of course but it scares me sometimes how easily I could lose everything I have done.

And don’t talk to me about RAID systems. I have trouble shooted and fixed enough electronic systems to know that when stuff runs together it needs only to be an overvoltage problem in a single power supply and your whole raid stem is fried.

My backup copying is now at May 2005. Perhaps I will make it this time as well. I swear, I will buy a new backup disk on salary day.

I use tar as my backup software almost exclusively and here is the trick:
Every week or two weeks or when you think it is necessary you make a full backup
Every time there has been changes or just daily to backup the changed files since the last full backup

Using tar this is pretty simple, in the first case

# A full backup of my entire home catalogue

tar c -f backup.tar /home/ichi

# I normally never compress my archives. There are two reasons for this and
# the most important is that there is a better chance to save the contents
# if the file is damaged if it is not compressed. The second reason is because
# of the large amount of data compressing / decompressing takes a lot of time
# the full backup is already several hours of work.

# An incremental backup since my last full backup

tar c -x -N 2008-10-21 -f backup-partial-`date -I`.tar /home/ichi

# The -N flag backs files that were touched after this date (and on this date).
# so in this example the last full backup was at 2008-10-21. All files touched on
# this date and later will then be backed up.
# The `date -I` is a nifty trick inserting todays date in the file name