You don’t understand that having backups of important data is a good idea before you’ve missed something very important. I have twenty years of experience with hard drives and learned very early that backups are a very good idea.
So people keeps asking me how I administrate my data. Now here’s the quick story:
1. Apple Time Machine
It does a constant backup of my system drive. With several versions and very easy restore. It has saved me a couple of times already.
2. Mirrored RAID on the media server
I have a couple of large disks with my video, audio and images. Stuff that I don’t want to loose. These disks are configured as something that’s called RAID 1. Or in plain english: each disk is actually two disks. On the computer they show up as one, but everything that is stored on them is actually stored on two disks. See the four disks in the image above? It’s not four terrabytes of data. It’s two. Giving instant backup. If one drive fails I simply replace it and the system will rebuild it. To do this you can use the built in RAID support in Mac OSX, but that is lame, so I use SoftRaid. And yes, when I buy a new external hard drive I buy two. Meaning twice the security and twice the price.
The RAID system has also saved me a couple of times. It’s not a question of “if” a drive fails. It’s “when”.
3. Portable hard drive
I also want an off site backup of my most important files. A backup that will survive burglary or a fire in my home. This is mainly for my images. For that I have a two layer system. First I have a drive that I fill up and keep in a drawer at work.
4. Network based backup
Of course I don’t bother to bring the portable disk back and forth very often, so if everything fails at home I would probably loose months of images. That’s why I have an extra layer of remote backup for the last year of data. A system that immediately starts uploading files to a remote server as I add images and important files. There are several systems for this. Back in the days when I used Windows I relied on Carbonite. Another good solution is Mozy. But I use JungleDisk. Because I trust Amazon and their encryption. And because it gives me a virtual drive that lets me access my files at all times from any computer.
That’s it. Time Machine. RAID1. Portable disk and JungleDisk. Share your backup strategy and fire away with questions in the comments.
eirikso 
I have two laptop’s. Every latop has it’s own external Firewire disk running Timemachine. The biggest external disks also contain all my photos lib. The smallest external disk is @work and also contain a backup of all my photo-lib’s.
Unable to use any network based backup since here in NZ we have to PAY for net usage! That is all going to change when I move back to Norway in the end of this year, Rsync is then going to be my new best friend 🙂
Assuming it’s all working, this is what I have.
Resources needing backed up:
o Work machine
o Photos
o Code
o General files
o Music
o Videos
Work machine
My laptop needs to be running for me to work. So I backup a full bootable image to an external WD Passport drive nightly. I have the WD Passport drive partitioned into two logical drives, so I have two days worth of bootable images on it. I use SuperDuper! to do this, and I can’t recommend it enough.
This has saved me two times since I began the automated process in January. I work on a MacBook, but also have a Mac Mini as a media slave computer…kind of to play music and sync to my iPhone and such. The bootable image from my MacBook has actually allowed me to boot my Mac Mini using it – saving my bacon and allowing me to work a couple times when the MacBook was in the shop.
Photos
I recently moved from a NAS-based photo solution to putting all the photos directly on my MacBook. I had to finally cave and start using iPhoto since Picasa is always coming to the Mac “later.” iPhoto is usable with 20k local images. Not so much over the wire.
So the local images get backed up with the aforementioned SuperDuper! image. I’m also using Jungle Disk to store them remotely.
Code
I mostly don’t worry about it anymore. Current stuff backed up nightly with SuperDuper! Git as my source code repository.
General files
SuperDuper! Eventually I’ll throw them on the Jungle Disk as well.
Music
I recently broke down and changed up my FLAC music repository for MP3. I wanted to save filespace. The ultimate goal was to allow myself to start buying music digitally (DRM-free iTunes or Amazon). Here’s where it gets messy.
I have music on an Ubuntu box working as a sort of NAS. Then I also have an Ubuntu “backup” box that uses rsnapshot to backup important stuff from the NAS. So rsync backs up my entire digitized music library weekly and saves two versions.
Downside: These two boxes sit right next to each other. And I’ve pretty much quit making monthly external drive backups to bring offsite. I need to put this on my list to improve – though I’m not sure my odds of success. Don’t want to send this to Jungle Disk. Wonder if lala.com will let me download my synced music? (I don’t think so.) Perhaps another WD Passport hooked to the mini (opposite side of the house) is sufficient.
Note: This rsnapshot stuff used to be much more prevalent in my setup. I’d store a lot of files on the NAS, and they’d be versioned to beat the band…back forever. Then I went Mac. And now I’m on a quest to tear down all of this home-infrastructure I’ve built up.
Videos
I’m at a loss. The only real backup I do is to have a digital and tape version of DV video. Any Flip or still camera video is just digital and at risk. Jungle Disk seems too expansive (and slow) of an option. Each DV tape amounts to 13GB, and even today that can go fast.
I almost wonder if two DVD copies of content (one saved and one sent to family) would be the best option?
I have set up a ZFS filesystem (configured to work as a jbod) which is shared on the [W]LAN via authenticated Samba/CIFS and IP-restricted NFS simultaneously) on FreeBSD 7.
All computers in the home use this for backup-relevant data (XP/Vista supports to move parts of or the entire users “My Documents” to a different location, and in my case this is a networked drive, which works completely transparent.
So, by doing this I have all important data stored in one location, but using high-risk jbod (not even raidz[1,2], but that will arrive when I get myself a few more disks in the storage server). By having all this data in one place makes it easy to send backups over the internet to a backup host, which I have in Oslo (I live in Bergen). Four times every day an rsync-job runs and keeps the backup location 100% synchronized with my home system. Currently 180GB and nearly 250.000 files are kept in sync every time, and a sync without many changes takes about 30s.
Since this backup solution runs on the server, it is not dependant of the different machines and laptops actually being turned on.
Quickly summed up I use offsite mirroring, with local central storage. As you say, the disks in my storage server WILL fail at one moment, I just have not experienced it yet (which is rather strange, since I’ve used computers since I was eight years old using an 20MB HDD (Tandon XT 4,77MHz) back in 1986. More than 20 years of faultfree disk performance on consumer hardware, even having run on the IBM “DeathStar” disks in their heyday.
There you have it.
A couple of hours since I posted this story and we already have lots of interesting information in the comments. Thank you very much!
I love this:
“I have set up a ZFS filesystem (configured to work as a jbod) which is shared on the [W]LAN via authenticated Samba/CIFS and IP-restricted NFS simultaneously) on FreeBSD 7.”
…say that to your grandmother…
My first HD was also a 20 MB box. For my Amiga 500 back in 1987 or something. I have been running consumer grade disks 24/7 on my server at home for the last 10 years and have had two or three disks breaking completely down.
Hot tip: when your hard drive starts making noises that you’ve never heard before you should back up immediately.
Your strategy is almost the same as mine, except I don’t use Time Machine. At the moment, I’m concidering dropping the external FireWire disks and only use a network-based solution. However, backing up somewhat 2.5 TB of data on an 3 mbit/s upload will take bloody ages. At least the first time. Besides, I haven’t found a provider delivering 2.5 TB diskspace for an affordable price. Any ideas?
It depends on what you call affordable. 🙂
Mozy Unlimited is supposed to be just that. Unlimited. For $5,- a month. With today’s dollar that would amount to about NOK 1,50. Hahaha.
For those of you that still have a windows centric home network I recommend looking into Windows Home Server. It does full backup of your client pc’s and it’s NAS functionality allows for mirroring of data between physical disks (Very easy to set up and manage compared to RAID). You can also install an add-in from Jungle disk allowing for remote backup on S3.
@Martin
Buy a cheap server and place it at work or at a friends house. Use a rsync based backup solution (rsnapshot is great), it handles big data sets very well. You can make the first backup over local network and then move the server.
Thanks for the tip on Mozy, Eirik. Is it reliable?
Buying a server to house 2.5 TB of data is not going to be cheap in any way, I imagine. Still, backup to several FireWire disks is not cheap either. That’s why I’m now concidering the network backup only.
I’ve only tested Mozy on a basic level. Looked very good. I don’t know anything about reliability or speed on their servers. JungleDisk let you back up to Amazon’s servers in Europe. Giving better speed. But Amazon will end up pretty expensive with 2,5 TB.
You could contact @noplay on Twitter. I think he’s currently using Mozy.
I know that some of the “unlimited” services are not completely unlimited. Something that sometimes is noted in small print somewhere… Mozy has a pro service as well, and I don’t know if they will consiter 2,5 TB a “small office” or a “home” 🙂
You could contact them through mail. That’s always a good idea before buying a service. If they don’t answer they’re probably not good at support either…
Very true indeed, Eirik. I’ll do some further research.
Thanks for the information! 🙂
@Martin
You can buy a cheap server at ebay or finn, put in a sata pci card and add disks. You don’t need a fast server at all. I used a laptop with 1GHz cpu, 512MB ram and external disks running FreeBSD for backup a year ago and had no problem with speed.I doubt you find a cheaper off site solution.
My dad is running Mozy in addition to my rsnapshot script. He has 250 000files and a total of 40GB. In the log I found the two last backup runs. One backed up 630MB in 7hours and on backed up 6MB in 1,5hour. It’s on a slow Telenor adsl line, but i don’t think Telenor is the one to blame. I don’t have access to my rsnapshot logs right now so I can’t compare them.
For photos, I like to burn DVDs. They are cheap and easy to stash away at secure places. I put the last ~4GB of photos on them which usually provides some redundancy over multiple discs. Who knows about long-term integrity, though.
Please don’t judge me by my website… it’s been under wraps for a while… hey guys—don’t you believe in write once read many?!? This is what I do, besides the normal backup procedure. Version everything. Use subVersion. Use Time Machine or use something dammit! RAIDs are great, but hardware expenses are high—reserve your RAIDs for video, use physical media for backup, and make it *incremental*—this is how you deal with 40Tb backups!! Mahalo, Adam.
I’d like to ad another category: Archiving (+ back-up).
Currently, being photographers, we shoot loads of images (each at least 23 MB) per day. These cannot be thrown away to minimize size, because clients might request another image. Even those that seem imperfect, may still be asker for a different composition or to fix another image/composition.
We used to write all to DVD (x2/3) but due to ever growing file sizes, this has become unmaintainable. We’ve tried a RAID solution (Lacie 2.5TB), but they are expensive, no conclusive information on what to do when everything does go wrong and, seriously, the Lacie biggest made too much noise to comfortably work with.
So this is what we’ve adopted now: a WiebeTech RTX-200H-QJ.
The beauty of this device is that it is ‘Trayless’. Which mean that you can insert naked harddrives, as if it were a floppy (well, more a SyQuest, for those who remember).
Currently we have 5 500GB drives (+ back-up drive) and 3 750GB drives (+back-up) to manage all our files. (of course, not all disks are filled, but getting close)
Cheers,
Maarten Van Coile
Cool. That TrayFree + Trayless Technology looks neat!
And how is the noise level from these disks? I need drives that are very quiet…
[…] My current backup strategy | eirikso.com: A nice explanation of how one media producer stores his backup, with other suggestions in the comments. […]
[…] posted my current backup strategy a while ago. And after a quick chat with Rodrigo on Twitter I decided to post my image workflow as […]