This thread is to discuss imaging/drive backup tools for Linux partitions.

Here are some of the available free tools:

Partimage
http://www.partimage.org/Main_Page

CLonezilla Live
http://www.clonezilla.org/

CloneZilla Live purports to be a Norton Ghost alternative, so I'd recommend using that. Partimage is used by Clonezilla, which adds a better user interface.

I haven't put Clonezilla through its paces yet, and still use an old copy of Ghost (version 8-ish) for partition backups, which handles Linux partitions fine.

Ghost is commercial, but the only version I use is the "DOS" mode version, which boots from a CD into its own GUI environment, which is neither Win or Linux, so you're not supporting Windows or MS using Ghost ;).

Acronis has an imaging tool also, but I haven't used it.

Other tips/suggestions re; Linux partition/drive imaging?

A great HOWTO is here (http://gentoo-wiki.com/HOWTO_Backup). Definitely read the part about dd'ing your MBR.

I've been meaning to try Bacula for some time now. Any feedback on that one?

DD is the best way to go, to image everything...

then you have stuff like
gpartd = bootable partition editor..

and
rsync, for backing up to servers


also setting up some scripts in cron to run at specified times is good

I'm also a fan of running dd against the entire drive or partition. I run a myth frontend and a virtualized (kvm) myth backed on my file server. The myth front end uses a 4GB compact flash card and a sata compact flash adapter that sits in a PCI slot location. I can simply power down the HTPC, slap the CF card into a USB adapter and create an image using dd. If something happens to the CF card, I can replace it without opening the case. If you're interested, the SATA CF can be seen at:

http://www.addonics.com/products/flash_memory_reader/adsacf.asp

dd is an awesome tool and works perfectly for imaging, but it has some limitations when it comes to system backups.

First, if you have, say, an 80GB partition (or drive) with maybe only 10-20GB of data on it, dd will still read the full 80GB and the resulting file will be 80GB, also. Although the file may be compressed or dd's output may be piped through a compression utility, it's still time-consuming, and should that *one file* become corrupt, you've lost the entire partition (or drive).

Second, if you want to backup your whole system and change the partitioning scheme and/or filesystems used, then you can't use dd since it copies the underlying filesystem as well as the data; and, even though there may be only 10-20GB of data, you would still need 80GB of space to get it there.

So you need some good backup software to provide you with that level of functionality. I haven't done much research into what all is available, but I've had my eye on Bacula (http://www.bacula.org/en/) for some time now because I'm impressed with its feature set and its scalability. I've yet to try it, but that will soon change, though, as I plan to backup my current single-drive install (with default partitioning and ext3) to an external hard drive, add a couple of more drives, and restore to XFS RAID0.

Yeah, I don't bother with any fancy tools to copy my personal data. Just plug in a USB drive and just do a plain old copy of the data over to the external drive. That way I can get to individual files without a bunch of hassles.

OTOH, I haven't come up with a way that I like for backing up the OS. Right now, I just relie on reinstalling from scratch. Which isn't too terrible except for having to redo all the little customizations of the window manager and having to reinstall all the windows apps into wine. :(

I just got through using Clonezilla, thanks to Rgb's suggestion. I wanted to make an image before I did an update from Ubuntu 7.10 to 8.04.

Background -- I downloaded the latest version of Clonezilla-Live and burned it to a CD-R. I used a 500GB external HDD (eSATA & USB case w/SATA drive) - because I had it. I'm running Linux on a HP laptop with a 160GB internal drive (I replaced the original 80GB one, which still has the Windows XP stuff on it). I first had to figure out how to use "gparted" to format the external drive into 2 partitions, one Linux and one FAT32 (for compatibility use). It wasn't obvious, but I figured it out by reading the instructions followed by trial and error - the instructions were not correct (when taken literally).

My observations on Clonezilla: It works just fine following the default path through the menus. I had about 68GB of stuff on the laptop drive. The first problem that I ran into was that Clonezilla would not recognize my eSATA adapter and mount the HDD as /dev/sda1, so I had to switch to the much slower USB 2 connection (which did mount the drive without a problem).

The entire task took 2 hours & 59 minutes for the total internal disk (using the default format - it said it was supposed to be a .tar.gz file, but it wasn't). The direct copy of my data files (about 95% of the contents) took less than an hour to the FAT32 portion of the HDD, using the eSATA connection.

After the image was finished, I found that I could not make a copy of it. This means that a Clonezilla image can't be moved to another location, or storage medium. This is a big detraction for the program, in my opinion. Unless, of course, I was doing something wrong. I did read the use instructions, but couldn't make a lot of sense out of them until I started actually using the program. Even then, a lot of the terms used did not make a lot of sense, which is why I used the defaults.

PS -- 25+ years ago I was able to wade through the UNIX instructions, and MAN pages, for a specialized PC system that I was the Administrator for (no one else wanted the job, although the vendor and our IT guys did help me over the sticking points). The MAN pages weren't always correct either.

Conclusion -- I will continue to look for a different way to make a disk image. Clonezilla doesn't do the job I want it to, and the way I want it to make a backup that is useable for my purposes. The last backup software that really did the job I wanted, was the one supplied with Windows 95/98.

BTW, the upgrade to Ubuntu 8.04LTS went without a hitch (it does look like I will have to reset the keyboard defaults - I can't use the numeric keypad right now). There may be some other changes that will need to be updated and I will do them as I run across problems.

Yeah, I don't bother with any fancy tools to copy my personal data. Just plug in a USB drive and just do a plain old copy of the data over to the external drive. That way I can get to individual files without a bunch of hassles.

OTOH, I haven't come up with a way that I like for backing up the OS. Right now, I just relie on reinstalling from scratch. Which isn't too terrible except for having to redo all the little customizations of the window manager and having to reinstall all the windows apps into wine. :(
All user settings are in one place, the home directory. Just back up your home directory, and when you restore it to a new OS, all your settings will follow. If you modify any system-wide settings, the configuration will be stored in /etc, so you can either backup individual config files (e.g. xorg.conf for your global X settings), or the entire directory.

A simple way to improve your current system is to use rdiff-backup. It will have the same result (a mirror of your files on the external drive), but it will be faster because it only copies what has changed. It supports incremental backups, And it is simple to setup a cron job to automate the backup process.

OK, I'm setting up the bacula director on my gentoo box so I can bare-metal restore the htpc after I pick up the other drives.

This is basically the backup routine I use:

http://ubuntuforums.org/showthread.php?t=35087

I like it because it matters not what the partition size was/is. Also handy for moving between systems and makes a very small file.

...

A simple way to improve your current system is to use rdiff-backup. It will have the same result (a mirror of your files on the external drive), but it will be faster because it only copies what has changed. It supports incremental backups, And it is simple to setup a cron job to automate the backup process.

Speaking of rdif-backup, I've noticed that that Keep uses it for it's backend. Perhaps I'll give Keep a try.

This is basically the backup routine I use:

http://ubuntuforums.org/showthread.php?t=35087

I like it because it matters not what the partition size was/is. Also handy for moving between systems and makes a very small file.
Do you specifically mean the first post, which uses tar for backup? Or some other post in that 563 post thread? Do any of the other 562 posts contribute anything useful to the discussion?

I just got through using Clonezilla, thanks to Rgb's suggestion. I wanted to make an image before I did an update from Ubuntu 7.10 to 8.04.

BTW, the upgrade to Ubuntu 8.04LTS went without a hitch (it does look like I will have to reset the keyboard defaults - I can't use the numeric keypad right now). There may be some other changes that will need to be updated and I will do them as I run across problems.

CT-

Thanks for doing the Clonezilla exercise and reporting back- now I don't have to ;)

You might want to try Ghost- unfortunately, it looks like the free imaging tools need more time to mature re: user interface and fault tolerance.

Rgb -- Thanks for reminding me. I had said that my laptop's numeric keypad didn't respond. A toggle of the "num lock" switch and a shut-off seemed to have fixed it because, at the next boot, it worked as it should. There were one or two other minor things (I've forgotten which ones) that behaved the same way.

Phantom Gremlin -- I believe he means only the first post. I scanned through some of the other posts, but did not find them informative.

What I've done with my data files (plus the ".mozilla" and ".mozilla-thunderbird" folders - which contain my browser and email profiles & data) is to just open a File Browser window (set to show hidden files), right click on the appropriate folder, and select "Create Archive". That lets me backup a compressed (.tar.gz, or other format, like .zip) copy of the files and save them to an external HDD. The external HDD is formatted as a Fat32 partition, for Windows compatibility, if needed (using "gparted" as the partition manager - another "discovery" I made in stumbling around).

I plan on using Heliode's method (the first post referenced) to save the OS files. I do like not having (and understanding of) control over the backup processes, especially since I'm a linux newbie and am still trying to get more familiar with the Linux OS use.

Although I'm not a stranger to the use of the "Terminal" mode, I've been so GUI-ized over the years, that if I can do the job via a GUI, I would prefer it. So far I like the standard Ubuntu Gnome version (they call it Nautilus, I think ?) - I haven't tried the KDE version at all.

Do you specifically mean the first post, which uses tar for backup? Or some other post in that 563 post thread? Do any of the other 562 posts contribute anything useful to the discussion?

Yes, the first post describes the how- to- use "tar" to backup and restore. This is the main method used on the tape backups system that have been used for years and year in *nix world. It is also also used as the most common archiver program. The other posts concern minor discussions on some of the temp files and directories that don't or do need to be backuped up but I think the first post was edited to include all that discussion.

Tar really is the most viable solution for a backup system. It works extremely well and lends itself to being very versitile. At my institution we use tar in combination with TSM which is an IBM server based backup solution. Ultimately tar is all a single user really needs.

One way that I do it is have cron run a backup script with the following in it:

cd /home
tar -czf 'date +%y%b%dhomedirs.tar.gz' *
mv 'date +%y%b%dhomedirs.tar.gz' /var/backups
cd /var/backups
rm -f `date --date='5 days ago' +%y%h%d*`

This allows me to keep the last 5 days worth of backups and discard the rest. You could set that variable to anything you wanted but the point is that this is likely the best way for you to do your backups in linux. This cron job runs every night around 2am.

Tar really is the most viable solution for a backup system. It works extremely well and lends itself to being very versitile. At my institution we use tar in combination with TSM which is an IBM server based backup solution. Ultimately tar is all a single user really needs.

One way that I do it is have cron run a backup script with the following in it:

cd /home
tar -czf 'date +%y%b%dhomedirs.tar.gz' *
mv 'date +%y%b%dhomedirs.tar.gz' /var/backups
cd /var/backups
rm -f `date --date='5 days ago' +%y%h%d*`

This allows me to keep the last 5 days worth of backups and discard the rest. You could set that variable to anything you wanted but the point is that this is likely the best way for you to do your backups in linux. This cron job runs every night around 2am.

Looks like a great way for /home backups.

What about multi-gigabyte images typical of OS partitions, i.e. the / partition/mountpoint, or wherever the OS/apps/drivers are installed on $YOUR_DISTRO.

Would tar be sufficient for an OS image backup? Any issues with hidden/system files or root privileges?

What about simply settting up a cron rsync job to peridcally duplicate changes to a mirror backup on another disk?

Looks like a great way for /home backups.

What about multi-gigabyte images typical of OS partitions, i.e. the / partition/mountpoint, or wherever the OS/apps/drivers are installed on $YOUR_DISTRO.

Would tar be sufficient for an OS image backup? Any issues with hidden/system files or root privileges?

What about simply settting up a cron rsync job to peridcally duplicate changes to a mirror backup on another disk?

Honestly the best way to ensure data integrity due to failure is having a complete RAID 5 array with a hotspare... imo. This allows for drive failure and ensures that the failed disk is replaced with the hot spare immediately giving you time to replace the broken disk. On top of this it is then advisable to employ an Rsync solution as well (as you discuss) which does a compare and backup based on changes to the source being backed up.

This solution is not difficult to setup these days with all the modern linux distros supporting software RAID. The trouble for some can be cost for the disk space or the time in learning how to setup this stuff.

Now the minimum size of a RAID 5 set is 3 disks plus the hot spare. You lose one disk to the RAID set and of course the hotspare is not useable disk space. So you need 4 disks and only two of those are usable. Now if you grow beyond those 4 you are adding storage space. Disk space is not that expensive any longer so you can pickup 4 320gb drives for about 400+change and you would have a 720gb completely backed-up RAID set that is fault tolerant to one disk. Technically you could have 2 disks go and your system would still be operational just a little taxed. Also most home towers can accomodate at least 4 disks inside.

The Rsync side of this solution allows you to have a remote backup solution; and by remote I mean off-site. If you are just going to rsync to another machine in the same room then you might as well invest in a RAID 50+1 solution... but I digress. Rsync is also good for people who want a snapshot backup daily that they can refer to. This allows you to keep go back in case of a corrupt file or incase of a file deletion. The RAID set does not allow for this operation and it is not designed for that. It is designed for fault tolerance.

This is a solution I suggest for people in a bigger production environment which insists upon data integrity and redundancy.

What I do at home is much simpler. I have two drives running in a RAID 1 set (mirror mode) running software RAID through my distro. The mobo I have has a RAID bios but given that I don't trust consumer grade RAID cards I would rather just use software instead. Heck I don't even backup my home dirs because there isn't enough change to bother so I don't worry about my data loss due to error.

The solution I posted above (about taring files) actually is employed on one of my production servers to provide daily snapshots of development.

So to finalize my ramblings... you need to consider what you are trying to do when backing up. Are you trying to ensure date integrity due to failure or data integrity due to corruption or user error or both? When you determine your need you can proceed to an applicable solution.

Tar really is the most viable solution for a backup system. It works extremely well and lends itself to being very versitile. At my institution we use tar in combination with TSM which is an IBM server based backup solution. Ultimately tar is all a single user really needs.

One way that I do it is have cron run a backup script with the following in it:

cd /home
tar -czf 'date +%y%b%dhomedirs.tar.gz' *
mv 'date +%y%b%dhomedirs.tar.gz' /var/backups
cd /var/backups
rm -f `date --date='5 days ago' +%y%h%d*`

This allows me to keep the last 5 days worth of backups and discard the rest. You could set that variable to anything you wanted but the point is that this is likely the best way for you to do your backups in linux. This cron job runs every night around 2am.

It appears these commands backup the /home directory and everything below it.

What would be the appropriate tar command(s) to backup the OS/ apps/ drivers in a typical Ubuntu/ Xubuntu/ Mythbuntu load, i.e. everything *except* user data?

The point is to be able to take a snapshot of the OS/drivers/apps/settings before installing a new driver/app/config file change/automatic update so that if the change does something bad, you can restore the OS and all its settings/apps drivers back to the state before you tried the change. This is particularly important for the Automatic updates systray function in Ubuntu, which brings down lots of random updates, any one of which can hose a working system- nothing against the Updates funtion or the Ubuntu guys- it's the nature of software engineering and software changes and all the sensitivities/dependencies inherent in complicated software systems.

Rgb. from the post early in this thread you can use tar like this:
tar cvpzf backup.tgz \
--exclude=/proc \
--exclude=/lost+found \
--exclude=/backup.tgz \
--exclude=/mnt \
--exclude=/sys \
/


This backups everything needed from the root and all it's folders. I also exclude the /media directory due to CDs or USB drives possibly being mounted. Also note the backup file *should * be place in a folder that is NOT being backed up, otherwise tar will keep trying to backup its backup file!
My main server runs a script like this nightly and I always backup my system using tar before an update.

I'd like to breathe some life back into the clonezilla live solution. At work we use Clonezilla Live to save and restore Windows Images from 160GB drives that have about 18GB of data on 2 partitions that are NTFS and FAT. It takes about 30 to 45 minutes to 'save' the image and about 10 minutes to 'restore' the image over a 1GB link. The image is around 8GB in size. I am sorry to say I have not done it with Linux but it would probably work better than with Windows. We choose the gzip and dd options to save and restore images via a drive mounted via sshfs. I have done the same using an USB hard drive and it is only about 40% longer and has been very reliable. You can restore to any drive as long as it is the same or larger. It cannot (yet) restore an image to a smaller drive.

I use RAID. Once a month or so, I take one drive out of the array and put it in the safe deposit box. Then I take the one out of the box and put it into the array.

Before installing OS updates, you should image the drive as discussed before.

Over the weekend, I tested a recent Ghost (edit: v11.0.x) to make an image backup and restore of Mythbuntu 8.04.

It appeared to work fine, no issues I could see.

Use the "DOS" bootable version, which boots into its own "DOS" gui. Select Partition-> Image file, then select the / partition, then select where to write the image file and give it a name.

I like to write image files to FAT32 partitions, so I made a ~20G fat32 partition on the same drive as / and swap, an 80G OS drive.

The FAT32 partition was created in Acronis Disk Director (= gparted/Partition Magic, but appears "better"), at the "end" of the drive. Then I installed Mythbuntu 8.04 with the wizard, using the "Guided, use available unused space" pick at the partioner prompt. It automagically sets up / and swap, with a ~2G swap and the rest for /.

This set up works well for Linux, as the FAT32 image file storage partition is not auto mounted, so its hidden in normal use. When you need to restore it, just boot from the Ghost boot disk, and restore the image to /. It takes less than 5 minutes in the test I did, better than the approx 30min it takes to re-install Mythbuntu from scratch with the Nvidia driver and other tweaks, even longer if you've add a lot of apps.

Of course, you should also burn the image file(s) to DVD-R. Writing to FAT32 has another benefit- the image file is split into ~2G chunks, and the partition can be seen with any OS back to Win98 and any known DOS/freeDOS utility without drivers. The easiest way to burn the image file without having to mount the FAT32 partition in Mythbuntu is to boot with a PuppyLinux 3/4 liveCD and use its built in quick mount and burn utilities.

The image method avoids the nuances of needing to know what directories to exclude from tar due to mounted devices or filesystems, eliminates permissions issues, etc.

Rgb. from the post early in this thread you can use tar like this:
tar cvpzf backup.tgz \
--exclude=/proc \
--exclude=/lost+found \
--exclude=/backup.tgz \
--exclude=/mnt \
--exclude=/sys \
/


This backups everything needed from the root and all it's folders. I also exclude the /media directory due to CDs or USB drives possibly being mounted. Also note the backup file *should * be place in a folder that is NOT being backed up, otherwise tar will keep trying to backup its backup file!
My main server runs a script like this nightly and I always backup my system using tar before an update.


I finally settled on using this technique for my backups.
The actual command I use is:

tar cvpzf backup.tgz \
--exclude=/proc/* \
--exclude=/lost+found/* \
-- exclude=/media/* \
--exclude=/mnt/* \
--exclude=/tmp/* \
--exclude=/sys/* \
/


Very similar with some minor changes. I figure there's no point in keeping the tmp or mnt files and I keep all the directory structure, but not the files in the excluded directories (probably doesn't matter in the real world, I'm just being a little anal here). I store my backups on an external USB hard drive and since I exclude "media", I don't need to exclude the backup file too.

I only backup after installing a bunch of new stuff (all my user data is
backed-up by other means, this is just my OS backup technique). Since I always use Synaptic to install things, I copy the "History" and paste it into the "Notes" section of the backup file. Then can always tell what new stuff was installed in that backup file.

There is an app in the Ubuntu repo's called "Simple Backup Suite for desktop use". It is as named, very simple and effective. In Synaptic search for sbackup.

Something else I've been meaning to try is remastersys:

http://klikit.pbwiki.com/Remastersys

I use the tar method as well, except I use bz2 for better recoverability, and I make separate files for the user subdirectories, because otherwise if you get a bad archive you can lose everything. If I could atomize it further practically, I would.

I just do it manually on the first of each month (so I'm sure), and transfer the files to another machine using sshfs. Then I do a dist-upgrade. This system has worked fine for many years; I still have all my emails from '98.

I don't care for the image backup method for several reasons:
- Backing up the OS is unnecessary;
- One glitch and the whole img is shot;
- Can't easily access individual files for restore;
- Can only be restored to the same size drive.

BTW, a reminder to those who only have settings in their home: your home directory is where -all- your work, photos, music, etc belongs. Only a very few personal files belong outside here, notably in /usr/local. Sure, back up /etc, and /usr/local too.

My main server backups every night and it sends the error file to me via email (if there is one) - every morning. This way I know I don't have any bad backups or bad archives and if I don't receive that e-mail in the morning I know something is wrong ! :)

I add this to the end of the backup script:

--snip--
--exclude=proc \
. > /dev/null 2> /mnt/raid0/systembackup/backup.err

# now send email

cat /mnt/raid0/systembackup/backup.err | mail Michaez@somewhere.com -s"$(hostname) Nightly Backup Error"

# all done !

Hopefully you have it set to check the archive for integrity. Archive errors can happen without showing in a finishing error code.

It appears Clonezilla, a FOSS Ghost/Acronis alternative, does the job-

http://clonezilla.org/

I used it the past couple of weeks to image a niece's Ubuntu load, then restored it back to another build, as I wanted to re-use all the work put into the app loads/tweaks/etc (see Official Ubuntu 8.10 thread for screencaps of the load for the niece, with the wbar dock).

The restore worked perfectly, and the bare PC was up and running with the restored load in about 15 minutes, no issues- Ubuntu 8.10 autodetected all the different hardware, though I used the same video card (FX5200).

Be sure you understand Linux drive/partition naming conventions
http://www.linuxplanet.com/linuxplanet/tutorials/4269/3/

before using Clonezilla.

If you're willing to pay for offline storage then I found out about this place rsync.net (http://www.rsync.net/index.html) while listening to some old episodes of the Gutsy Geeks radio show.

Offline storage sites that support Linux are few and far between and this one is a lot more than just a "backup" site. You can even mount your space as a drive and use it directly.

BTW, this thread should probably be sticky. I had a hell of time finding it even though I knew it existed.

I also learned about this app from Gutsy Geeks:

TimeVault (https://wiki.ubuntu.com/TimeVault)

This app doesn't seem to be in the main repository so it looks like you'll have to get it from launcpad if you want it.


The interesting difference with most backup tools is the following:

It adds a new tab to the "properties" dialog for files so that...

"Restore functionality is integrated into Nautilus - previous versions of a file or directory that has a snapshot can be accessed by examining the properties and selecting the 'Previous Versions' tab."

Yeah, a lot of people will find it gimicky. But it sure would make restoring files a lot easier for most users. For instance, with this app your wife or kids, who likely don't know how you're doing the backups or even where they're being stored would be able to restore a file without asking for your assistance.

If you're willing to pay for offline storage then I found out about this place rsync.net (http://www.rsync.net/index.html) while listening to some old episodes of the Gutsy Geeks radio show.

Offline storage sites that support Linux are few and far between and this one is a lot more than just a "backup" site. You can even mount your space as a drive and use it directly.

BTW, this thread should probably be sticky. I had a hell of time finding it even though I knew it existed.

I looked into rsync.net and it looked good, but at least at the time it was a little pricey compared to some other sites for the size of files I wanted to backup. I use SpiderOak, which works well for me (you can't mount the backup, but it does have syncing and sharing functions in addition to backup). Their block-level deduplication and encryption is nice. Jungle Disk also supports Linux and does allow mounting...

I looked into rsync.net and it looked good, but at least at the time it was a little pricey compared to some other sites for the size of files I wanted to backup. I use SpiderOak, which works well for me (you can't mount the backup, but it does have syncing and sharing functions in addition to backup). Their block-level deduplication and encryption is nice. Jungle Disk also supports Linux and does allow mounting...

I can't recommend rsync.net enough. I'm not sure when you last looked at their pricing, but they do a price reduction every 6-12 months, and when they do this, existing customers get space added to their account to bring their price per GB in line with the new pricing.

So a few years ago I was paying $1.20 or $1.60 per GB, but now I am paying something like 60 cents per GB (I have a bit of a quantity discount in there I think).

If you can find any reason to do business with these people, do it. They're fantastic.

Just an update on Clonezilla.

http://clonezilla.org/

Since my original post, Clonezilla has improved GREATLY.

I now use it exclusively for imaging my drives/OS partitions.

When I first posted, Clonezilla Live (CZL) was still buggy, wouldn't boot properly on some machines, or wouldn't work with some partition types well.

Since then, CZL has been revved many times and now works better than Ghost or Acronis for me. No more need for commercial stuff (or Hiren's ;) ).

I can't recommend rsync.net enough. I'm not sure when you last looked at their pricing, but they do a price reduction every 6-12 months, and when they do this, existing customers get space added to their account to bring their price per GB in line with the new pricing.

So a few years ago I was paying $1.20 or $1.60 per GB, but now I am paying something like 60 cents per GB (I have a bit of a quantity discount in there I think).

If you can find any reason to do business with these people, do it. They're fantastic.
I think that's their prices per month right? I pay $6.25 a month for 100GB ( so a little over 6 cents per GB per month). So that's a lot less...