Backing up sparse files ... VM's and TrueCrypt ... etc

[jay-R5TnC2l8y5lBDgjK7y7TUQ at public.gmane.org]

This is just one of the reasons I wish they could work out the legal issues between zfs and linux.  The latest version of zfs has block level dedupe, both in the fs and the backup, and both can be configured independently.

------Original Message------
From: Edward Ned Harvey
Sender: discuss-bounces-mNDKBlG2WHs at public.gmane.org
To: 'Tom Metro'
To: 'Richard Pieri'
Cc: BLU
Subject: RE: Backing up sparse files ... VM's and TrueCrypt ... etc
Sent: Feb 18, 2010 8:18 PM

> Richard Pieri wrote:
> >> tar -cf - /path/to/sparse | gzip --rsyncable > sparse.tgz
> >
> > For optimal performance with this trick you should zero out freed
> > sectors on the sparse image and then compact the image.  Very
> > efficient usage of the storage, but it's slow and tedious.
> 
> That would be true if tar read the sparse file as an ordinary file, but
> I'm assuming it knows how to properly skip over the holes. I see there
> is an option to handle sparse files:
> 
>         -S, --sparse
>                handle sparse files efficiently
> 
> I would expect with that option, you'll never see the junk that happens
> to be occupying the unused sectors.

The --sparse option of tar only seems to have any effect when you're
extracting the tarball.  It will look for files with a lot of sequential
zeros, as it extracts them, they will be extracted sparse.  So ...

Tar does in fact do alright at backing up such files, and restoring them.
But only by doing a Full every time.  Not incremental.

One thing I would add here:  When creating a tar backup of a sparse file ...
It appears that tar tries to read the whole file, and during the zero
sections, the system is just generating as many zeros as the CPU can
generate.  This is surprisingly slow compared to what you'd think, but it is
much faster than actually reading zeros from disk.  I forget now exactly,
but I think my disk can read 500 Mbit/s, and reading an all-zero sparse file
went at something like 2 Gbit/s.  That's pretty much all I know.  There are
tons of loose ends, like, could some other compiled version of tar perform
better, and so on.  I just don't know.  I didn't care to explore it, due to
lack of ability to do incrementals.

_______________________________________________
Discuss mailing list
Discuss-mNDKBlG2WHs at public.gmane.org
http://lists.blu.org/mailman/listinfo/discuss



Sent from my BlackBerry® smartphone with SprintSpeed