[Discuss] Tired of unreliable external drives... recommendations?

[Richard Pieri]

On 7/10/2012 8:06 AM, Daniel Barrett wrote:
> 1. Are there ANY reliable, quiet, external drives of size >= 3TB today,
> that fit into a safety deposit box (i.e., no RAID appliances) and work with
> Linux?

No.  Disk drives are mechanical devices.  Mechanical devices fail.  Take 
it as read that the drives you use for backups will fail, data will be 
damaged, and you cannot predict when that will happen.

The problem isn't the reliability of the media.  It's the reliability of 
the data on that media.  Insert rant about how RAID isn't about data 
reliability.  That applies to single disks as well.

Here are a couple of things to look into:

A checksum file system.  A ZFS dataset can be configured to make 1, 2 or 
3 copies of every file it stores.  This way if a block becomes corrupted 
it can recover that data from a duplicate copy with a simple scrub 
operation.  Think of it as something like a single disk mirror set.  The 
drawback is that it costs you two or three times as much disk capacity 
to store the data.

A parchive tool to generate error correction data for your backups and 
store them on separate media.  Calculating discrete parity sets is 
time-consuming and it can be a tedious extra step, but like a checksum 
file system the par sets can be used to verify data and recover damaged 
files.  par sets take less space since they're just the checksum and EC 
data, not full replicas.

A backup tool that generates error correction data in parallel with the 
backup such as piping output through rsbep.

None of these are perfect solutions.  Sufficient damage or corruption 
can thwart any error correction mechanism.  But any of them are better 
than nothing.

-- 
Rich P.