[Discuss] Help with data migration

[Greg Rundlett (freephile)]

I'm not having good luck with my data migration.  The details are below for
those playing along at home. I'm trying to spend time with my son today, so
I can't say that I'll be able to respond immediately to any comments.  I
can say that I really appreciate all the help on this list.  (Sorry for the
rich-text email, but it helps to convey the information)

Note that this is a consulting project that I'm working on.  If you are
interested in working remotely, or better on-site (Billerica or Salisbury,
MA), to lead the systems administration side of this project you should
reach me at info [@] eQuality-Tech.com

In the past, on boxes I've setup, rsync just flies.  In this case, we've
got 10gE cards and robust hardware, quad core, 16GB RAM in the VM instance
and VMWare ESX running on the iron, and something is just not right.  In
summary, after NFS mounting the source, and switching to cp, I'm getting
speeds as slow as 2.2MB/s for transfer.  rsync "locally" is only doing
about 9MB/s.  I'm not sure how to check where the bottleneck is or what's
wrong.

*lsb_release -a*
LSB Version:
 :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: RedHatEnterpriseWorkstation
Description:    Red Hat Enterprise Linux Workstation release 6.4 (Santiago)
Release:        6.4
Codename:       Santiago

*cat /proc/cpuinfo*
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 1
model name      : AMD Opteron(TM) Processor 6276
stepping        : 2
cpu MHz         : 2300.027
cache size      : 2048 KB
physical id     : 0
siblings        : 4
core id         : 0
*cpu cores       : 4*
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
rdtscp lm constant_tsc rep_good tsc_reliable nonstop_tsc aperfmperf
unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx
hypervisor lahf_lm extapic abm sse4a misalignsse 3dnowprefetch osvw xop fma4
bogomips        : 4600.05
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:


Disk Write performance - destination
*(EMC NAS from svn-srv1)*
I tested disk writes to the NAS. I can *write* over 1GB in 13 seconds (*80.7
MB/s*).  So, the write performance on the NAS is not a problem.

LANG=C dd if=/dev/zero of=test.img bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 13.3035 s, *80.7 MB/s*

*
*
Disk Read performance - source
*(videosrv1)*
I get similar performance *reading* from videosrv1 (*71.3 MB/s)*
*
*

[root at videosrv1 home]# dd if=lucian_home_dir_020713.tgz of=/dev/null bs=1M
295+1 records in
295+1 records out
309876257 bytes (310 MB) copied, 4.34782 seconds, *71.3 MB/s*


Network performance
I already calculated the performance of the transfer end-to-end, and found
that it was too slow (<10 MB/s).  The READ/WRITE tests confirm that the
bottleneck is the network.  So, I enabled the epel and remi repos for
videosrv1 so that I could install *iperf** and run some *network
performance tests*.

The rate is 483 Mbits/sec which is *59 MB/sec*

[root at svn-svr1 tests]# iperf -c videosrv1
------------------------------------------------------------
Client connecting to videosrv1, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[  3] local 172.16.0.25 port 54568 connected with 192.168.2.254 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   576 MBytes   483 Mbits/sec


Although the network is the slowest component, it still is capable of much
more throughput than what I'm witnessing with the rsync transfer.

The solutions that I'm aware of (in order of preference)
a) mount the source file system using NFS so that the transfer is "local"
and network performance is handled by NFS.  *I did this and the performance
is actually worse (see next paragraph).*
b) setup an rsync daemon so the transfer is handled by the rsync protocol
instead of SSH
c) switch to a simpler encryption like arcfour or even disable encryption
using a custom/alternate rsync - since we're transferring on a private
network.

After successfully mounting the source filesystem over NFS, I did a small
test using cp which showed a rate of 29 MB/s which although "slow" is 3
times faster than the rates I've been getting from rsync.  So I switched to
using cp instead of rsync and found that the transfer speed on a couple
larger datasets came in at 2.22 MB/s - 4.3 MB/s

The (source) /home directory is 4.3 TB (4,403.2GB) and I've transferred
only 1.5 TB so far


Greg Rundlett