AnsweredAssumed Answered

Increasing the speed at which blocks are replicated

Question asked by davidjericho on Apr 15, 2013
Latest reply on Apr 16, 2013 by davidjericho
I've got a cluster that's taking some time to replicate data. At most, it appears I'm doing about 10MByte/s while replicating to a recently added node.

Between the nodes using either a web download of a file or iperf, I get 10Gbits/s throughput reliably so bandwidth isn't the issue. Speed when writing to the MapR NFS server on the localhost, and over a network is fast as well.

The cluster appears to be be transferring at about 600 kbyte/s to 1 Mbyte/s per replication thread to a new node, and my mfs.log is full of messages like:

<pre>
2013-04-09 00:02:42,3559 INFO  containerrestore.cc:406 x.x.0.0:0 Returning EAGAIN for FSID 2094340271159275139, 192.168.0.100:5660,  replicacid 2617, srccid 2617 because container need slow resync and all slowresync slots are full
</pre>

Looking at network monitoring tools, and even the node information, the node is replicating at roughly 16 times 1MB/s, give or take.

I know in non-MapR Hadoop land, hdfs-site.xml has a setting that can be modified to increase this number, but MapR M5 does not have this file.

Can I increase the concurrency to get aggregate throughput, increase the throughput put thread, or do both to get throughput? I have plenty of network whitespace I'd like to see utilised, as fast replication is of interest.

Outcomes