AnsweredAssumed Answered

Performance with 24 drives

Question asked by user888 on Aug 23, 2013
Latest reply on Sep 2, 2013 by Ted Dunning
I am trying to test performance using NFS and hadoop API.

The server is a pretty beefy server with 24 cores, 96GB of memory. Attached storage is 24 drives (JBOD). Each drive can do about 150MB/sec , these are 4TB enterprise grade drives. I tried concurrent reads on all 24 drives at once using dd's and I get close to 3GB/sec in aggregate.

I am trying to read 1Ge files from the MaprFs NFS filesystem. The NFS filesystem is mounted on localhost.

All chunks are local to that server, so I am not using the network at all.




<code>
[root@~]# for i in {1..10} ; do dd if=/mapr/test/test$i.zip iflag=direct of=/dev/null bs=1024k count=1024& done
[root@ ~]# 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 17.9504 s, 58.4 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 18.0461 s, 58.1 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 18.1585 s, 57.7 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 18.1727 s, 57.7 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 18.1887 s, 57.6 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 18.2644 s, 57.4 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 18.3378 s, 57.2 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 18.3362 s, 57.2 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 18.3689 s, 57.1 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 18.3928 s, 57.0 MB/s
</code>

As you can see the performance is very disappointing. I am getting ~ 570 MB/sec in aggregate from 24 drives.

I can see 8 storage pools online with 3 drives in each storage pool.

I am using the default chunksize and trying to avoid compression by using the .zip suffix on each file. How can I improve this performance and get performance close to hardware speeds.

Thanks.

Outcomes