AnsweredAssumed Answered

Is My Cluster Performance Good?

Question asked by pratap20 on Apr 10, 2015
I have 7 node cluster with 2x6 core cpu & 128 RMA on each node & 12x4TB disk storage on each node.

After installing MapR  M7 with version 3.1.1 I run Test DFSIO on 1 node  & got got below performance .Kindly confirm wheather it is good enough or I need to work more on it again ? kindly confirm.  I also tried to copy 15 Gb files from local system to MapR cluster.
Apreciate quick response.
[root@test02 ~]# hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-test.jar TestDFSIO -write -nrFiles 10 -fileSize 100
TestDFSIO.0.0.4
15/04/06 13:57:44 INFO fs.TestDFSIO: nrFiles = 10
15/04/06 13:57:44 INFO fs.TestDFSIO: fileSize (MB) = 100.0
15/04/06 13:57:44 INFO fs.TestDFSIO: bufferSize = 1000000
15/04/06 13:57:44 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
15/04/06 13:57:44 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 10 files
15/04/06 13:57:44 ERROR fs.MapRFileSystem: Failed to delete path /benchmarks/TestDFSIO/io_control, error: No such file or directory (2)
15/04/06 13:57:45 INFO fs.TestDFSIO: created control files for: 10 files
15/04/06 13:57:45 ERROR fs.MapRFileSystem: Failed to delete path /benchmarks/TestDFSIO/io_data, error: No such file or directory (2)
15/04/06 13:57:45 ERROR fs.MapRFileSystem: Failed to delete path /benchmarks/TestDFSIO/io_write, error: No such file or directory (2)
15/04/06 13:57:45 INFO fs.JobTrackerWatcher: Current running JobTracker is: blbdcprod05/192.168.252.35:9001
15/04/06 13:57:45 INFO mapred.FileInputFormat: Total input paths to process : 10
15/04/06 13:57:45 INFO mapred.JobClient: Creating job's output directory at maprfs:/benchmarks/TestDFSIO/io_write
15/04/06 13:57:45 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/benchmarks/TestDFSIO/io_write/_logs
15/04/06 13:57:45 INFO mapred.JobClient: root, realuser: null
15/04/06 13:57:45 INFO mapred.JobClient: Running job: job_201504052322_0001
15/04/06 13:57:47 INFO mapred.JobClient:  map 0% reduce 0%
15/04/06 13:57:54 INFO mapred.JobClient:  map 30% reduce 0%
15/04/06 13:57:55 INFO mapred.JobClient:  map 90% reduce 0%
15/04/06 13:57:56 INFO mapred.JobClient:  map 100% reduce 0%
15/04/06 13:58:00 INFO mapred.JobClient:  map 100% reduce 100%
15/04/06 13:58:01 INFO mapred.JobClient: Job job_201504052322_0001 completed successfully
15/04/06 13:58:01 INFO mapred.JobClient: Counters: 26
15/04/06 13:58:01 INFO mapred.JobClient:   Job Counters
15/04/06 13:58:01 INFO mapred.JobClient:     Launched reduce tasks=1
15/04/06 13:58:01 INFO mapred.JobClient:     Aggregate execution time of mappers(ms)=16928
15/04/06 13:58:01 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
15/04/06 13:58:01 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
15/04/06 13:58:01 INFO mapred.JobClient:     Launched map tasks=10
15/04/06 13:58:01 INFO mapred.JobClient:     Data-local map tasks=10
15/04/06 13:58:01 INFO mapred.JobClient:     Aggregate execution time of reducers(ms)=2748
15/04/06 13:58:01 INFO mapred.JobClient:   FileSystemCounters
15/04/06 13:58:01 INFO mapred.JobClient:     MAPRFS_BYTES_READ=4130
15/04/06 13:58:01 INFO mapred.JobClient:     MAPRFS_BYTES_WRITTEN=1048591211
15/04/06 13:58:01 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=569492
15/04/06 13:58:01 INFO mapred.JobClient:   Map-Reduce Framework
15/04/06 13:58:01 INFO mapred.JobClient:     Map input records=10
15/04/06 13:58:01 INFO mapred.JobClient:     Reduce shuffle bytes=92
15/04/06 13:58:01 INFO mapred.JobClient:     Spilled Records=100
15/04/06 13:58:01 INFO mapred.JobClient:     Map output bytes=754
15/04/06 13:58:01 INFO mapred.JobClient:     CPU_MILLISECONDS=6250
15/04/06 13:58:01 INFO mapred.JobClient:     Map input bytes=260
15/04/06 13:58:01 INFO mapred.JobClient:     Combine input records=0
15/04/06 13:58:01 INFO mapred.JobClient:     SPLIT_RAW_BYTES=1100
15/04/06 13:58:01 INFO mapred.JobClient:     Reduce input records=50
15/04/06 13:58:01 INFO mapred.JobClient:     Reduce input groups=5
15/04/06 13:58:01 INFO mapred.JobClient:     Combine output records=0
15/04/06 13:58:01 INFO mapred.JobClient:     PHYSICAL_MEMORY_BYTES=6201434112
15/04/06 13:58:01 INFO mapred.JobClient:     Reduce output records=5
15/04/06 13:58:01 INFO mapred.JobClient:     VIRTUAL_MEMORY_BYTES=29111275520
15/04/06 13:58:01 INFO mapred.JobClient:     Map output records=50
15/04/06 13:58:01 INFO mapred.JobClient:     GC time elapsed (ms)=0
15/04/06 13:58:01 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
15/04/06 13:58:01 INFO fs.TestDFSIO:            Date & time: Mon Apr 06 13:58:01 BDT 2015
15/04/06 13:58:01 INFO fs.TestDFSIO:        Number of files: 10
15/04/06 13:58:01 INFO fs.TestDFSIO: Total MBytes processed: 1000
15/04/06 13:58:01 INFO fs.TestDFSIO:      Throughput mb/sec: 696.3788300835655
15/04/06 13:58:01 INFO fs.TestDFSIO: Average IO rate mb/sec: 700.0659790039062
15/04/06 13:58:01 INFO fs.TestDFSIO:  IO rate std deviation: 50.614042431940696
15/04/06 13:58:01 INFO fs.TestDFSIO:     Test exec time sec: 16.337
15/04/06 13:58:01 INFO fs.TestDFSIO:
[root@test02 ~]# hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-test.jar TestDFSIO -write -nrFiles 10 -fileSize 1000
TestDFSIO.0.0.4
15/04/06 14:04:42 INFO fs.TestDFSIO: nrFiles = 10
15/04/06 14:04:42 INFO fs.TestDFSIO: fileSize (MB) = 1000.0
15/04/06 14:04:42 INFO fs.TestDFSIO: bufferSize = 1000000
15/04/06 14:04:42 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
15/04/06 14:04:42 INFO fs.TestDFSIO: creating control file: 1048576000 bytes, 10 files
15/04/06 14:04:42 INFO fs.TestDFSIO: created control files for: 10 files
15/04/06 14:04:43 INFO fs.JobTrackerWatcher: Current running JobTracker is: blbdcprod05/192.168.252.35:9001
15/04/06 14:04:43 INFO mapred.FileInputFormat: Total input paths to process : 10
15/04/06 14:04:43 INFO mapred.JobClient: Creating job's output directory at maprfs:/benchmarks/TestDFSIO/io_write
15/04/06 14:04:43 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/benchmarks/TestDFSIO/io_write/_logs
15/04/06 14:04:43 INFO mapred.JobClient: root, realuser: null
15/04/06 14:04:43 INFO mapred.JobClient: Running job: job_201504052322_0002
15/04/06 14:04:44 INFO mapred.JobClient:  map 0% reduce 0%
15/04/06 14:04:51 INFO mapred.JobClient:  map 90% reduce 0%
15/04/06 14:04:53 INFO mapred.JobClient:  map 100% reduce 0%
15/04/06 14:04:58 INFO mapred.JobClient:  map 100% reduce 100%
15/04/06 14:04:58 INFO mapred.JobClient: Job job_201504052322_0002 completed successfully
15/04/06 14:04:58 INFO mapred.JobClient: Counters: 26
15/04/06 14:04:58 INFO mapred.JobClient:   Job Counters
15/04/06 14:04:58 INFO mapred.JobClient:     Launched reduce tasks=1
15/04/06 14:04:58 INFO mapred.JobClient:     Aggregate execution time of mappers(ms)=27300
15/04/06 14:04:58 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
15/04/06 14:04:58 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
15/04/06 14:04:58 INFO mapred.JobClient:     Launched map tasks=10
15/04/06 14:04:58 INFO mapred.JobClient:     Data-local map tasks=10
15/04/06 14:04:58 INFO mapred.JobClient:     Aggregate execution time of reducers(ms)=2706
15/04/06 14:04:58 INFO mapred.JobClient:   FileSystemCounters
15/04/06 14:04:58 INFO mapred.JobClient:     MAPRFS_BYTES_READ=4168
15/04/06 14:04:58 INFO mapred.JobClient:     MAPRFS_BYTES_WRITTEN=10485775627
15/04/06 14:04:58 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=569492
15/04/06 14:04:58 INFO mapred.JobClient:   Map-Reduce Framework
15/04/06 14:04:58 INFO mapred.JobClient:     Map input records=10
15/04/06 14:04:58 INFO mapred.JobClient:     Reduce shuffle bytes=93
15/04/06 14:04:58 INFO mapred.JobClient:     Spilled Records=100
15/04/06 14:04:58 INFO mapred.JobClient:     Map output bytes=773
15/04/06 14:04:58 INFO mapred.JobClient:     CPU_MILLISECONDS=13870
15/04/06 14:04:58 INFO mapred.JobClient:     Map input bytes=260
15/04/06 14:04:58 INFO mapred.JobClient:     Combine input records=0
15/04/06 14:04:58 INFO mapred.JobClient:     SPLIT_RAW_BYTES=1100
15/04/06 14:04:58 INFO mapred.JobClient:     Reduce input records=50
15/04/06 14:04:58 INFO mapred.JobClient:     Reduce input groups=5
15/04/06 14:04:58 INFO mapred.JobClient:     Combine output records=0
15/04/06 14:04:58 INFO mapred.JobClient:     PHYSICAL_MEMORY_BYTES=6230532096
15/04/06 14:04:58 INFO mapred.JobClient:     Reduce output records=5
15/04/06 14:04:58 INFO mapred.JobClient:     VIRTUAL_MEMORY_BYTES=29333237760
15/04/06 14:04:58 INFO mapred.JobClient:     Map output records=50
15/04/06 14:04:58 INFO mapred.JobClient:     GC time elapsed (ms)=0
15/04/06 14:04:58 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
15/04/06 14:04:58 INFO fs.TestDFSIO:            Date & time: Mon Apr 06 14:04:58 BDT 2015
15/04/06 14:04:58 INFO fs.TestDFSIO:        Number of files: 10
15/04/06 14:04:58 INFO fs.TestDFSIO: Total MBytes processed: 10000
15/04/06 14:04:58 INFO fs.TestDFSIO:      Throughput mb/sec: 830.6337735692333
15/04/06 14:04:58 INFO fs.TestDFSIO: Average IO rate mb/sec: 873.5184326171875
15/04/06 14:04:58 INFO fs.TestDFSIO:  IO rate std deviation: 254.85826527309658
15/04/06 14:04:58 INFO fs.TestDFSIO:     Test exec time sec: 15.599
15/04/06 14:04:58 INFO fs.TestDFSIO:


For 15 Gb files copy from local disk to mapr disk
time cp ./test_pratap /mapr1/test_pratap
I got below result:
real    0m19.544s
user    0m0.028s
sys     0m13.595s

real    0m18.609s
user    0m0.022s
sys     0m12.834s

real    0m18.568s
user    0m0.020s
sys     0m13.175s

real    0m21.459s
user    0m0.034s
sys     0m16.126s

real    0m21.439s
user    0m0.018s
sys     0m15.052s

Thanks in advance

Outcomes