Mapreduce tasks failing with 'No space left on device' errors when the cluster has sufficient free space

Document created by jbubier Employee on Feb 13, 2016
Version 1Show Document
  • View in full screen mode

Author: Jonathan Bubier

 

Original Publication Date: July 25, 2014

 

When running a map-reduce job one or more tasks which write to MapR-FS may fail with an error indicating "No space left on device".  An exception similar to the below example can typically be found in the logs for the failed task attempt(s):

java.io.IOException: 2654.1262997.465432632 /benchmarks/TestDFSIO/io_data/5532/test_io_27674966 (No space left on device)

at com.mapr.fs.Inode.throwIfFailed(Inode.java:283)

at com.mapr.fs.Inode.flushJniBuffers(Inode.java:416)

at com.mapr.fs.Inode.syncInternal(Inode.java:492)

at com.mapr.fs.Inode.sync(Inode.java:502)

at com.mapr.fs.Inode.closeWrite(Inode.java:512)

at com.mapr.fs.Inode.close(Inode.java:961)

at com.mapr.fs.MapRFsOutStream.close(MapRFsOutStream.java:215)

at com.mapr.fs.MapRFsDataOutputStream.close(MapRFsDataOutputStream.java:57)

at org.apache.hadoop.fs.TestDFSIO$WriteMapper.doIO(TestDFSIO.java:500)

at org.apache.hadoop.fs.TestDFSIO$IOMapperBaseMy.map(TestDFSIO.java:419)

at org.apache.hadoop.fs.TestDFSIO$IOMapperBaseMy.map(TestDFSIO.java:288)

at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)

at org.apache.hadoop.mapred.Child$4.run(Child.java:282)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1117)

at org.apache.hadoop.mapred.Child.main(Child.java:271)

This error can occur despite the cluster having sufficient free space to run the map-reduce job based on the usage reported in the MCS or from 'maprcli dashboard info'. 

 

From the above exception the first line indicates the File ID (FID) being accessed and the full path to the corresponding file.  Using this example the path is /benchmarks/TestDFSIO/io_data/5532/test_io_27674966 indicating the map-reduce job is the TestDFSIO benchmark.  Further in this example the FID is 2654.1262997.465432632 indicating the chunk being accessed resides within container ID (CID) 2654. This information is important to determine which MapR fileserver (MFS) nodes host replicas of the container in question.  The command 'maprcli dump containerinfo' can be used to display detailed information about one or more containers including the location of each replica. 

Example:

 

# maprcli dump containerinfo -ids 2654 -json

 

{

"timestamp":1406305838093,

"timeofday":"2014-07-25 09:30:38.093 GMT-0700",

"status":"OK",

"total":1,

"data":[

{

"ContainerId":2654,

"Epoch":10,

"Master":"10.10.80.30:5660--10-VALID",

"ActiveServers":{

"IP:Port":[

"10.10.80.30:5660--10-VALID",

"10.10.80.32:5660--10-VALID",

"10.10.80.31:5660--10-VALID"

]

},

"InactiveServers":{

 

},

"UnusedServers":{

 

},

"OwnedSizeMB":"406 MB",

"SharedSizeMB":"0 MB",

"LogicalSizeMB":"726 MB",

"TotalSizeMB":"406 MB",

"NumInodesInUse":713,

"Mtime":"Wed Jun 04 09:55:13 PDT 2014",

"NameContainer":"false",

"VolumeName":"mapr.cluster.root",

"VolumeId":20349717,

"VolumeReplication":3,

"VolumeMounted":true

}

]

}


From the above output the MFS nodes with replicas of CID 2654 are 10.10.80.30, 10.10.80.31, 10.10.80.32 and 10.10.80.30 is the master replica.

The 'No space left on device' error indicates that when writing to FID 2654.1262997.465432632 the MFS node which hosts the master replica of container 2654 responded it does not have sufficient space to accept the write.  This response indicates the underlying storage pool (SP) which contains CID 2654 does not have sufficient space to accept the incoming write(s). 

 

In order to determine which storage pool contains CID 2654 use the following command:

$ /opt/mapr/server/mrconfig -h <host IP> info dumpcontainers | grep "cid:<cid> "

<host IP> must be replaced with the IP of the MFS node and <cid> must be replaced with the desired container ID.  Note that the '-h' option is not needed with 'mrconfig' if the command is executed directly on the corresponding MFS node.  Ex:

$ /opt/mapr/server/mrconfig -h 10.10.80.31 info dumpcontainers | grep "cid:2654 "
cid:2654 volid:20349717 sp:SP1:/dev/sdb spid:cafcd2b8f869745c005255d7ad062877 prev:0 next:0 issnap:0 isclone:0 deleteinprog:0 fixedbyfsck:0 stale:0 querycldb:0 resyncinprog:0 shared:0 owned:52066 logical:93039 snapusage:0 snapusageupdated:1 maxUniq:1840944

 

From the resulting output note the 'sp' field and the corresponding SP name.  In the above example the SP that contains CID 2654 is SP1. 

 

Once the SP is identified review the output of '/opt/mapr/server/mrconfig -h <host IP> sp list' to determine the current usage of each storage pool.  Ex:

 

$ /opt/mapr/server/mrconfig -h 10.10.80.31 sp list

ListSPs resp: status 0:1

No. of SPs (1), totalsize 289918 MB, totalfree 279377 MB

 

SP 0: name SP1, Online, size 289918 MB, free 279377 MB, path /dev/sdb

 

 

To see the current usage for all storage pools on all nodes use the command 'maprcli dump balancerinfo' as an alternative to the above steps.  This command will print the current usage in MB and percentage and total capacity for every storage pool in the cluster.  Ex:

 

$ maprcli dump balancerinfo

ip:port                             usedMB  fullnessLevel  fsid                 spid                              percentage  outTransitMB  inTransitMB  capacityMB

10.10.80.30:5660-                   11527   Average        5467480383183449671  01c27f6188fd2c06005255d7a40ceed5  3           0             0            289918

10.10.80.33:5660-                    512     Average        7470720190951518492  096f163ca1092b1e00538905920c8804  0           0             0            293605

10.10.80.32:5660-                   10288   Average        7239042079423190840  a00b8e0f1c0d6e930052e6c9500709f0  3           0             0            289509

10.10.80.31:5660-                   10541   Average        1661095064916420420  cafcd2b8f869745c005255d7ad062877  3           0             0            289918

 

No active container moves

 

 

Using either of the above methods, if the storage pool is found to have little or no free space available verify that MapR's disk balancer is enabled.  This can be done in two ways:

 

1.  Using maprcli execute the following command: maprcli config load -keys cldb.balancer.disk.paused

Ex:

$ maprcli config load -keys cldb.balancer.disk.paused

 

cldb.balancer.disk.paused

1

 

The return value of "1" indicates that the disk balancer is not enabled.  To enable it execute the following command: maprcli config save -values {"cldb.balancer.disk.paused":"0"}

 

Verify that the configuration is saved and the disk balancer is now enabled: maprcli config load -keys cldb.balancer.disk.paused
Ex:

$ maprcli config load -keys cldb.balancer.disk.paused

 

cldb.balancer.disk.paused

0

 

2. Using the MCS enable the disk balancer from the dashboard by clicking on the colored circle next to "Cluster Utilization" in the upper right corner.  If the circle is yellow the disk balancer is already enabled and no further steps need to be taken to enable the disk balancer.  If the circle is orange or purple the disk balancer is not currently enabled.

 


After clicking on the circle a dialogue box similar to the following should appear:

 

balancer configuration.png

 


Click the 'Off' slider next to "Disk Balancer" to enable the disk balancer.  More information on the disk balancer in MapR and the settings in the above dialogue box can be found on the MapR documentation site at the following URL: http://doc.mapr.com/display/MapR/Configuring+Balancer+Settings

 

Once the disk balancer is enabled, whether using maprcli or using MCS allow for it to run for some time to begin balancing activity.  To monitor the activity of the disk balancer review the output of the command 'maprcli dump balancerinfo -json'.  The output will display the disk usage for all storage pools in the cluster as mentioned above as well as containers that are currently being moved in order to balance out the disk usage.  Once all storage pools have a sufficient amount of free space and the overall utilization is more distributed across the cluster re-run the map-reduce job that was previously failing to confirm it is now successful. 

 

If the job continues to see task failures due to 'No space left on device' errors please contact MapR support at support@mapr.com to investigate the environment further.

 

Attachments

    Outcomes