AnsweredAssumed Answered

How to resolve No space left on device error when creating volume?

Question asked by wilki2021 on Jun 24, 2012
Latest reply on Jun 24, 2012 by wilki2021
I am attempting a simple 3-node cluster using Ubuntu 12.04 VMs in VirtualBox.  All instructions in the quick-start guide were followed and the cluster is partially operational, but I am seeing issues related to the above error message.  Because this is for proof-of-concept only (and I am limited in disk space because it is all allocated off a single disk), I used the alternate method shown in the documentation that does not require formatting physical disks. My disktab file contents are:

# MapR Disks Sun Jun 24 07:49:51 2012
/root/storagefile 5FC948DE-C86F-2202-F593-0CED0CE74F00

and the actual file is a 3G file shown here (this replicated on each of 3 nodes):

$ sudo ls -l /root
-rw-r--r-- 1 root root 3047108608 Jun 24 19:38 storagefile

This is the output of the 'volume list' command - some volumes were created but new creates fail.

maprcli volume list -columns volumename,mountdir,quota,used
quota  mountdir                             logicalUsed  used  volumename                       
0                                           0            0     mapr.cldb.internal               
0      /                                    0            0     mapr.cluster.root                
0      /hbase                               0            0     mapr.hbase                       
0      /var/mapr/cluster/mapred/jobTracker  0            0     mapr.jobtracker.volume           
0      /var/mapr/metrics                    0            0     mapr.metrics                     
0      /var/mapr/local/rtw-mapr-node3/logs  0            0     mapr.rtw-mapr-node3.local.logs   
0                                           0            0     mapr.rtw-mapr-node3.local.mapred 
0      /var/mapr                            0            0     mapr.var                         
0      /var/rtw                             0            0     rtw-test                         

Here is example of error:

2012-06-24 18:53:13,035 INFO org.apache.hadoop.mapred.TaskTracker: Checking for local volume. If volume is not present command will create and mount it. Command invoked is : /opt/mapr//server/createTTVolume.sh rtw-mapr-node2 /var/mapr/local/rtw-mapr-node2/mapred/ /var/mapr/local/rtw-mapr-node2/mapred/taskTracker/
2012-06-24 19:05:04,483 ERROR org.apache.hadoop.mapred.TaskTracker: Failed to create and mount local mapreduce volume at /var/mapr/local/rtw-mapr-node2/mapred/. Please see logs at /opt/mapr//logs/createTTVolume.log
2012-06-24 19:05:04,483 ERROR org.apache.hadoop.mapred.TaskTracker: Command ran /opt/mapr//server/createTTVolume.sh rtw-mapr-node2 /var/mapr/local/rtw-mapr-node2/mapred/ /var/mapr/local/rtw-mapr-node2/mapred/taskTracker/
2012-06-24 19:05:04,484 ERROR org.apache.hadoop.mapred.TaskTracker: Command output
2012-06-24 19:05:04,486 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start TaskTracker because org.apache.hadoop.util.Shell$ExitCodeException: Failed to create local mapred volume.
Command used: /opt/mapr/bin/maprcli volume create -name "mapr.rtw-mapr-node2.local.mapred" -localvolumehost rtw-mapr-node2 -localvolumeport 5660 -shufflevolume true -rereplicationtimeoutsec 300 -replication 1

and in createTTVolume.log
ERROR (10003) -  Volume create mapr.rtw-mapr-node2.local.mapred failed, No space left on device

I was able to mount the file system through nfs successfully, but again any file creation operation failed with the 'no space left on device' error.  In the mounted file system, it presents itself like this through 'df' command (not sure if this is supposed to be accurate - can't explain the 15% usage):

rtw@rtw-VirtualBox:/mapr/my.cluster.com/var$ df -k .
Filesystem     1K-blocks   Used Available Use% Mounted on
-                2662400 376832   2286592  15% /mapr/my.cluster.com

Any clues or tips appreciated.  Wondering if there is a potential issue because each disk is only 3G or if something else is going on?  Thanks.

Outcomes