AnsweredAssumed Answered

VOLUME_ALARM_DATA_UNAVAILABLE for a volume

Question asked by bgajjela on Nov 30, 2016
Latest reply on Dec 1, 2016 by mufeed

Hi ,

 

we are experiencing VOLUME_ALARM_DATA_UNAVAILABLE  for  a volume. Below is the  Investigation so far:

$hadoop fs -ls /
2016-11-30 09:22:52,7787 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:1360 Thread: 4962 Lookup of volume volumename failed, error No data available(61) vtype 0 wantMirror 0 , CLDB: hostname:7222
2016-11-30 09:22:52,7788 ERROR Client fs/client/fileclient/cc/client.cc:4082 Thread: 4962 Failed to fetch attributes for volume volumename, error 61for fid 2049.16.2

 

$hadoop fs -ls /volumename
2016-11-30 09:24:12,2541 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:1360 Thread: 5901 Lookup of volume volumename failed, error No data available(61) vtype 0 wantMirror 0 , CLDB: hostname:7222
ls: `volumename': No such file or directory

 

1) I can see the volume is mounted in mcs

2) The volume in question is having replication factor 1

3) The volume is having below alarms

               {
                        "entity":"volumename",
                        "alarm name":"VOLUME_ALARM_DATA_UNDER_REPLICATED",
                        "alarm state":1,
                        "alarm statechange time":1480517714047,
                        "description":"Volume desired replication is 1, current replication is 0"
                },
                {
                        "entity":"volumename",
                        "alarm name":"VOLUME_ALARM_DATA_UNAVAILABLE",
                        "alarm state":1,
                        "alarm statechange time":1480517714046,
                        "description":"Volume data unavailable"
                }
        ]
}

 

4)$rpm -qa | grep mapr
mapr-hadoop-core-2.7.0.37549.GA-1.x86_64
mapr-spark-historyserver-1.6.1.201605311547-1.noarch
mapr-hive-1.2.201606020917-1.noarch
mapr-hiveserver2-1.2.201606020917-1.noarch
mapr-fileserver-5.1.0.37549.GA-1.x86_64
mapr-zk-internal-5.1.0.37549.GA-1.x86_64
mapr-hivemetastore-1.2.201606020917-1.noarch
mapr-oozie-internal-4.2.0.201603011312-1.noarch
mapr-oozie-4.1.0.201503061405-1.noarch
mapr-core-5.1.0.37549.GA-1.x86_64
mapr-spark-1.6.1.201605311547-1.noarch
mapr-nodemanager-2.7.0.37549.GA-1.x86_64
mapr-pig-0.15.201603241057-1.noarch
mapr-mapreduce2-2.7.0.37549.GA-1.x86_64
mapr-mapreduce1-0.20.2.37549.GA-1.x86_64
mapr-drill-1.8.0.201609121517-1.noarch
mapr-core-internal-5.1.0.37549.GA-1.x86_64
mapr-zookeeper-5.1.0.37549.GA-1.x86_64

 

5) From mfs.logs-3 i can see that there are missing containers for volume , including NameContainer(2434).

fileserver.cc:12414 Verified Missing Containers for Volume.
fs/server/container/containerreport.h:136 ID : 161286316
fs/server/container/containerreport.h:157 cid : 2434 2435 2446
fs/server/container/containerreport.h:162 cid : 2437
fs/server/container/containerreport.h:162 cid : 2457
fs/server/container/containerreport.h:141 ID : 161286316 conmap done

fileserver.cc:12414 Verified Missing Containers for Volume.
fs/server/container/containerreport.h:136 ID : 227716776
fs/server/container/containerreport.h:162 cid : 2626
fs/server/container/containerreport.h:141 ID : 227716776 conmap done
instancemfs.cc:608 ShrinkSlabsAI 0x277ef8000

 

Please let me know how to resolve this issue.

 

Thanks,

Bharath

Outcomes