AnsweredAssumed Answered

Help needed. Service ResourceManager failed in state STARTED: protocol message contained an invalid tag (zero).

Question asked by patrcgit on Jan 18, 2018
Latest reply on Mar 7, 2018 by awisawe

Hi, 

Can someone help me regarding this error? I just installed MapR6 on AWS EC2 on CentOS 7. At first, it was ok but suddenly, Resource manager can't be started due to a corrupted file in hdfs which is /var/mapr/cluster/yarn/rm/system/FSRMStateRoot/RMAppRoot/application_1516194338077_0016/application_1516194338077_0016. I have deleted the file but the resourcemangaer is still unable to start. Here is the latest logs:

 

2018-01-18 19:24:12,791 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics system started
2018-01-18 19:24:12,807 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.RMAppManagerEventType for cla
ss org.apache.hadoop.yarn.server.resourcemanager.RMAppManager
2018-01-18 19:24:12,808 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncherEventTyp
e for class org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher
2018-01-18 19:24:12,808 WARN org.apache.hadoop.metrics2.util.MBeans: Failed to register MBean "Hadoop:service=ResourceManager,name=RMNMInfo": Instance already exists.
2018-01-18 19:24:12,808 INFO org.apache.hadoop.yarn.server.resourcemanager.RMNMInfo: Registered RMNMInfo MBean
2018-01-18 19:24:12,808 INFO org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher: YARN system metrics publishing service is not enabled
2018-01-18 19:24:12,808 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2018-01-18 19:24:12,811 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService: Loading allocation file /opt/mapr/hadoop/hadoop-2
.7.0/etc/hadoop/fair-scheduler.xml
2018-01-18 19:24:12,814 INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state STARTED; cause: org.apache.hadoop.service.ServiceStateEx
ception: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
org.apache.hadoop.service.ServiceStateException: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1034)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1075)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1071)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1071)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1111)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1248)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto.<init>(YarnServerResourceManagerRecoveryProtos.java:3486)
at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto.<init>(YarnServerResourceManagerRecoveryProtos.java:3450)
at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto$1.parsePartialFrom(YarnServerResourceManagerRecoveryProtos.java:3533)
at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto$1.parsePartialFrom(YarnServerResourceManagerRecoveryProtos.java:3528)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto.parseFrom(YarnServerResourceManagerRecoveryProtos.java:3653)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.getAndIncrementEpoch(FileSystemRMStateStore.java:183)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:585)

 

Thanks,

Outcomes