AnsweredAssumed Answered

spark crashes resource manager - com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero)

Question asked by waynetheron on Jan 19, 2018
Latest reply on Jan 21, 2018 by maprcommunity

Hi

 

Busy evaluating Mapr for a large financial exchange and in my demo env i am running into an issue. here are the machine details and issue:

 

16gb mem

8 cores

mapr version 6.0.0.20171109191718.GA

single node cluster

 

Everything is working fine except there seems to be an issue with resource manager which seems to crash node manager. Spark is able to run a hive query but i get  a crash exception when showing data that mentions the resource manager is not available. In the MCS console i note the resource manager and node manager crash after i get this error.

 

the error is

INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state STARTED; cause: org.apache.hadoop.service.ServiceStateException: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
org.apache.hadoop.service.ServiceStateException: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
    at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1034)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1075)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1071)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1071)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1111)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1248)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
    at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
    at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
    at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto.<init>(YarnServerResourceManagerRecoveryProtos.java:3486)
    at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto.<init>(YarnServerResourceManagerRecoveryProtos.java:3450)
    at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto$1.parsePartialFrom(YarnServerResourceManagerRecoveryProtos.java:3533)
    at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto$1.parsePartialFrom(YarnServerResourceManagerRecoveryProtos.java:3528)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
    at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto.parseFrom(YarnServerResourceManagerRecoveryProtos.java:3653)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.getAndIncrementEpoch(FileSystemRMStateStore.java:183)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:585)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
    ... 10 more
2018-01-19 09:57:16,807 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning to standby state
2018-01-19 09:57:16,807 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to standby state
2018-01-19 09:57:16,807 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
org.apache.hadoop.service.ServiceStateException: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
    at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1034)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1075)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1071)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1071)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1111)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1248)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
    at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
    at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
    at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto.<init>(YarnServerResourceManagerRecoveryProtos.java:3486)
    at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto.<init>(YarnServerResourceManagerRecoveryProtos.java:3450)
    at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto$1.parsePartialFrom(YarnServerResourceManagerRecoveryProtos.java:3533)
    at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto$1.parsePartialFrom(YarnServerResourceManagerRecoveryProtos.java:3528)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
    at org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos$EpochProto.parseFrom(YarnServerResourceManagerRecoveryProtos.java:3653)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.getAndIncrementEpoch(FileSystemRMStateStore.java:183)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:585)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
    ... 10 more

Outcomes