AnsweredAssumed Answered

Streamsets+MapR: stream stage UnknownTopicOrPartitionException: could not get TopicInfo

Question asked by reedv on Jan 18, 2018
Latest reply on Jan 19, 2018 by maprcommunity

Trying to use streamsets to create a pipeline from a maprFS origin to a mapr stream destination, but getting an error that seems to indicate that the stream can't be found. The error reads:

Error getting metadata for topic '/path/to/my/stream/mytopic' from MapR Streams. Reason: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: could not get TopicInfo, err 22

The pipeline in in cluster batch mode and is correctly configured for hadoop impersonation (eg. the pipeline can run in cluster batch mode when using a maprFS destination). I have no quota set on the volume holding the table (since I thought it may be a problem, due to this post), though I don't really think it should matter since I was able to successfully create the stream in the MCS. Furthermore, I can see that the stream exists by:

[mapr@mapr001 clarity_eap_hph_sa]$ maprcli stream info -path /path/to/my/stream/streamname
produceperm copyperm ischangelog logicalsize adminperm topicperm ttl path numtopics defaultpartitions autocreate consumeperm compression physicalsize
u:mapr u:mapr false 0 u:mapr u:mapr 604800 /path/to/my/stream/streamname 1 1 false u:mapr lz4 0

Furthermore, I can see the stream and topic with:

[mapr@mapr001 clarity_eap_hph_sa]$ maprcli stream topic list -path /path/to/stream/streamname
partitions maxlag logicalsize topic consumers physicalsize
5 0 0 mytopic 0 0

and

[mapr@mapr001 clarity_eap_hph_sa]$ maprcli stream topic info -path /path/to/my/stream/streamname -topic mytopic -json
{
"timestamp":1516386956104,
"timeofday":"2018-01-19 08:35:56.104 GMT-1000 AM",
"status":"OK",
"total":5,
"data":[
{
"partitionid":0,
"physicalsize":0,
"logicalsize":0,
"maxoffset":-1,
"minoffsetacrossconsumers":0,
"mintimestamp":"1969-12-31T02:00:00.000-1000 PM",
"maxtimestamp":"1969-12-31T02:00:00.000-1000 PM",
"mintimestampacrossconsumers":"1969-12-31T02:00:00.000-1000 PM",
"fid":"2657.33.131256",
"master":"mapr003.ucera.local:5660",
"servers":"mapr006.ucera.local:5660, mapr002.ucera.local:5660, mapr003.ucera.local:5660"
},
{
"partitionid":1,
"physicalsize":0,
"logicalsize":0,
"maxoffset":-1,
"minoffsetacrossconsumers":0,
"mintimestamp":"1969-12-31T02:00:00.000-1000 PM",
"maxtimestamp":"1969-12-31T02:00:00.000-1000 PM",
"mintimestampacrossconsumers":"1969-12-31T02:00:00.000-1000 PM",
"fid":"2657.33.131256",
"master":"mapr003.ucera.local:5660",
"servers":"mapr006.ucera.local:5660, mapr002.ucera.local:5660, mapr003.ucera.local:5660"
},
...

...

...
]
}

Has anyone seen this kind of problem before or have any ideas of what's going on here?

Thanks.

Outcomes