AnsweredAssumed Answered

How can we overcome this error "Out of Memory error for very large single line input record"?

Question asked by elloyd79 on May 30, 2018
Latest reply on May 30, 2018 by maprcommunity

We are currently using HUNK and MapR to read in some dummy data which is basically a collection of repeating events. We are trying to test the data ingestion of MapR.
In my script, at first, I had the event divided by one newline. Then this error happened. So I added a new newline thinking that maybe a few events were being concatenated. But the error still came back.
The question is.. how do we avoid seeing this error? It says we can set mapreduce.input.linrecordreader.line.maxlength to a lower value but as far as I can see the only place that is located is in a java file. Is there an xml properties files this is located in I can change?
Anyone else had an error like this?
IOException - Out of memory error while reading a very large single line input record. To skip this record set mapreduce.input.linerecordreader.line.maxlength to a lower value. Current value: 2147483647, jvm heap size: 508035072, potential value: 31752192

Outcomes