AnsweredAssumed Answered

writing to maprfs using sample code

Question asked by elleg on Jun 25, 2014
Latest reply on Jun 26, 2014 by elleg
Hello,

My colleague and I are trying out writes to maprfs from a stream of json data. We used the [sample code][1] in your docs. When we write using multiple concurrent writer instances (in Java), we see only 2000 odd messages actually written to the mapr file.

When we bring it down to a single writer instance, then all the messages are written. What could we be doing wrong in the case of the multiple writers, and how can we make sure this works correctly?

Here is the file system setup in EACH writer instance:

            String dirName = "/data";
            boolean overwrite = true;
            int bufferSize = 2048;
            int replication = 1;
            long blockSize = 67108864;

            conf = new Configuration();
            fileSystem = FileSystem.get(conf);
            dirPath = new Path(dirName + "/dir");
            wFilePath = new Path(dirName + "/file.w");
            //rFilePath = wFilePath;//new Path(DIR_NAME + "file.r");

            outputStream = fileSystem.create(wFilePath,
                                             overwrite,
                                             bufferSize,
                                             (short) replication,
                                             blockSize);

            LOGGER.info("wFilePath - " + wFilePath);

Here is the writer code:

        byte[] messageBytes = json.getBytes();

        try {
            outputStream.write(messageBytes);
            messageCount.incrementAndGet();
        } catch (IOException e) {
            LOGGER.error("Error writing to the MapR output stream.");
        } catch (Exception e) {
            LOGGER.error("Unknown error writing to the MapR output stream.", e);
        }

There are no errors being logged, so we don't know why this happens.

Thanks!


  [1]:  http://doc.mapr.com/display/MapR2/Accessing+MapR-FS+in+Java+Applications

Outcomes