AnsweredAssumed Answered

PACC: MaprDB OR MapR Streams (Python)

Question asked by mandoskippy on Feb 23, 2018
Latest reply on Mar 20, 2018 by maprcommunity

So I am trying to get  a PACC container (MapR 6.0) where both MapR Streams and MapR DB works.  MapR DB is fairly straight forward, if I FROM PACC and then put this in


RUN git clone && cd python-bindings && python3 build && \

    for j in build/lib/maprdb/dependency/*.jar; do F=$(basename $j) ; if [ -f /opt/mapr/lib/$F ]; then cp /opt/mapr/lib/$F build/lib/maprdb/dependency/; fi done && \

    python3 install


I can get a PACC Container working with MapR DB Python bindings. 


If I try to do MapR Streams, It's a big harder, I need to "put in" the librdkafka, but since it doesn't install nicely without the full client installed, I can't just dpkg -i the deb file, I copy the files into the proper locations, then I can install the Python bindings like this:


ENV LIBRARY_PATH=/opt/mapr/lib

ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

ENV LD_LIBRARY_PATH=/opt/mapr/lib:$JAVA_HOME/jre/lib/amd64/server


# Get the librdkafka and copy the files to the /opt/mapr locations


RUN wget && mkdir -p ./tmp && dpkg-deb -R mapr-librdkafka_0.9.1.201711121549_all.deb tmp && \

    cp -R ./tmp/opt/mapr/include/librdkafka /opt/mapr/include/ && cp -R ./tmp/opt/mapr/librdkafka /opt/mapr && cp ./tmp/opt/mapr/lib/  /opt/mapr/lib/ && ln /opt/mapr/lib/  /opt/mapr/lib/ && \

    rm mapr-librdkafka_0.9.1.201711121549_all.deb && rm -rf ./tmp && ldconfig


# Install the MapR Python Bindings for Streams:

RUN pip3 install --global-option=build_ext --global-option="--library-dirs=/opt/mapr/lib" --global-option="--include-dirs=/opt/mapr/include/"


This "sorta" works.  When I try to invoke the python code 


from mapr_streams_python import Producer


I get:


Traceback (most recent call last):

  File "./", line 8, in <module>

    from mapr_streams_python import Producer

  File "/usr/local/lib/python3.5/dist-packages/mapr_streams_python/", line 2, in <module>

    from .cimpl import *

ImportError: cannot open shared object file: No such file or directory



Doing some research, I tried this:


LD_LIBRARY_PATH=/opt/mapr/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server ./


Then that works, it loads properly. 


However, If I try to do "both" things (import maprdb, and from mapr_streams_python import Producer)

I get this goofy thing:


$ LD_LIBRARY_PATH=/opt/mapr/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server ./

Making connection with maprdb.connect()

Trying to get table with maprconn.get(wifitable)

log4j:WARN No appenders could be found for logger (com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See for more info.

2018-02-23 18:59:32,0383 ERROR JniCommon fs/client/fileclient/cc/ Thread: 570 Mismatch found for java and native libraries java build version, native build version java patch vserion $Id: mapr-version: 42646:812878ab1269 $, native patch version $Id: mapr-version: e892229b271c98c75ccb

2018-02-23 18:59:32,0383 ERROR JniCommon fs/client/fileclient/cc/ Thread: 570 Client initialization failed.

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/maprdb-0.0.3-py3.5.egg/maprdb/", line 98, in wrapper

    ret = f(*args, **kwargs)

  File "/usr/local/lib/python3.5/dist-packages/maprdb-0.0.3-py3.5.egg/maprdb/", line 69, in get

    j_table = self.MapRDB.getTable(name)

jpype._jexception.DBExceptionPyRaisable: com.mapr.db.exceptions.DBException: <init>() failed.,


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "./", line 92, in <module>


  File "./", line 29, in main

    maprtable = maprconn.get(wifitable)

  File "/usr/local/lib/python3.5/dist-packages/maprdb-0.0.3-py3.5.egg/maprdb/", line 101, in wrapper

    raise MapRDBError(str(e)) from e

maprdb.utils.MapRDBError: com.mapr.db.exceptions.DBException: <init>() failed.,



Is there a reasonable way to use both MapR Streams and MapRDB via Python in a MapR PACC?