AnsweredAssumed Answered

PACC: MaprDB OR MapR Streams (Python)

Question asked by mandoskippy on Feb 23, 2018
Latest reply on Mar 20, 2018 by maprcommunity

So I am trying to get  a PACC container (MapR 6.0) where both MapR Streams and MapR DB works.  MapR DB is fairly straight forward, if I FROM PACC and then put this in

 

RUN git clone https://github.com/mapr-demos/python-bindings && cd python-bindings && python3 setup.py build && \

    for j in build/lib/maprdb/dependency/*.jar; do F=$(basename $j) ; if [ -f /opt/mapr/lib/$F ]; then cp /opt/mapr/lib/$F build/lib/maprdb/dependency/; fi done && \

    python3 setup.py install

 

I can get a PACC Container working with MapR DB Python bindings. 

 

If I try to do MapR Streams, It's a big harder, I need to "put in" the librdkafka, but since it doesn't install nicely without the full client installed, I can't just dpkg -i the deb file, I copy the files into the proper locations, then I can install the Python bindings like this:

 

ENV LIBRARY_PATH=/opt/mapr/lib

ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

ENV LD_LIBRARY_PATH=/opt/mapr/lib:$JAVA_HOME/jre/lib/amd64/server

 

# Get the librdkafka and copy the files to the /opt/mapr locations

 

RUN wget http://package.mapr.com/releases/MEP/MEP-4.1.0/ubuntu/mapr-librdkafka_0.9.1.201711121549_all.deb && mkdir -p ./tmp && dpkg-deb -R mapr-librdkafka_0.9.1.201711121549_all.deb tmp && \

    cp -R ./tmp/opt/mapr/include/librdkafka /opt/mapr/include/ && cp -R ./tmp/opt/mapr/librdkafka /opt/mapr && cp ./tmp/opt/mapr/lib/librdkafka.so.1  /opt/mapr/lib/ && ln /opt/mapr/lib/librdkafka.so.1  /opt/mapr/lib/librdkafka.so && \

    rm mapr-librdkafka_0.9.1.201711121549_all.deb && rm -rf ./tmp && ldconfig

 

# Install the MapR Python Bindings for Streams:

RUN pip3 install --global-option=build_ext --global-option="--library-dirs=/opt/mapr/lib" --global-option="--include-dirs=/opt/mapr/include/" http://package.mapr.com/releases/MEP/MEP-4.1.0/mac/mapr-streams-python-0.9.2.tar.gz

 

This "sorta" works.  When I try to invoke the python code 

 

from mapr_streams_python import Producer

 

I get:

 

Traceback (most recent call last):

  File "./wifidata.py", line 8, in <module>

    from mapr_streams_python import Producer

  File "/usr/local/lib/python3.5/dist-packages/mapr_streams_python/__init__.py", line 2, in <module>

    from .cimpl import *

ImportError: librdkafka.so.1: cannot open shared object file: No such file or directory

 

 

Doing some research, I tried this:

 

LD_LIBRARY_PATH=/opt/mapr/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server ./wifidata.py

 

Then that works, it loads properly. 

 

However, If I try to do "both" things (import maprdb, and from mapr_streams_python import Producer)

I get this goofy thing:

 

$ LD_LIBRARY_PATH=/opt/mapr/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server ./wifidata.py

Making connection with maprdb.connect()

Trying to get table with maprconn.get(wifitable)

log4j:WARN No appenders could be found for logger (com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

2018-02-23 18:59:32,0383 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:687 Thread: 570 Mismatch found for java and native libraries java build version 5.2.1.42646.GA, native build version 6.0.0.20171109191718.GA java patch vserion $Id: mapr-version: 5.2.1.42646.GA 42646:812878ab1269 $, native patch version $Id: mapr-version: 6.0.0.20171109191718.GA e892229b271c98c75ccb

2018-02-23 18:59:32,0383 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:704 Thread: 570 Client initialization failed.

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/maprdb-0.0.3-py3.5.egg/maprdb/utils.py", line 98, in wrapper

    ret = f(*args, **kwargs)

  File "/usr/local/lib/python3.5/dist-packages/maprdb-0.0.3-py3.5.egg/maprdb/connection.py", line 69, in get

    j_table = self.MapRDB.getTable(name)

jpype._jexception.DBExceptionPyRaisable: com.mapr.db.exceptions.DBException: <init>() failed.,

 

The above exception was the direct cause of the following exception:

 

Traceback (most recent call last):

  File "./wifidata.py", line 92, in <module>

    main()

  File "./wifidata.py", line 29, in main

    maprtable = maprconn.get(wifitable)

  File "/usr/local/lib/python3.5/dist-packages/maprdb-0.0.3-py3.5.egg/maprdb/utils.py", line 101, in wrapper

    raise MapRDBError(str(e)) from e

maprdb.utils.MapRDBError: com.mapr.db.exceptions.DBException: <init>() failed.,

 

 

Is there a reasonable way to use both MapR Streams and MapRDB via Python in a MapR PACC? 

 

Thanks!

 

John

Outcomes