AnsweredAssumed Answered

MapR python APIs for DB and streams don't seem to work together

Question asked by reedv on Dec 26, 2017
Latest reply on Jul 19, 2018 by VivekG

TLDR:

To replicate the error in as simple way as possible, I am doing a cut and paste of the code from the mapr-demos github repo here (https://github.com/mapr-demos/maprdb_python_examples/blob/master/load.py) and adding the lines:
from mapr_streams_python import Producer
p = Producer({'streams.producer.default.stream': '/my_stream'})
near the top.
Then configure the virtualenv I am running the python code in according to the docs for using the mapr streams python api (https://maprdocs.mapr.com/60/AdvancedInstallation/InstallingStreamsPYClient.html) (https://maprdocs.mapr.com/60/MapR_Streams/MapRStreamCAPISetup.html#task_qxg_h2m_3z). Then running:
$python load.py
to produce the error:
-------------------------------------------------------------------------------------------
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
Traceback (most recent call last):
  File "/home/mapr/Documents/MT_dev/db-streams-compat/lib/python3.6/site-packages/maprdb/utils.py", line 98, in wrapper
    ret = f(*args, **kwargs)
  File "/home/mapr/Documents/MT_dev/db-streams-compat/lib/python3.6/site-packages/maprdb/connection.py", line 97, in exists
    return self.MapRDB.tableExists(name) == 1
jpype._jexception.DBExceptionPyRaisable: com.mapr.db.exceptions.DBException: tableExists() failed.,
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "load.py", line 50, in <module>
    t = open_table(db, TABLE_SENSORS_PATH)
  File "load.py", line 40, in open_table
    if connection.exists(p):
  File "/home/mapr/Documents/MT_dev/db-streams-compat/lib/python3.6/site-packages/maprdb/utils.py", line 101, in wrapper
    raise MapRDBError(str(e)) from e
maprdb.utils.MapRDBError: com.mapr.db.exceptions.DBException: tableExists() failed.,
2017-12-26 11:06:36,9191 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:687 Thread: 60477 Mismatch found for java and native libraries java build version 5.2.1.42646.GA, native build version 6.0.0.20171109191718.GA java patch vserion $Id: mapr-version: 5.2.1.42646.GA 42646:812878ab1269 $, native patch version $Id: mapr-version: 6.0.0.20171109191718.GA e892229b271c98c75ccb
2017-12-26 11:06:36,9191 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:704 Thread: 60477 Client initialization failed.
-------------------------------------------------------------------------------------------
Seems like a simple test of whether the mapr db and streams APIs for python are going to work together. A test which is failing, but why?
END TLDR;

 

I am trying to run the python mapr db example programs provided by mapr (https://github.com/mapr-demos/maprdb_python_examples) (with some additional code using the mapr python api for streams (simply importing the Producer package and creating an instance)) and getting an error that I do not understand related to the python maprdb package, printed below:

 

---------------------------------------------------------------------------------------

(db2stream2db) [mapr@mapr001 maprdb_python_examples]$ python load.py

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Traceback (most recent call last):

File "/home/mapr/Documents/MT_dev/db2stream2db/lib/python3.6/site-packages/maprdb/utils.py", line 98, in wrapper ret = f(*args, **kwargs)

File "/home/mapr/Documents/MT_dev/db2stream2db/lib/python3.6/site-packages/maprdb/connection.py", line 97, in exists return self.MapRDB.tableExists(name) == 1

jpype._jexception.DBExceptionPyRaisable: com.mapr.db.exceptions.DBException: tableExists() failed.,

 

The above exception was the direct cause of the following exception:

 

Traceback (most recent call last):

File "load.py", line 47, in <module> t = open_table(db, TABLE_SENSORS_PATH)

File "load.py", line 37, in open_table if connection.exists(p):

File "/home/mapr/Documents/MT_dev/db2stream2db/lib/python3.6/site-packages/maprdb/utils.py", line 101, in wrapper raise MapRDBError(str(e)) from e maprdb.utils.MapRDBError: com.mapr.db.exceptions.DBException: tableExists() failed.

---------------------------------------------------------------------------------------

 

Note that this error does not occur when running just the mapr db python example from the link provided above. However, the problem occurs when trying to use this python api in tandem with another mapr python api, the python streams api (https://maprdocs.mapr.com/60/MapR_Streams/MapRStreamsPythonApplications.html) (which is why I believe this is a valid community post). So far as I can tell, installing python maprdb package ("pip install maprdb") and running the load.py script from the github maprdb demo, runs fine. However, after setting up the environment using the instructions for setting up python streams api (https://maprdocs.mapr.com/60/AdvancedInstallation/InstallingStreamsPYClient.html) (https://maprdocs.mapr.com/60/MapR_Streams/MapRStreamCAPISetup.html#task_qxg_h2m_3z), I get the error printed above. Its seems like configuring the environment variables (as specified here: https://maprdocs.mapr.com/60/MapR_Streams/MapRStreamCAPISetup.html#task_qxg_h2m_3z) is what causes the error message to appear.

 

This may also have something to do with another stream error that occurs regardless of whether mapr db is being used or not, occurs even when trying to only use the streams api on its own after setting up according to the Mapr-ES python docs linked to above (ie. installing the python package and setting the environment variables):

 

-------------------------------------------------------------------------------------------

(db2stream2db) [mapr@mapr001 src]$ python simple_producer.py Traceback (most recent call last):

File "simple_producer.py", line 3, in <module> from mapr_streams_python import Producer

File "/home/mapr/Documents/MT_dev/db2stream2db/lib/python3.6/site-packages/mapr_streams_python/__init__.py", line 2, in <module> from .cimpl import * ImportError: librdkafka.so.1: cannot open shared object file: No such file or directory

--------------------------------------------------------------------------------------------

 

 

To replicate the error in as simple way as possible, I am doing a cut and paste of the code from the mapr-demos github repo here (https://github.com/mapr-demos/maprdb_python_examples/blob/master/load.py) and adding the lines:
from mapr_streams_python import Producer
p = Producer({'streams.producer.default.stream': '/my_stream'})
near the top.
Then configure the virtualenv I am running the python code in according to the docs for using the mapr streams python api (https://maprdocs.mapr.com/60/AdvancedInstallation/InstallingStreamsPYClient.html) (https://maprdocs.mapr.com/60/MapR_Streams/MapRStreamCAPISetup.html#task_qxg_h2m_3z). Then running:
$python load.py
to produce the error:
-------------------------------------------------------------------------------------------
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
Traceback (most recent call last):
  File "/home/mapr/Documents/MT_dev/db-streams-compat/lib/python3.6/site-packages/maprdb/utils.py", line 98, in wrapper
    ret = f(*args, **kwargs)
  File "/home/mapr/Documents/MT_dev/db-streams-compat/lib/python3.6/site-packages/maprdb/connection.py", line 97, in exists
    return self.MapRDB.tableExists(name) == 1
jpype._jexception.DBExceptionPyRaisable: com.mapr.db.exceptions.DBException: tableExists() failed.,
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "load.py", line 50, in <module>
    t = open_table(db, TABLE_SENSORS_PATH)
  File "load.py", line 40, in open_table
    if connection.exists(p):
  File "/home/mapr/Documents/MT_dev/db-streams-compat/lib/python3.6/site-packages/maprdb/utils.py", line 101, in wrapper
    raise MapRDBError(str(e)) from e
maprdb.utils.MapRDBError: com.mapr.db.exceptions.DBException: tableExists() failed.,
2017-12-26 11:06:36,9191 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:687 Thread: 60477 Mismatch found for java and native libraries java build version 5.2.1.42646.GA, native build version 6.0.0.20171109191718.GA java patch vserion $Id: mapr-version: 5.2.1.42646.GA 42646:812878ab1269 $, native patch version $Id: mapr-version: 6.0.0.20171109191718.GA e892229b271c98c75ccb
2017-12-26 11:06:36,9191 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:704 Thread: 60477 Client initialization failed.
-------------------------------------------------------------------------------------------

 

Assistance would be appreciated as it seems like these packages should be able to work together since they are both mapr python apis. Thank you.

Outcomes