We have a highly concurrent client application communicating with a MaprDB server of a quite a small size. I am trying to figure out the tuning knobs of optimizing the client performance.
To set the context, the application is using version 1.1 of Hbase API with Mapr-DB server of 5.1 community edition. The client uses a managed thread pool to concurrently trigger a number of scan, put and get requests. However, in all the possible configurations, I could see the CPU to be hovering at 98%. I could see that all the CPU time is spent in the native (GET, PUT and Scan API).
Here's my question:
1) Mapr-DB Client offers a configuration "fs.mapr.threads" for controlling the client concurrency. How is that mapped to user threads. Do requests from multiple user threads for various API's are offloaded to the underlying thread pool.
2) Even after lowering the above setting to very less, close to 2 for a 4 core, 8 hyperthreaded machine, I couldn't get the CPU to below 98%.
3) I could see my client threads consuming a lot of CPU, while I expect them to be simply waiting for I/O, or the underlying thread pool.
4) Even after lowering the concurrent requests to be as less than 10, the CPU is hovering at 98%, while I am simply making from scan, get and put's.
Attached are some snapshots: