AnsweredAssumed Answered

Uneven Distribution of Container Masters

Question asked by snelson on Oct 21, 2014
Latest reply on Oct 24, 2014 by snelson
We're getting inconsistent and poor performance using our M7 tables. Does MapR balance container masters among nodes and among disks? Am I understanding correctly that the master is exclusively responsible for serving data for its container? I've conducted an analysis on two of our volumes that shows uneven distribution, and I would like to correct that. Below is a sample output from my analysis program. As you can see, node 2 has 4 times as many masters as node 1, and that's causing the load on node 2 to be 4x as much as the load on node 1 (see graphs below). The distribution of masters is causing half of the nodes in my cluster to be significantly under-used compared to the other half. One of the nodes with the highest number of masters happens to be my CLDB Master, and the CLDB container is on a disk that is master for at least 18 other containers.

    Node 1 is master of 45 containers
      Storage Pool /dev/xvdf is master of 6 containers
      Storage Pool /dev/xvdg is master of 2 containers
      Storage Pool /dev/xvdh is master of 4 containers
      Storage Pool /dev/xvdi is master of 6 containers
      Storage Pool /dev/xvdj is master of 2 containers
      Storage Pool /dev/xvdk is master of 3 containers
      Storage Pool /dev/xvdl is master of 3 containers
      Storage Pool /dev/xvdm is master of 5 containers
      Storage Pool /dev/xvdn is master of 2 containers
      Storage Pool /dev/xvdo is master of 3 containers
      Storage Pool /dev/xvdp is master of 1 containers
      Storage Pool /dev/xvdq is master of 1 containers
      Storage Pool /dev/xvdr is master of 2 containers
      Storage Pool /dev/xvdx is master of 2 containers
      Storage Pool /dev/xvdy is master of 3 containers
    Node 2 is master of 205 containers
      Storage Pool /dev/xvdf is master of 19 containers
      Storage Pool /dev/xvdg is master of 14 containers
      Storage Pool /dev/xvdh is master of 12 containers
      Storage Pool /dev/xvdi is master of 15 containers
      Storage Pool /dev/xvdj is master of 9 containers
      Storage Pool /dev/xvdk is master of 11 containers
      Storage Pool /dev/xvdl is master of 16 containers
      Storage Pool /dev/xvdm is master of 10 containers
      Storage Pool /dev/xvdn is master of 13 containers
      Storage Pool /dev/xvdo is master of 15 containers
      Storage Pool /dev/xvdp is master of 20 containers
      Storage Pool /dev/xvdq is master of 11 containers
      Storage Pool /dev/xvdr is master of 14 containers
      Storage Pool /dev/xvdx is master of 9 containers
      Storage Pool /dev/xvdy is master of 17 containers



![alt text][1]![alt text][2]

![alt text][3]


  [1]: http://s2.postimg.org/5lduire3t/node2.png
  [2]: http://s22.postimg.org/xef95ej8x/node1.png
  [3]: http://s29.postimg.org/50u33gepj/perfscreen.png

Outcomes