AnsweredAssumed Answered

Best way to reconfigure disks?

Question asked by jsumali on Jan 16, 2013
Latest reply on Jan 16, 2013 by Ted Dunning
Hi,

We're currently running a 4-node cluster, and we want to expand the disk space available by re-configuring the physical disks on each server.

Our current strategy is:
 - for each server, remove it from the cluster, reconfigure the disks offline, add it back to the cluster.

The challenge we're facing is that our current cluster is quite full (78% of 6.2 TB), and we have a large volume with a replication factor of 3 and a minimum replication of 3. If we keep the replication factor as-is and remove a server, we won't have enough space to deal with the replication that will kick in since a server will be removed from the cluster. One option may be to reduce the minimum replication factor from 3 to 2, then proceed with the above strategy. According to the answer by Steven at http://answers.mapr.com/questions/2525/minimum-replication-factor-and-volume_alarm_data_under_replicated, at a minimum replication factor of 2, the cluster would wait some time before re-replicating (which is when we'd do our upgrades). We would not like to have to reduce the main replication factor from 3 to 2 if possible.

Questions:

 1. Is this a reasonable approach for
    re-configuring the cluster, or is
    there a better way ? We'd like to
    avoid full downtime if possible.
 2. Instead of having to reduce replication (main, or minimum) from 3 to 2, is there a
    way to either disable replication when we remove a server,
    or perhaps to make the cluster
    read-only? (e.g. bin/hadoop dfsadmin
    -safemode ) ?
 3. If we decide to change the minimum replication from 3 to 2, does anyone know exactly how long we have before re-replication takes place?   

The following chart shows the problem we have: keeping replication at 3 with only 3 servers (normally 4) means we won't have enough space.

![alt text][1]


  [1]: https://dl.dropbox.com/u/2110776/Untitled.png

Thanks,
Josh

Outcomes