AnsweredAssumed Answered

Error: No space left on device(28); yet plenty of space is available. MapR filling up only one disk on each node..

Question asked by edmond on Jun 25, 2015
Latest reply on Jun 26, 2015 by edmond
I've created a 20TB test cluster on AWS EC2 for evaluating MapR for our company. We have upwards of 10+ million files we ultimately need to store (once we deploy in production). Using a python test script, I ran a test to write 100,000 files of various sizes to the cluster.

The test failed after writing only approx 700GiB of files with a "Metrics Write Problem Alarm" on all nodes on the MapR dashboard.

Errors in mapr1:/opt/logs/hoststats.log:

> 2015-06-25 15:23:56,1075 ERROR
> Hoststats hoststats.cc:269 Thread:
> 140489095640960 Failed to open for
> writing!
> /var/mapr/local/mapr1/metrics/clustermetrics.2015-06-25.15:00:0
> with error count since last shown
> error: 59

Errors in mapr1:/opt/logs/hoststats.err:

> 2015-06-25 15:26:06,2287 ERROR Client
> fs/client/fileclient/cc/client.cc:1239
> Thread: 21717 Create failed for file
> clustermetrics.2015-06-25.15:00:00,
> error No space left on device(28) fid
> 2061.16.2 2015-06-25 15:26:06,2288 ERROR LibHdfsApiSupport
> fs/client/fileclient/cc/libhdfs/api_support.cc:735
> Thread: 21717 Create failed for file
> /var/mapr/local/mapr1/metrics/clustermetrics.2015-06-25.15:00:0,
> error: No space left on device(28)

The cluster is actually reporting 19 TB space free. However, if I now try to copy any file to the cluster I get a an error:

> cp: cannot create regular file
> ‘/mnt/mapr/storage/filelist.txt’: No
> space left on device

There are no disk quotas set for any volume. The nodes also have plenty of space available on their system volumes.

Edit: MapR is actually filling up the first disk on each node to 100%, and is ignoring the rest of the disks (10 per node). Why would this be the case?

Edit2: See additional comments below. MapR is writing only to Storage Pool #1, even on a freshly-installed cluster. All other storage pools are ignored.

Thank you!

Outcomes