AnsweredAssumed Answered

Failed to format /dev/sda,/dev/sdb for MapR FS caused by device is busy

Question asked by dzlabs on Apr 6, 2015
Latest reply on Nov 28, 2017 by jimzzhou
I'm trying to install MapR on a cluster of three nodes using the mapr-setup script. The installation fails with Failed to format /dev/sda,/dev/sdb for MapR FS message error. Here is the content of my config.example:

    # Each Node section can specify nodes in the following format
    # Hostname: disk1, disk2, disk3
    # Specifying disks is optional. If not provided, the installer will use the values of 'disks' from the Defaults section
    [Control_Nodes]
    192.168.2.200: /dev/sda, /dev/sdb
    192.168.2.223: /dev/sda, /dev/sdb
    192.168.2.201: /dev/sda, /dev/sdb
    [Data_Nodes]
    #data-node1.mydomain
    #data-node2.mydomain: /dev/sdb, /dev/sdc, /dev/sdd
    #data-node3.mydomain: /dev/sdd
    #data-node4.mydomain: /dev/sdb, /dev/sdd
    [Client_Nodes]
    #client1.mydomain
    #client2.mydomain
    #client3.mydomain
    [Options]
    MapReduce1 = false
    YARN = true
    HBase = false
    MapR-DB = true
    ControlNodesAsDataNodes = true
    WirelevelSecurity = false
    LocalRepo = false
    [Defaults]
    ClusterName = dz.cluster.com
    User = mapr
    Group = mapr
    Password = mapr
    UID = 2000
    GID = 2000
    Disks = /dev/sda,/dev/sdb
    StripeWidth = 3
    ForceFormat = false
    CoreRepoURL = http://package.mapr.com/releases
    EcoRepoURL = http://package.mapr.com/releases/ecosystem-4.x
    Version = 4.0.2
    MetricsDBHost =
    MetricsDBUser =
    MetricsDBPassword =
    MetricsDBSchema =
    
    #[Spark]
    #SparkVersion = 0.9.1
    #SparkMasters = control-node1.mydomain, control-node2.mydomain
    #SparkSlaves = data-node1.mydomain, data-node2.mydomain, data-node3.mydomain
    #SparkMem = 2
    #SparkWorkerMem = 1
    #SparkDaemonMem = 16
    
    #[Hive]
    #HiveVersion = 0.12
    #HiveServers = control-node1.mydomain
    #HiveMetaStore = control-node2.mydomain
    #HiveClients = client-node1.mydomain, data-node3.mydomain

Here is the content of the log file /opt/mapr/logs/disksetup.0.log generated during the installation process:

    2015-04-05 11:47:40,213 2399 InitLogging:128 INFO
    
    ***** Starting disksetup: PID 2399 *****
    
    2015-04-05 11:47:40,213 2399 <module>:1240 INFO argv: ['/opt/mapr/server/disksetup', '-W', '3', '/opt/mapr/conf/disks.txt']
    2015-04-05 11:47:40,214 2399 <module>:1241 INFO Options:
    Using stripe width 3
    
    2015-04-05 11:47:40,214 2399 SetCldbRole:996 INFO Create cid = 1 for cldb role
    2015-04-05 11:47:40,214 2399 LoadDiskTab:459 INFO No disktab file: /opt/mapr/conf/disktab
    2015-04-05 11:47:40,214 2399 LoadInputFile:480 INFO Start parsing input file: /opt/mapr/conf/disks.txt
    2015-04-05 11:47:40,214 2399 LoadInputFile:484 INFO /dev/sda
    /dev/sdb
    
    2015-04-05 11:47:40,214 2399 LoadInputFile:509 INFO Loaded disks:
    2015-04-05 11:47:40,214 2399 LoadInputFile:511 INFO ([], 0, ['/dev/sda'], 1, '/dev/sda')
    2015-04-05 11:47:40,214 2399 LoadInputFile:511 INFO ([], 0, ['/dev/sdb'], 1, '/dev/sdb')
    2015-04-05 11:47:40,214 2399 GetMfsUp:1001 INFO Lookup for : /opt/mapr/conf/mapr-clusters.conf
    2015-04-05 11:47:40,214 2399 RunCmd:197 INFO mapr-mfs status:
    /etc/init.d/mapr-mfs status
    2015-04-05 11:47:40,287 2399 RunCmd:200 ERROR rc=3
    2015-04-05 11:47:40,287 2399 RunCmd:201 ERROR  * mfs is not running
    2015-04-05 11:47:40,287 2399 RunCmd:205 INFO mapr-mfs status tried. err 3
    2015-04-05 11:47:40,287 2399 RunCmd:197 INFO Fileserver start:
    /etc/init.d/mapr-mfs start
    2015-04-05 11:47:40,458 2399 RunCmd:197 INFO Disk list -v :
    /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660  disk list -v
    2015-04-05 11:47:40,832 2399 LogCurrentInfo:957 INFO start setup Trying to capture debug info, following errors can be ignored
    
    2015-04-05 11:47:40,832 2399 RunCmd:197 INFO Disk list :
    /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660  disk list
    2015-04-05 11:47:40,852 2399 LogCurrentInfo:962 INFO ListDisks resp: status 0 count=0
    2015-04-05 11:47:40,852 2399 RunCmd:197 INFO DiskGroup list :
    /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660  dg list
    2015-04-05 11:47:40,862 2399 LogCurrentInfo:967 INFO No. of DGs (0)
    
    2015-04-05 11:47:40,862 2399 RunCmd:197 INFO sp list :
    /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660  sp list
    2015-04-05 11:47:40,872 2399 LogCurrentInfo:972 INFO 2015-04-05 11:47:40,8724 ERROR Global mrconfig.cc:545 No SP on this disk
    
    ListSPs resp: status 0:0
    2015-04-05 11:47:40,873 2399 LogCurrentInfo:974 INFO
    Done capturing debug info
    
    
    2015-04-05 11:47:40,873 2399 AddUdevRules:753 INFO Fixing udev rules
    2015-04-05 11:47:40,873 2399 BackupUdevRules:689 INFO /lib/udev/rules.d/99-mapr-disk.rules do not exist.
    2015-04-05 11:47:40,873 2399 UpdateUdevRules:793 INFO Update udev rules
    KERNEL=="sdb", GROUP="mapr", MODE="0660"
    KERNEL=="sda", GROUP="mapr", MODE="0660"
    
    2015-04-05 11:47:40,873 2399 RunCmd:197 INFO UpdateUdevRules:
    /bin/mv /tmp/99-mapr-disk.rules.2399 /lib/udev/rules.d/99-mapr-disk.rules
    2015-04-05 11:47:40,963 2399 RunCmd:197 INFO UpdateUdevRules:
    /sbin/udevadm control --reload-rules
    2015-04-05 11:47:41,037 2399 RunCmd:197 INFO Change mode: chmod g+rw /dev/sda:
    chmod g+rw /dev/sda
    2015-04-05 11:47:41,050 2399 RunCmd:197 INFO Change group /dev/sda:
    chgrp mapr /dev/sda
    2015-04-05 11:47:41,070 2399 RunCmd:197 INFO Disk init /dev/sda:
    /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660  disk init /dev/sda
    2015-04-05 11:47:41,078 2399 RunCmd:200 ERROR rc=16
    2015-04-05 11:47:41,079 2399 RunCmd:201 ERROR 2015-04-05 11:47:41,0781 ERROR Global mrconfig.cc:4047 InitDisk failed Device or resource busy.(16).
    2015-04-05 11:47:41,079 2399 RunCmd:205 INFO Disk init /dev/sda tried. err 16
    2015-04-05 11:47:41,079 2399 InitDisk:290 INFO Disk init /dev/sda failed.Disk init /dev/sda
    2015-04-05 11:47:41,079 2399 AbortWithError:192 ERROR /dev/sda failed. Error 16, Device or resource busy. Disk is used by some other module/process.['  File "/opt/mapr/server/disksetup", line 1253, in <module>\n    RunDiskSetup();\n', '  File "/opt/mapr/server/disksetup", line 1136, in RunDiskSetup\n    InitAllDisks(force);\n', '  File "/opt/mapr/server/disksetup", line 555, in InitAllDisks\n    disk.InitDevices(forceOption);\n', '  File "/opt/mapr/server/disksetup", line 429, in InitDevices\n    InitDisk(dev, forceOption);\n', '  File "/opt/mapr/server/disksetup", line 291, in InitDisk\n    AbortWithError(rc, errMsg, devicePath);\n', '  File "/opt/mapr/server/disksetup", line 190, in AbortWithError\n    stack_trace = traceback.format_stack(frame)\n']
    2015-04-05 11:47:41,080 2399 LogCurrentInfo:957 INFO Abort Trying to capture debug info, following errors can be ignored
    
    2015-04-05 11:47:41,080 2399 RunCmd:197 INFO Disk list :
    /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660  disk list
    2015-04-05 11:47:41,087 2399 LogCurrentInfo:962 INFO ListDisks resp: status 0 count=0
    2015-04-05 11:47:41,087 2399 RunCmd:197 INFO DiskGroup list :
    /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660  dg list
    2015-04-05 11:47:41,093 2399 LogCurrentInfo:967 INFO No. of DGs (0)
    
    2015-04-05 11:47:41,094 2399 RunCmd:197 INFO sp list :
    /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660  sp list
    2015-04-05 11:47:41,100 2399 LogCurrentInfo:972 INFO 2015-04-05 11:47:41,1001 ERROR Global mrconfig.cc:545 No SP on this disk
    
    ListSPs resp: status 0:0
    2015-04-05 11:47:41,101 2399 LogCurrentInfo:974 INFO
    Done capturing debug info
    
    
    2015-04-05 11:47:41,101 2399 RestoreUdevRules:694 INFO Udev restoration not required.
    2015-04-05 11:47:41,101 2399 RunCmd:197 INFO Fileserver stop:
    /etc/init.d/mapr-mfs stop
    2015-04-05 11:47:42,666 2399 ExitDiskSetup:160 ERROR /dev/sda failed. Error 16, Device or resource busy. Disk is used by some other module/process.['  File "/opt/mapr/server/disksetup", line 1253, in <module>\n    RunDiskSetup();\n', '  File "/opt/mapr/server/disksetup", line 1136, in RunDiskSetup\n    InitAllDisks(force);\n', '  File "/opt/mapr/server/disksetup", line 555, in InitAllDisks\n    disk.InitDevices(forceOption);\n', '  File "/opt/mapr/server/disksetup", line 429, in InitDevices\n    InitDisk(dev, forceOption);\n', '  File "/opt/mapr/server/disksetup", line 291, in InitDisk\n    AbortWithError(rc, errMsg, devicePath);\n', '  File "/opt/mapr/server/disksetup", line 190, in AbortWithError\n    stack_trace = traceback.format_stack(frame)\n']

It says that the device is busy as locked by another process! How I could fix this?

Outcomes