AnsweredAssumed Answered

Hiveserver2: Failed to create/change scratchdir permissions to 777: Could not create FileClient

Question asked by alexwoolford on Feb 25, 2015
HiveServer2 crashes on my cluster. According the log file in `/opt/mapr/hive/hive-0.13/logs/mapr/hive.log`, there's a problem with permissions on the scratch directory:

    2015-02-24 21:21:08,187 WARN  [main]: server.HiveServer2 (HiveServer2.java:init(74)) - Failed to create/change scratchdir permissions to 777: Could not create FileClient java.io.IOException: Could not create FileClient
I checked the settings for the scratch directory using `hive -e 'set;' | grep scratch`:

    hive.exec.scratchdir=/user/mapr/tmp/hive/
    hive.scratch.dir.permission=700
I notice that `hive.scratch.dir.permission` is set to 700 and the error message suggests that it wants to change this to 777. However, according to the filesystem, `/mapr/my.cluster.com/user/mapr/tmp` has 777 permissions and belongs to the mapr user.

    mapr@hadoop01:/mapr/my.cluster.com/user/mapr/tmp$ ls -al
    total 2
    drwxr-xr-x  3 mapr mapr  1 Feb 22 10:39 .
    drwxr-xr-x  5 mapr mapr  3 Feb 24 08:40 ..
    drwxrwxrwx 56 mapr mapr 54 Feb 23 10:20 hive
Judging by the filesystem permissions, I would expect the mapr user to do whatever it wants with this folder and so don't understand the error message.

I'm curious to know if anyone's seen this before and, if so, how did you fix it?

**Update:**

I had a look at the source code, and notice some relevant comments just prior to the warning:

    // When impersonation is enabled, we need to have "777" permission on root scratchdir, because
    // query specific scratch directories under root scratchdir are created by impersonated user and
    // if permissions are not "777" the query fails with permission denied error.
I added set the following properties in hive-site.xml:

    <property>
      <name>hive.scratch.dir.permission</name>
      <value>777</value>
    </property>

    <property>
      <name>hive.exec.scratchdir</name>
      <value>/tmp/hive/</value>
    </property>
... and created the `/tmp/hive/` folder in HDFS with 777 permissions:

    mapr@hadoop01:~$ hadoop fs -ls -d /tmp/hive
    drwxrwxrwx   - mapr mapr          0 2015-02-27 08:38 /tmp/hive
Although this looked promising, I still got the same warning in `hive.log`.

Looking at `/opt/mapr/logs/cldb.log`, I see that the hs2 service gets deleted from Zookeeper and then the alarm gets raised:

    INFO ZKDataRetrieval [Thread-2-EventThread]: Process path: /services/hs2/master. Event state: SyncConnected. Event type: NodeDeleted
    INFO ZKDataRetrieval [Thread-2-EventThread]: Process path: /services/hs2. Event state: SyncConnected. Event type: NodeChildrenChanged
    INFO ZKDataRetrieval [Thread-2-EventThread]: Process path: /services/hs2/hadoop01.woolford.io. Event state: SyncConnected. Event type: NodeDeleted
    INFO ZKDataRetrieval [Thread-2-EventThread]: Process path: /services_config/hs2/hadoop01.woolford.io. Event state: SyncConnected. Event type: NodeDataChanged
    ERROR ZKDataRetrieval [Thread-2-EventThread]: Can not get data from /services_config/hs2/hadoop01.woolford.iowith error: KeeperErrorCode = NoNode for /services/hs2/hadoop01.woolford.io
    INFO ZKDataRetrieval [Thread-2-EventThread]: Process path: /services/hivemeta/hadoop01.woolford.io. Event state: SyncConnected. Event type: NodeDeleted
    INFO ZKDataRetrieval [Thread-2-EventThread]: Process path: /services/hivemeta. Event state: SyncConnected. Event type: NodeChildrenChanged
    INFO ZKDataRetrieval [Thread-2-EventThread]: Process path: /services/hivemeta/master. Event state: SyncConnected. Event type: NodeDeleted
    INFO ZKDataRetrieval [Thread-2-EventThread]: Process path: /services_config/hivemeta/hadoop01.woolford.io. Event state: SyncConnected. Event type: NodeDataChanged
    ERROR ZKDataRetrieval [Thread-2-EventThread]: Can not get data from /services_config/hivemeta/hadoop01.woolford.iowith error: KeeperErrorCode = NoNode for /services/hivemeta/hadoop01.woolford.io
    WARN Alarms [RPC-7]: Alarm raised: NODE_ALARM_SERVICE_HS2_DOWN; Cluster: my.cluster.com; Node: hadoop01.woolford.io; Message: Can not determine if service: hs2 is running. Check logs at: /opt/mapr/hive/hive-0.13/logs/mapr
    WARN Alarms [RPC-1]: Alarm raised: NODE_ALARM_SERVICE_HIVEMETA_DOWN; Cluster: my.cluster.com; Node: hadoop01.woolford.io; Message: Can not determine if service: hivemeta is running. Check logs at: /opt/mapr/hive/hive-0.13/logs/mapr

Outcomes