AnsweredAssumed Answered

Zookeeper Errors After 2.1.2 Upgrade

Question asked by impermisha on Mar 27, 2013
Latest reply on Mar 28, 2013 by impermisha
After our upgrade to 2.1.2, Zookeeper on 1 node in particular has seemed unstable.  We have noticed it spewing massive amounts to the log files.  It seems to happen both when it was a leader and a follower.  If you stop and restart the service it goes away - for a time at least. The other zookeeper nodes don't have that problem. The node that has this problem is the same one that has alot of the other services running like CLDB, file server, web server, nfs server, hoststats.

    -rw-r--r-- 1 root root        0 Mar 27 00:00 zookeepercleanup.log
    -rw-r--r-- 1 root root   433472 Mar 27 16:08 zookeeper.log
    -rw-r--r-- 1 root root 10485818 Mar 27 16:08 zookeeper.log.1
    -rw-r--r-- 1 root root 10485992 Mar 27 16:07 zookeeper.log.2
    -rw-r--r-- 1 root root 10485803 Mar 27 16:06 zookeeper.log.3
    -rw-r--r-- 1 root root 10486078 Mar 27 16:05 zookeeper.log.4
    -rw-r--r-- 1 root root        0 Mar 27 16:08 zookeeper.out

The other ZKs don't show that

    -rw-r--r-- 1 root root      87 Mar 27 00:00 zookeepercleanup.log
    -rw-r--r-- 1 root root 2650788 Mar 27 16:06 zookeeper.log
    -rw-r--r-- 1 root root       0 Mar 17 22:48 zookeeper.out
    
    -rw-r--r-- 1 root root       0 Mar 27 00:00 zookeepercleanup.log
    -rw-r--r-- 1 root root 1388047 Mar 27 16:15 zookeeper.log
    -rw-r--r-- 1 root root       0 Mar 17 22:59 zookeeper.out


The error

    java.nio.channels.CancelledKeyException
       at gnu.java.nio.SelectionKeyImpl.readyOps(libgcj.so.10)
       at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:241)
    2013-03-27 16:07:23,051 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn$Factory@272] - Ignoring unexpected runtime exception
    java.nio.channels.CancelledKeyException
       at gnu.java.nio.SelectionKeyImpl.readyOps(libgcj.so.10)
       at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:241)
    2013-03-27 16:07:23,052 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn$Factory@272] - Ignoring unexpected runtime exception
    java.nio.channels.CancelledKeyException
       at gnu.java.nio.SelectionKeyImpl.readyOps(libgcj.so.10)
       at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:241)
    2013-03-27 16:07:23,053 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn$Factory@272] - Ignoring unexpected runtime exception
    java.nio.channels.CancelledKeyException
       at gnu.java.nio.SelectionKeyImpl.readyOps(libgcj.so.10)
       at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:241)
    2013-03-27 16:07:23,054 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn$Factory@272] - Ignoring unexpected runtime exception
    java.nio.channels.CancelledKeyException
       at gnu.java.nio.SelectionKeyImpl.readyOps(libgcj.so.10)
       at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:241)
    2013-03-27 16:07:23,055 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn$Factory@272] - Ignoring unexpected runtime exception
    java.nio.channels.CancelledKeyException
       at gnu.java.nio.SelectionKeyImpl.readyOps(libgcj.so.10)
       at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:241)
    2013-03-27 16:07:23,056 - WARN
       [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn$Factory@272]
       - Ignoring unexpected runtime exception

Outcomes