AnsweredAssumed Answered

How can I connect to HDFS from my app?

Question asked by behrang on Jan 18, 2012
Latest reply on Jan 18, 2012 by Ted Dunning
**Warning: Hadoop newbie here! :)**


I am developing a Ruby (on Rails) app and I at the moment I am using the MapR VM for development.

I started Thrift for HBase by running this from the VM's shell:

    mapr@mapr-desktop:~$ /opt/mapr/hbase/hbase-0.90.4/bin/hbase thrift start
    12/01/17 22:25:25 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.2--1, built on 08/30/2011 17:53 GMT
    12/01/17 22:25:25 INFO zookeeper.ZooKeeper: Client environment:host.name=mapr-desktop
    12/01/17 22:25:25 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_26
    12/01/17 22:25:25 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
    12/01/17 22:25:25 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.26/jre

I could connect to HBase via the port 9090 using a Ruby gem called MassiveRecord to store and retrieve some documents.

As my documents have file attachments, I want to store the attachments directly in HDFS rather than in an HBase table. To do this I am using a Ruby gem named `ganapati`. However when I connect ganapati to the same port, I get this exception:

    Thrift::ApplicationException

What port do I have to use to connect to HDFS via ganapati? Also, the main reason I am trying to do this is that in HBase docs I stumbled upon a warning that HBase records should not be many megabytes. Does anybody have any suggestions, hints, or advices in this regard?

Thanks in advance,
Behrang

Outcomes