How to find block size in MapR?

Document created by schandhok Employee on Feb 8, 2016
Version 1Show Document
  • View in full screen mode

Author: Shashank Chandhok

 

Original Publication Date: May 12, 2015

 

The default chunk-size in MapR is 256M (the equivalent of the HDFS block size which is typically 64M). In MapR, when a file is written, it inherits the chunk-size from a setting in the directory where it is created in. So, if a file /a/b/c/f is written, it gets its chunk-size from the setting on the dir /a/b/c. To display it, use the command.

 

% hadoop mfs -ls /a/b/c/f # displays the chunksize for the file f, along with other info

Found 1 items

-rw-r--r-- Z U   1 root root          4 2015-05-12 00:21  268435456 /mapr/demo.mapr.com/a/b/c/f

               p 2049.1119.4332154  maprdemo:5660

% hadoop mfs -lsd /a/b/c # displays the setting on the dir c, along with other info

Found 1 items

drwxr-xr-x Z U   - root root          1 2015-05-12 00:21  268435456 /mapr/demo.mapr.com/a/b/c

               p 2049.1118.4332152  maprdemo:5660

% hadoop mfs -setchunksize 134217728 /a/b/c # sets the chunksize

% hadoop mfs -lsd /a/b/c

Found 1 items

drwxr-xr-x Z U   - root root          1 2015-05-12 00:21  134217728 /mapr/demo.mapr.com/a/b/c

               p 2049.1118.4332152  maprdemo:5660

 

Alternatively, you can view it via NFS by cat'ing the magic/hidden file .dfs_attributes like so:

 

% cat /mapr/demo.mapr.com/a/b/c/.dfs_attributes

ChunkSize=268435456

Compression=true

% vi /mapr/my.cluster.com/a/b/c/.dfs_attributes # edit the file to modify chunksize

 

NOTE: The magic/hidden file .dfs_attributes is not listed in the 'ls -la' output but can be operated on just by specifying the file name. This file is available under all the MFS directories.

 

Attachments

    Outcomes