I would like to know if the chunk size in Mapr is dynamic or static
Eg: if the set the chunk size to 64 gb, and file size is of 50gb wat will be the chunk
Chunk size is configurable, default is 256MB. ChunkSize define how a file is split into multiple chunks across different nodes. On your question about 40MB file with 64MB chunkSize, it uses only 40MB of the disk space. Blocks to each chunk are dynamically allocated.
BlockSize is different from ChunkSize. BlockSize is the disk block size used by MapRFS to store data of each file chunk. BlockSize is fixed at 8K. If you write a file smaller than 8K it will still consume 8K of disk space.
Hi sumathi saravanan,
I am not very clear what you meant by "dynamic or static"? Do you mean if the chunk size can be modified? Here is a doc explaining MapR chunk size and how to set chunk size: Chunk Size - MapR 5.0 Documentation - doc.mapr.com
Hope it is helpful.
Lets say I set the chunk size to 64 mb
Then am writting a file which of smaller size than chunk size like 40mb
In 64mb chunk it will take 40mb, then wat will happen to thw rest 24mb left over size in the chunk
sumathi saravanan MapR-FS works on pre-defined 8K filesystem blocks. To the best of my knowledge, this is not configurable by common users. This has been defined so considering the benefits of handling smaller blocks during I/Os.
If it were tunable, just to try and answer your query, any file size smaller than the defined default block size would leave the remaining space within the block as unused.
Hope that helps.
Thanks Mufeed Usman,
Gone through few documents where the block can be set from 64m to 128m something,
You have mentioned 8k block size which is pre defined, and it can not be changed could you plz add more info on that and it will be gr8 if you can share some documents for it.
Thought I might add this link here for a fair understanding on the way how compression work on MapR-FS data blocks - Interpreting compression ratios in hadoop fs command outputs
Thank you all.
You can also show your appreciation by endorsing Chandra and Mufeed's top skills under their Profile. Here is How to endorse someone's top skills.
Retrieving data ...