Grid FS
Grid FS is a distributed files system within MongoDB, it stores the files as binary data in documents in the chunks collection. The size of a chunk is 256MB.
The downside of this file system that it is a flat file system, there is no directories. If you push a file with the files original directory becomes its name.
We will have to collections, df.fs.files for metadata and db.fs.chunks for binary data it will be in the data field
Hadoop FS
A distributed file system, similar to GFS. The master server here is named name-node and chunk servers are name data-nodes.
HDFS provides traditional file system. It provides user quotas, access permission, Linux naming convention, and Linux like access command.
NOTE: Start any command with
hadoop fs
NOTE: HDFS does not support any type of file links. Also HDFS only supports appending to the end of the file. Random writes are not supported
“HDFS applications need a write-once-read-many access model for files. A file once created, written, and closed need not be changed except for appends and truncates. Appending the content to the end of the files is supported but cannot be updated at arbitrary point. This assumption simplifies data coherency issues and enables high throughput data access. A MapReduce application or a web crawler application fits perfectly with this model.” - Simple Coherency Model
Other assumptions are in the slides
| NameNode | DataNode |
|---|---|
| It executes file system namespace operations like opening, closing, and renaming files and directories. It also determines the mapping of blocks to Data-Nodes. | It is responsible for serving read and write requests from the file system’s clients. Also, it performs blocks creation, deletion, and replication upon instruction from the Name-Node. Blocks are stored on the data nodes. |
NOTE: Replication factor can be set any time to any file but will fail if there is no enough datanodes.
