tags: distributed systems, data analysis
A distributed file system, is a system that stores files in multiple locations and servers, but appears to the client as a single file system and the files are stored in a single server.
DFS Types
The difference between CFS and NFS, is that NFS are on a local network.
Google File System
Assumptions
Architecture
Files are divided into chunks or shards and they are also replicated according to a replication factor.
The last point is explained in this figure.
The master does not write the files but it writes only metadata.
The metadata is stored in the RAM of the master server so we can have very fast lookups.
Updates in the metadata is also stored in disks. This is done so during crashes we first load a snapshot of the metadata that was stored in memory and then we apply the same update that are stored in the log of updates done to the metadata.
Chunk Server
If a chunk-server does not send its heart-beat for a while, the server is assumed dead and the master either distributes the data the chink server had among other servers or allocates a new server.
Consistency Mode
Defined data means that it can not be mixed from two different writers, We have a defined protocol for edits, like for example the one who gets access first.
Consistent is not the same as defined
TODO: read the paper
Write mechanism
The DR said the master tells the client which replica is the primary. After the client writes to the closest, arrow number 3, it requests confirmation from the primary replica which finalizes all writes end sends a confirmation. Also DR said that the replicas should confirm the writes with the master.
Also if a client requests a chunk that has some serial, other chunk servers that don’t have the serial will be discarded.
