Posts

Analyze Key Features Processing and generating large data sets. Exploits a restricted programming model to parallelize the user program automatically and to provide transparent fault-tolerance. Details By distributing the workload to many machines and let them execute the tasks in parallel. Specifically, input files are split into $M$ pieces and master assign each one to an idle worker. Worker process it with the map function and save each key/value pair to one of the $R$ files according to the partioning function. When finishing processing all $M$ pieces, $R$ reduce workers will read data from the corresponding intermediate files, process it with the reduce function and save to output file eventually. ...

Analyze Key Features Coordination in large-scale distributed systems by providing general “coordination kernel” APIs that support a lot of use cases. Good performance Fault tolerance / high availability Details ZooKeeper can be used for group messaging, shared registers, and distributed lock services. Reliability By replicating the ZooKeeper data on each server that compses the service. The data is periodically snapshotted. Fuzzy Snapshot We do not lock the ZooKeeper state to take the snapshot. ...

A transaction is a way for an application to group several reads and writes together into a logical unit. Transactions are not a law of nature; they were created with a purpose, namely to simplify the programming model for applications accessing a database. 相当于数据库提供了一层重要的抽象，在编写应用程序时不用再去考虑那些能被事务处理的错误与问题了。 The Meaning of ACID The safety guarantees provided by transactions are often described by the well known acronym ACID, which stands for Atomicity, Consistency, Isolation, and Durability. ...

The main reason for wanting to partition data is scalability. Normally, partitions are defined in such a way that each piece of data (each record, row, or document) belongs to exactly one partition. Partitioning and Replication Partitioning is usually combined with replication so that copies of each partition are stored on multiple nodes. This means that, even though each record belongs to exactly one partition, it may still be stored on several different nodes for fault tolerance. ...

Replication Versus Partitioning There are two common ways data is distributed across multiple nodes: Replication Keeping a copy of the same data on several different nodes, potentially in different locations. Replication provides redundancy and can also help improve performance. Partitioning Splitting a big database into smaller subsets called partitions so that different partitions can be assigned to different nodes (also known as sharding). These are separate mechanisms, but they often go hand in hand: ...