Fork me on GitHub

Architecture of the System

architecture diagram

replikativ consists of two parts, a core of CRDTs (convergent replicated datatypes), especially a newly crafted one for the git-like CDVCS datatype (a bit more polished PaPoC 2016 paper) in the replikativ.crdt.cdvcs namespaces, and a generic replication protocol for CRDTs in replikativ.core and some middlewares. The fetch middleware deals with pulling all referenced data and the hash middleware ensures cryptographic integrity. The replication can be externally extended to any CRDT (as long as all connected peers support it then). We will provide as many implementations as possible by default for the open, global replication system. Together the CRDTs and the replication provides conflict-free convergent replication. The datatypes decouple resolution of application level state changes from replication over a network.

The replication protocol partitions the global state space into user specific places for CRDTs, [user-id crdt-id]. All replication happens between these distributed instances. All peers are supposed to automatically replicate CRDTs of each user they subscribe to.

We make heavy use of core.async to model peers platform- and network-agnostic just as peers having a pair of messaging channels from kabel for edn messages. We build on platform-neutral durable storage through konserve. At the core is a pub-sub scheme between peers, but most functionality is factored into middlewares filtering and tweaking the in/out channel pair of each peer’s pub-sub core. This allows decoupled extension of the network protocol.

For a detailed documentation of the CDVCS implementation you can have a look at the introduction. Or to understand the pub-sub message protocol for replication.

The joint system has won a best poster award at EuroSys 2016.