The Jepsen site is an essential resource for anyone thinking seriously about building software deployments at scale. Run by Kyle Kingsbury, he tests clusters of datastore-type systems by firing data at them, introducing internal partitions and failures, and comparing what the system told the clients that it had stored with what it had actually persisted.
As he says on his website:
Jepsen pushes vendors to make accurate claims and test their software rigorously, helps users choose databases and queues that fit their needs, and teaches engineers how to evaluate distributed systems correctness for themselves.
Jepsen offers lectures and workshops on understanding distributed systems, and the test framework for distributed systems verification is freely available on GitHub: https://github.com/jepsen-io/jepsen