Experiences with Scaling Blockchain-based Data Stores

In April, we were invited to Microsoft to talk about our decentralized identity work and the underlying technology that we're building. The visit included a talk at Microsoft Research (MSR), which is embedded below. Speaking at MSR was especially interesting, because researchers in distributed systems are not paying enough attention to the blockchain and it'd be great to get seasoned distributed system researchers to start paying attention to this area.

The talk basically had three parts:

  • An introduction to Bitcoin and Blockchain
  • Why we need decentralized identity (starts 00:23)
  • Experiences from our production network plus overview of Blockstore, our blockchain key/value store (starts 00:32).

Most researchers in the audience were not that familiar with the blockchain, but picked up the concepts very quickly and asked lots of insightful questions. The talk is available from Microsoft Research here (their link plays the talk slides as well). Special thanks to Lili Cheng and Qi Lu for hosting.

(Download PDF slides)

Abstract: In the past years cryptocurrency blockchains (like Bitcoin and Namecoin) have seen significant adoption with the promise of using such blockchains as general-purpose databases and/or key-value stores. Cryptocurrency blockchains provide a zero-trust infrastructure, where users can securely store and retrieve information while providing security guarantees that only the owner of a particular private-key can write/modify the data. In theory, many decentralized services/application can be built using cryptocurrency blockchains as key-value stores. However, the area is relatively new and rapidly evolving with little production experience/data available to guide design tradeoffs. In this talk, we describe our experience of operating a large real-world deployment of a decentralized naming service, called Blockchain Name System, built on top of a cryptocurrency blockchain (Namecoin). We present the various challenges that we had to overcome while registering/updating over 30,000 users on the blockchain and discuss how our experience informed the design of a new blockchain-based key-value store, called Blockstore. All of our code is available as open-source here.