Opening Up Consensus: Bitcoin Kernel

Does anyone have an idea what we're seeing on the chart over there? Just have a think. You don't have to answer. Nodes. Exactly. This is public data from Bitnodes going back to 2015. The striking thing here is that since basically the beginning of the timeline, all of the Bitcoin full node implementations that you see are Bitcoin Core or some very closely related variant of it. During the block size wars, we had Bitcoin Unlimited and ABC and a bunch of others. And obviously now in recent years we have the Knots faction taking increasing market share as well. But really, under the hood, all of this is Bitcoin Core.

​

In this talk I want to talk about one approach that might help improve the situation a bit. I'm Stefan, stickies-v. I'm funded by Brink to work on Bitcoin Core, and specifically I've been focusing on the Bitcoin Kernel project. Besides my developer work, I also host the London Socratic and I host Bitcoin Core Review Club.

​

Bitcoin is an emergent network, and that means it exists if and when people agree on the exact set of consensus rules that defines whether or not a block is valid. There is no central authority that determines what those rules are. It's not Satoshi. It's not a white paper. It's not Bitcoin Core developers. It's not a community. There is no voting. Consensus is whatever people actually end up running on their machine to validate those blocks.

​

One of the most crucial properties about consensus is that you want to avoid consensus failure. Consensus failure is essentially when, unintentionally, people who used to agree on consensus rules start disagreeing, which would typically lead to a chain split, which is extremely hard to recover from.

​

Because chain splits are so catastrophic, we end up with this interesting first mover advantage. Essentially, because Bitcoin Core has been around for so long and has gained massive market share in the full node market, running anything that is not Bitcoin Core puts you at quite a strong disadvantage. In case of consensus failure, in case of a chain split, it really does not matter who is right in an academic sense. It doesn't matter who better follows Satoshi's vision or who didn't have the bug. The only thing that matters, especially for you as a person running the node, is: am I going to be on the chain that survives? Am I going to be on the chain that is the most economically valuable? Because essentially Bitcoin is money, so that's what you want to maximize for.

​

Unless you have really significant reasons to run something that is not the majority node implementation, for example because you need certain features for your business or for your operations, or because you don't think the majority implementation is good software, you're probably just going to stick with whatever is majority if it works well enough for you.

​

I think the two main reasons that Bitcoin Core has been such a predominant implementation are, one, it has a very good track record of being a high quality software project. It is very well tested, it is very well reviewed, and it's got a very solid track record. All of that speaks to its strengths. But also, it does have the first mover advantage. It's been around for the longest. It has the majority, and so deviating from that is incredibly difficult.

​

I think that status quo makes sense. I think that's reasonable given where we are, given that Satoshi started Bitcoin without a formal specification. But I also think we should think about doing better for two main reasons.

​

One is, does anyone in this room think that free markets or competition are a bad idea? I see one hand. I can't see who that is, but I think I have an idea. I thought it was Shinobi. So I guess everyone else agrees. We want to have competition. It's better for end users if we can have different full node implementations that cater to different audiences, different use cases, different priorities, different philosophies. Ultimately, if we can do that in a safe way, that is going to benefit end users.

​

The second reason why I think this is important to think about is that it takes the pressure off of Bitcoin Core. The more that we can decouple the Bitcoin protocol and consensus rules from Bitcoin Core, the more the Bitcoin Core project has the freedom to make decisions and changes that it thinks are best for its users without causing some of the controversies and pushback that we have seen in recent times.

​

The project I'm working on with another group of people within the project is Bitcoin Kernel. What is Bitcoin Kernel? It is a stateful library that wraps all of the consensus logic within Bitcoin Core. I will walk you through that in a bit more detail.

​

Before Bitcoin Kernel, imagine this box is the Bitcoin Core codebase. We have some consensus logic indicated with the gear icon. I'll be calling this the consensus engine. And we also have non-consensus code, which is the other icon. Before Kernel, it was all tangled up a bit. There was no clear separation between which logic is consensus and which logic is not consensus.

​

The first thing we did is stage one: we're going to separate that. On the right you have the gear icon for consensus logic, and on the left you have everything that is not consensus code. This is purely internal, so everything is still within Core. That stage is mostly finished.

​

Now we've entered stage two, where we have the same diagram, but on the right we have the engine, which is Kernel. Because we're adding a public interface on top of this library, we can start to see competition among full node implementations that all use the same consensus engine. One of them could be Core, but the others could be any full node in any language, any stack, and with any priorities that you want to see.

​

Ideally we would go even further and move toward something based on a formal specification, so that multiple consensus engines can be built and formally verified against that specification. But it's out of scope for this talk, and it's also much less clear how feasible that is. Whereas Kernel is actually here.

​

The purpose of Kernel is that we want to have this library that is usable, useful, and the best choice for anyone who wants to use or build on Bitcoin consensus. That imposes quite a few requirements or design goals for the library.

​

The first one is that it should be usable in any tech stack. Whether you're developing in Rust, C, or JavaScript, if you want to use consensus, you should be able to use it. That's also why we chose to publish a C interface for the Kernel library. C has existed for a long time, and architecturally it works well to be used in multiple languages through its FFI interface. Pretty much any language supports using a C library. As a matter of fact, we currently have language bindings for Kernel in Rust, C++, Python, Go, C, Java, and I'm probably forgetting a few more. So that is actually working.

​

The second requirement to make this usable for everyone is that it must be performant. Bitcoin Core has already optimized performance quite a bit, and we want to make sure that we keep that property. Things get bad whenever people have to reimplement consensus logic for their own application because Kernel is too slow. Whatever we do, we cannot sacrifice on performance too much, and we try to squeeze as much as we can out of that little juice box.

​

A third property is that it absolutely needs to be secure and robust. To come back to the previous slides, the whole point of Kernel is to try to prevent consensus failures when people reimplement their own full node or any other business use case. In building this interface, we do everything that we can to remove foot guns from the interface. We try to make it as hard as we can to accidentally mess up and accidentally cause consensus failures.

​

That's the high-level theoretical overview of what Bitcoin Kernel is. So is it ready yet? It is usable, but it is not finished. It is currently in an experimental phase. That means the interface is not versioned. We can and will introduce backwards-incompatible changes at any time while we're trying to make as much progress as we can without being held back by versioning. But the underlying logic, the actual consensus logic, is the exact same code that is used within Bitcoin Core. Of course, it has a long track record. It is well tested. It is usable.

​

It is usable, but it is not finished and it is not stable yet. I just want to quickly walk you through an example here. This is a Python code snippet that you see on the screen using the Python language bindings, and I want to highlight some of the ways that Kernel works.

​

In this snippet, I want to show you how you can build a very simple full node. Imagine you're on an island with a couple of hundred other citizens. You have no internet, but you have access to a satellite that can stream you blocks. So you just get bytes from the satellite, and you also have a mesh network on the island. It's pretty low bandwidth, so you can't stream all the blocks and you can't run a full node on it. But you have enough bandwidth, for example, to stream transactions to your fellow citizens. You're the Python guy. You're the Bitcoin guy. So you're building this node so that the whole island can live in a Bitcoin economy.

​

The first thing we're going to do is instantiate a chainstate manager, which is the entry point into Kernel. It holds the UTXO set and the chainstate. It does disk operations and database operations. It really is the engine and the starting point for using Kernel. In this case, we're going to initiate this on mainnet, and we're also attaching some callbacks. In this case, we use one callback that fires whenever a block is disconnected from the tip, which usually represents a reorg, like removing one block from the tip.

​

In this case I implemented a very simple function, on reorg, that just shoots an alert to our fellow citizens to inform them that we've had a reorg, be careful, because there might be double spends happening, so double check what's happening.

​

Then, once we have the chainstate manager, we can start to actually do something with it. For example, in this case we start by getting blocks from our satellite. This is just a byte stream. We use Kernel to deserialize those bytes into a block object. Then we're going to use this same consensus logic that we use within Core to make sure that the satellite is not cheating us, that the data we're getting is actually a valid block, and that it adheres to all of the consensus rules that we want to stick to.

​

We do kind of trust the satellite, so we expect it not to send us bogus data. But just in case something is wrong, if it actually ends up being an invalid block, again, we can shoot an alert message to our fellow citizens that something seems to be wrong, so make sure you inspect that data.

​

After we've processed a block, we've now updated the chainstate, our UTXO set is updated, the chain is updated, and all the kind of magic stuff that is necessary when you're doing validation has happened. Now we can implement some other logic specific to our use case.

​

This highlights that you can also use Kernel to inspect the blockchain. You can iterate over blocks, transactions, and outputs. You can use that for data analysis or build any kind of data engineering pipelines that you want. It's much faster than using the RPC interface that previously you probably would have used.

​

In this case, we're just going to iterate over the transactions in the block and then over all the outputs. We're going to compare the outputs to a list of scripts that we have saved. Because we don't care about privacy, we're just reusing scripts. We're a remote island. We have to make some concessions. As soon as we see that there's a match between output scripts that our citizens have and ones that we find in a block, then we know these blocks might be sending transactions to our citizens. So you use the mesh network to forward all of these to the rest of the island, and that's it.

​

That's a full node. You just used Kernel to build your own node that does something that you want to do, and you can reuse all of this logic that Bitcoin Core has.

​

To recap, Kernel is a stateful library. It wraps the Bitcoin Core consensus engine. You can use it to build anything on consensus. I think it's a cool first step to help increase competition on full nodes across the ecosystem. We would love for anyone in this room to use it and give us feedback on how it works for you, or if you want to change anything.

​

On that note, I am hosting a workshop right on the corner right after this talk. If you want to actually build something, bring your laptop. We're using Python and it'll be interactive. You'll walk away having built your own full node in less than 45 minutes.

​

Thank you for showing up. If you're interested in Kernel, I hope to see you there.