Centralization Risks: Funding, Power & Bitcoin’s Client Diversity Problem

Hey everybody. We have a great panel. We have two Erics and a Jon, and we’re going to start with an origin story. Let’s start with Eric Voskuil. When you started working on Bitcoin, what was the hook that caused you to be captured by Bitcoin and made it something you were going to work on for the rest of your life?

​

I got started on Bitcoin after my third software startup. I was kind of burned out on that and was looking to do something different. The trigger was that I read the Andy Greenberg Silk Road article in Forbes Magazine, and all of a sudden I went right to the white paper and started reading. I’ve been working on it pretty much every day since then, with Philip, who was at my last startup at the same time. So about 13 years ago.

​

It just combined a lot of things I liked. I was ex-military. I did a lot of tech startup stuff. Economics. I’m a longtime libertarian type. So all the things that I like doing all fit together.

​

Hi, everybody. My name’s Jon. My origin story was around 2014 or 2015. I was interested in Bitcoin. I was doing tech lead and corporate CTO innovation lab roles for European multinationals in software development, and I was orange pilling my colleagues at stand-up meetings every day. I was interested in Bitcoin and following it.

​

One day, around 2015, Vladimir van der Laan, the lead maintainer of Bitcoin Core for nearly a decade at that time, followed me on Twitter. So I started looking into Bitcoin Core, the project. I really liked Vladimir’s leadership style as a humble servant leader on the Bitcoin Core project.

​

I had some significant open source experience maintaining open source projects and contributing to large open source projects as well. I had certain opinions about what I did not like in open source leadership and what I did like. Vladimir’s leadership style ticked all the boxes, and I was interested in Bitcoin. So that gave me the initial inspiration to look at working one day on Bitcoin Core, which I began to do in 2019.

​

My background comes more from commercial software development. I worked at a few startups earlier in my life. Then around 2011, I came across Bitcoin, and it was this super exciting idea that I thought was really neat. At the beginning, I just thought it was a fun idea that maybe would get some traction. Nobody really knew, but it seemed like a really neat idea.

​

I’d never really done much open source stuff before, so I hopped on. It was really an interesting experience to hop onto the whole thing. I got to see a lot of the earlier stuff that happened in Bitcoin, and it’s been a crazy journey since then. A lot of stuff has happened, a lot of new things. It’s crazy that we’re here, that we can actually organize this kind of event and that so many people are so interested in this right now. It’s mind-boggling to look back on it. Back then, only a handful of people cared about it. It was hard to fill a room even this size.

​

We have three topics we’re going to discuss: funding, power, and Bitcoin’s client diversity problem. Let’s start with the client diversity problem. What does that even mean?

​

I know it’s something you made up.

​

It’s something I made up?

​

I think you submitted it to this panel.

​

Well, I guess some people might consider it a problem, and others might not. When we say client, we’re talking about a node, which can be confusing sometimes. I always think of a node in Bitcoin as a direct analogy to a router for the internet. You can’t outsource your router, and if you don’t have one, you’re not on the internet. Same thing with a node. You can’t outsource it. If you don’t have it, you don’t have Bitcoin. You have a promise, maybe, of something.

​

Nodes are important. The Satoshi implementation, which is certainly the predominant node out there, was first and is most widely used. But it’s also very conservative because it’s so widely used, and it doesn’t have the opportunity to take on big architectural changes.

​

In 2011, Amir Taaki, who started our project, Libbitcoin, made not the only one, but the very first attempt to make another node. We’ve taken on major architectural changes and approached things in a different way.

​

From the perspective of the problem, the problem is that without competing software implementations, one can become very stagnant and not evolve. We look at Satoshi’s implementation as a prototype, and I think that’s truly the right way to look at it. It was an attempt to get something working. Rule number one in software engineering is you never ship your prototype, because you get stuck with it. Everything has to be compatible with it.

​

So we think it’s important that there be other nodes out there, not just from the perspective of having different choices and more software. The node community builds these things and maintains them, and that community becomes the experts in the field. The technical experts in Bitcoin are really the people who can build. When we say core software, Bitcoin Core is a brand, but core development before Bitcoin Core was called that was really just building nodes.

​

You need experts who can do that. Different implementations allow those experts to compete both in software and for people’s support, especially around things like soft forks, hard forks, different protocol changes, policy, and things that have become very prevalent recently in terms of conflicts. If there’s only one group, it tends toward groupthink and lack of evolution.

​

I think we can agree that it works pretty well, right? It’s been the primary client for a pretty long time.

​

It’s incredible that it works. It’s like a miracle that it works so well. It’s unbelievable.

​

Where are the limitations? What’s wrong with it? Why would we need alternative implementations? It’s hard to do a teardown because it would be mid-flight, right? But what are the current limitations that you see? What would you clean up if you could fix one thing?

​

Most people would answer that from a technical point of view, but I would answer from a different point of view. But go ahead.

​

I think the first original implementation was a prototype. It was a proof of concept. I don’t think Satoshi really knew whether it was going to work or not. It was just kind of like, let’s see if this thing flies.

​

There are a lot of unanswered questions about how to maintain it long term. I think there was no succession model for how to organize the project or leadership. It has this model where you have maintainers who take care of the code and make merges. It’s open to anyone to submit pull requests, but I think most of the actual work, or the hardest part of the development process, is the review process and the testing process. It’s much easier to write code than to read other people’s code.

​

I think it has gotten to a point where that is a huge bottleneck. It worked when it was a small project, but the bigger it grows and the more people submit pull requests and try to make changes, even without getting into consensus stuff at all, the harder it is to review. What that has led to is small incremental changes to the codebase that limit ambition and mean you can’t really change too much. If you want to do a major refactor or major rearchitecture, it’s too difficult to review. The process right now basically doesn’t allow that.

​

That’s not even getting into consensus changes like soft forks. I was very heavily involved in the whole SegWit soft fork and in some soft forks before that. Before SegWit, basically everything flew under the radar. Nobody knew, except for the handful of developers working on stuff. Nobody in the ecosystem even knew that soft forks were happening.

​

I think SegWit was really the first major controversial one that got a lot of spotlight. Ever since then, it has basically become impossible. We’re at a gridlock as far as how we’re actually going to change the code. I don’t think Satoshi ever really thought through how we would change consensus rules if we were ever to change consensus rules.

​

There are a few quotes about how things needed to be cast in stone, but there is a lot of new cryptography that has come out in the last 15 years that is really cool and would be really nice to add at the base layer. We just don’t have a way to do it.

​

I think we need some kind of better model for how to specify the Bitcoin protocol itself and to have a clearer way to ensure that different implementations are actually interoperable, so that we can have different approaches. There can be multiple projects working in different ways. Everyone can have their own style. As long as they’re all interoperable, it shouldn’t be a problem.

​

Right now, it has been really hard to get out of this model where Bitcoin Core is the reference implementation and everyone kind of uses it as the spec. It’s not a spec that is very human readable. It’s not a spec that is really easy to verify, even with machines. We need something better.

​

How do you solve the problem? If everybody is here in Vegas, can’t we just get together over some beers? Jon, what’s the pathway?

​

This is a topic I spend a lot of time thinking about. I think there is a growing awareness that it is an issue to have one legacy implementation that is considered the reference and considered the spec, to which sometimes the BIPs are even modified because the spec may be different in Bitcoin Core. So Bitcoin Core de facto becomes the spec, and the BIPs need to be modified.

​

There is a confluence of things. There was a leadership transition over the past half decade. The culture in Core has changed in a few ways. I think there is a growing awareness that if we had a spec that different implementations could formally verify to ensure correctness and interoperability, it would be a good thing in terms of competition as well. It would be healthy, I believe. But it is also more difficult.

​

The solution would require more funding, not less, because you would have not one Core that is taking, I don’t know, a $10 million to $20 million per year budget. You would need several of these in order for there to be a relatively even playing field and a reasonable bar that different implementations could reach.

​

That is where we’re seeing some current discussions around funding. Before, Bitcoin Core development was sort of automatically earmarked to the reference implementation. The questions now are more: does funding also need to be earmarked, maybe equivalent amounts, for competing implementations or for supporting Libbitcoin or other ones that might be in the works? That would require growing the pie rather than cutting smaller slices of the same one, I believe.

​

You said the culture has changed. How has it changed?

​

When I began working on Core around 2019, it was a very different project with a completely different set of leaders and a different leadership style. The leaders completely changed in terms of the maintainers.

​

Also, nowadays the best way to make a career in Core is more or less to join one of the main big offices located in New York City, London, and San Francisco, as well as one that might be growing in Amsterdam. If you are not willing to relocate into one of those offices and mentor under the guidance of one of the senior people there, it is a much more difficult path. Let’s say you’re married with a mortgage and living in another country in Europe or elsewhere on the planet.

​

The culture has changed more toward big cities and younger people who enter under the seniors. It might be good not only to have options for node runners, but also for developers. I’m personally based in El Salvador for four years. It would be good to have implementations that are more welcoming of more senior people with more seasoned experience who are not willing to relocate to New York City or San Francisco, to have options where the culture is more welcoming to those kinds of developers.

​

That is, of course, a little bit divisive, and that is not the only reason. But it is also a reason why it might be more useful to have different options and paths available for people who desire to work on Bitcoin.

​

It seems a bit antithetical to have centralized offices.

​

It’s not how things started. It’s not how it was when I joined.

​

It wasn’t when I joined either.

​

You also run the risk of having a lower level of scrutiny for peer review. You work next to somebody, you get lunch with them every day, you start to choose. As an independent contributor, you come across social process locks, where your review comes against a block of five or six reviewers who are allied.

​

I think we can move into the funding and power element of this panel with the time that we have left. I want to collapse the two because they are kind of the same thing at the end of the day. We referenced a couple of the offices, so we’re talking about Chaincode, Brink, Spiral, and Presidio. These are sort of the localized hubs for developers, and they largely rely on philanthropy. They have donors who have been in Bitcoin for a long time.

​

Apart from Chaincode.

​

Apart from Chaincode?

​

Spiral, which is funded by Jack Dorsey.

​

This is important to get funding there, but how is it distributed in your view? Is it distributed in the most effective and efficient manner? Or are there issues that we should be discussing when it comes to funding? Because that ends up being the centralized power we’re also discussing.

​

One thing I wanted to add is that we can talk about power as far as being well-funded. Of course being well-funded is a good thing, and I think it definitely gives you more options. But there are really severe constraints on what can actually be changed in Bitcoin just because of the nature of it. It’s a Schelling point, a very strong Schelling point. It’s not like people can just arbitrarily start making changes to whatever they want.

​

I think the issue right now is more that we’re boxed into this particular model of how to do development, and there should be different approaches. As long as they are all compliant with the same consensus rules and are all interoperable, they can all sync from each other and converge on the same chain tip. There is no issue, really.

​

I don’t even really see that as competition in the traditional commercial sense, because we’re talking about free open source software. One analogy I like to draw is a large high-rise building. There can be a lot of firms that work in it. Some of them might agree on some stuff, some might not. Some might even be competitors, but they all rely on the same building foundation.

​

The protocol, the actual spec, the consensus rules, the network messaging formats, those kinds of things are the very ground floor and foundation of the building. If that’s not built well, it doesn’t matter if people further up have ideas of how to improve the building. The building is still going to collapse. We need something very solid at the bottom that grounds everyone, so all implementations have something very solid to follow.

​

There are certain things that are very important and that all implementations must do. Then everything else is free. If you want to work at an office, fine. If you want to do it remote, fine.

​

You were talking about a formally verified specification. That would be a long-term goal. Even for Vladimir, the lead maintainer of Bitcoin Core, for years he was hoping for that to move forward.

​

But there is also the money issue. I would say that three quarters of the funding of the entire Bitcoin ecosystem comes from one very generous person, and that is Jack Dorsey. He funds not only Spiral. I believe he is the main funder of OpenSats and of many other offices that depend on funding.

​

It would be good to find other people, Michael Saylor, for instance, who has been reticent to fund developers, but other people besides Jack. Almost all of my funding over the past eight years has been from Jack. Thank you, Jack. So it is very centralized in that one way. That is not a criticism of Jack, but a criticism of other people who could step up and help him.

​

The second thing is that some funding is, for example, through OpenSats, which, full disclosure, I help review applications for. They are a more independent entity that collects donations and redirects them to developers and also to some of the offices. There is an ongoing discussion right now about what would be an ideal repartition between helping offices and helping individual developers directly.

​

There are issues with funding centralization that I think could be better, particularly more people like Jack helping Jack, because the ecosystem really depends a lot on Jack Dorsey. Thank you. I’m very grateful to Jack for his funding support.

​

We need a few more Jacks.

​

There is one more thing I want to touch on. I think there is plenty of money out there in the space to fund this kind of stuff. It’s just that people don’t really know how to productively invest it into something. What are they actually putting their money into? What are they going to get out of it?

​

Right now, with the current architecture and structure of the system, it doesn’t really lend itself to an ecosystem that can grow organically that way and support different models. It doesn’t let a lot of different people who want to fund this stuff actually promote their particular way of doing it, which can be their own unique, idiosyncratic way of funding this thing. That is totally fine, as long as they stick to the same consensus rules and can sync to the same tip as all the other implementations.

​

Eric, you and I have been working on this for a while. I also want to shout out the Human Rights Foundation because they’re starting to make a dent on this issue with their funding of developers and different projects. What’s your view on the state of funding and how it has changed in recent years as opposed to when we first went out, I guess back in 2018 or 2017, to try to get some funding for Libbitcoin?

​

My experience from when we first went out to today is that it hasn’t changed much. The vast majority of what we call core development, Bitcoin funding, or stuff around the near periphery of node development, from what I can see, goes only to projects that are somehow tied to Bitcoin Core.

​

For example, we’ve had developers who have been with us for 13 years, 10 years, 12 years. People have a long history and do good work. People are aware of it, but they have been explicitly rejected because the work is not tied to Bitcoin Core.

​

Bitcoin Core has a project that used to be called libbitcoinconsensus, and now it’s kernel or something. It’s script processing. We were told that if we adopted that, then the person could get funded. But if we maintained our own script engine, or if we didn’t have some other connection to Bitcoin Core, they weren’t going to fund us. And they weren’t funded.

​

I’ve seen this for years. I do now have one developer who is funded by OpenSats. Yes, we have had people funded. I’m not saying we haven’t gotten people funded, but it’s extremely unusual. The first foundation funding we ever got was OpenSats just recently.

​

In the past, going back 10 years, we had a miner fund a couple of our developers for a couple of years because they were looking for alternatives to Bitcoin Core. That probably came out of the 2017 SegWit thing. A few years ago, we had one whale who wanted to remain anonymous fund one of our developers for two years. Aside from a little contract work here and there from some of our guys, that’s all we’ve ever been funded.

​

We have four guys sitting right here, and myself. We have another one in Italy. We have a real team, and we’re all basically volunteers. Most are working full time. The people who have gotten funded have probably funded themselves three quarters of the time.

​

When you and I went out to do this, I told you it wasn’t going to work. I said, we’re not going to get funded. This is going back a few years. I’ve never taken money. It’s my policy. I’m retired. I’ve never taken any money to do anything related to Bitcoin, and that’s about independence.

​

I said that we’re not going to get funded until we’re already successful. Once we’re out there, we’re running, we have a chunk of the network, then people will come and fund us. But it’s not going to happen unless we volunteer to do it. That has been the case over the last 13 years. That has been my experience. We do get some, but it’s a trickle compared to what Core gets.

​

I think it’s the same for the other node projects as well. There aren’t that many, and most that have attempted this have gone away. This is a lot of work.

​

I do think there may be some change coming, and that’s a consequence of some of the things Jon is talking about: culture changes at Core. It has frustrated some people. The controversy around Knots, which has been around for a long time, as a fork of Core. We never really put it in the same category. But because there hasn’t really been an alternative, conflicts rise and people start looking for one.

​

We haven’t released a node in eight years, really. We have an old one that’s not really worth running, and we have a very significant one now that is going to be released soon. That will provide people an alternative. But there really have been no other alternatives. Knots is a fork of Core, which anybody can do, of course. There’s btcd, which I don’t know the level to which it has been maintained. I think it’s a decent project, but I don’t really see it as competitive or differentiated architecturally.

​

Knots and btcd would be happy to have developers working on the projects. They need more help too.

​

To be clear, OpenSats has made it more or less clear, at least on social media, that they are willing to fund developers for these projects. But in the year and a half, almost two years that I’ve been looking at applications, I haven’t seen one from a developer to work on Knots. So where do developers go? They go to what is seen as the only game in town until very recently, and they apply to work on Core.

​

There is definitely a chicken and egg problem.

​

I’m going to be encouraging our folks to put out more applications, because that’s kind of a new thing, at least from my perspective. You can apply and your application fans out to a bunch of these organizations, and they have review processes and boards. Over the last few years, this has become the norm.

​

I do think there is a lot of goodwill out there, and there is a desire to do more. I’m saying that up to this point, that has been very limited. I think the desire to do more is coming from these controversies where people are starting to realize there really need to be alternatives.

​

Not necessarily politically. Not about changing consensus. We don’t work on anything pertaining to changing policy or changing consensus. We just do the engineering work to make a node perform well. That’s all we do.

​

Because of these conflicts and controversies, this has become more visible. People have realized, what would happen if Core continued down this road of declining performance? It’s an architectural issue. It’s a linear validator. Eventually, we won’t be doing validation anymore.

​

A lot of the engineering design work in Bitcoin Core to solve performance issues in initial block download and full validation has been oriented toward no longer validating: assumevalid, assumeUTXO, a lot of assumptions. Eventually we’ll get miner commitments and it’s going to be done. We’re not going to validate anymore. That’s because of an engineering problem. It’s not a consensus issue. It can be solved and it should be solved, but it’s too difficult to make those substantial changes in that codebase.

​

So we set out to do it. Until recently, when we demonstrated it actually working and performing well, people didn’t think it was possible. That’s what alternatives provide: a different point of view.

​

Even on the other side, there could be a niche, perhaps, and I’m not necessarily advocating for it, for a more conservative Core. Core has a lot in flight and some very large changes both coming and just past, like the legacy BDB wallet database removal. There could also be a niche for people to have one that lags behind Core.

​

Something that lags behind Core and does things more slowly, but perhaps has backward compatibility for a longer period of time. Core only supports two versions back beyond the current version, so that’s somewhat aggressive for people who want to run nodes that are older and not upgrade, or maybe they have an old legacy wallet and they’re scared to.

​

We need slower development. There could be both progress, like Knots is a bit more progressive in some ways. Maybe Libbitcoin is completely different and very aggressive in moving the architecture. There are some developers who I believe are thinking about working on new clients that might be more featureful in some ways, more progressive even yet. Maybe a slow Core might be useful to some people as well, because Core has a lot in flight too.

​

We’re basically out of time, but last question. We’ve seen so many changes throughout the culture and community. It’s not just within the development community. We have ETFs and treasuries and Bitcoin dive bars now. When you think about the next five years of Bitcoin, real quick: optimism, concern, what do you think is in the pipeline? Is there something that could happen that would cause you to stop working on Bitcoin?

​

I’ve always been very concerned about consensus failures and, at some point, people not converging on the same chain. A lot of people started to tinker around with consensus rules early on and create a lot of shitcoins and stuff like that. I think experimentation was necessary.

​

But my biggest concern is for posterity’s sake. All of the current Core developers and the previous ones are going to retire or have retired already. There are no really strong guidelines for the next generation to know what they’re doing. That’s my biggest concern.

​

I’m not really that concerned about the price or speculation. I think as long as it holds together, the value proposition remains intact.

​

The new developer pipeline is very important, and educating and training new developers. Currently, that has been very centralized as well, at least since I’ve been around. I believe it’s improving, and I hope we continue to improve that job, as well as the funding decentralization.

​

Prediction is hard.

​

Anything going to cause you to stop? Nothing is going to cause him to stop. Look, we need more Jacks, but we also need more Erics and Jons. Thank you guys so much.