What Do Bitcoin Core Maintainers Do?
Speakers: Gloria Zhao
Date: August 11, 2022
Transcript By: Stephan Livera
Media: https://www.youtube.com/watch?v=a61lUwlOF80
podcast: https://stephanlivera.com/episode/404/
Stephan Livera: Gloria, welcome back to the show.
Gloria Zhao: Thank you for having me again.
Stephan Livera: Yeah, always great to chat with you, Gloria. I know you’ve got a lot of things you’re working on in the space and you recently took on a role as a maintainer as well, so we’ll obviously get into that as well as all your work around the mempool. So do you want to just start with — and we’re going to keep this accessible for beginners — so you’ve taken on this role as one of the maintainers of Bitcoin Core. So what is a maintainer? What do they do?
Gloria Zhao: Yeah, so people often focus on the technical, you have keys to — you have write access to — the repository, but I mean it’s easy to understand why people get really fixated on that because that’s the threshold of, Okay, your keys got merged. But a maintainer, I would say, is more conceptually like any software project: someone who takes on the responsibility of maintenance of that code. And so even if nobody was opening pull requests to request new features or point out things to Bitcoin Core, you would still have the job of finding bugs, fixing bugs, monitoring things upstream, updating dependencies, making sure we support things continuously downstream, cutting releases — to do that on a regular cadence, taking care of what happens on the GitHub repository — responding to issues and triaging pull requests and all those things, running CI — testing things. And so every software project requires maintenance. There’s a really good book by Nadia Eghbal on this called Working in Public — I’m sure someone’s mentioned that. And in the book she estimates like up to 70% of software project activity is actually maintenance work, and I think actually it’s more than that in Bitcoin Core because A) we are quite wide in scope, so we have everything from math libraries to managing ports, all the way up to a GUI, we have translations for dozens of languages — it’s quite wide in scope. We support many different architectures, and on top of that, we do deterministic builds for all those architectures. We have a unique release process — so: wide in scope, and B) we’re very very security-focused, so what might not really be a problem in another project, we’re like, We need to be on top of that. And that translates to everything as well where like upstream — the maintainers are constantly monitoring upstream projects, and almost like needing to make sure that things are going okay over there, and then if that’s not very well maintained then maybe we need to subtree something or we need to see if we can help maintain things up there. And then similarly downstream, I look at Lightning implementations, for example, because that’s $100 million dollars of Bitcoin is locked in those projects that depend on certain assumptions and Bitcoin Core being true. And so hopefully this gives you a picture of — conceptually — what is required to maintain Bitcoin Core. It’s not mutually exclusive with the role of maintainer — there are a lot of people who don’t have write access who do a lot of maintenance work.
Stephan Livera: Yeah, sure. So let’s break some of that down a little bit so just for listeners who are totally new. Part of the understanding with Bitcoin is we’re operating in this open source environment meaning: anyone can read the code, anyone theoretically should be able to contribute to the code, and there are different ways you can contribute to the code. One of those is by looking at something that you want to change in the code base and saying, Okay, I’m going to open a pull request. But that’s only one form of contribution — importantly. Secondly, there’s things like reviewing those pull requests, and there people will give feedback and say, Oh look, I think you need to fix this part of the code because you might be introducing another security problem. And so then part of the maintainer’s role is also doing a bit of a review but then also assessing the comments, the contributors, the reviewers, and then making that decision whether a particular change or something gets merged into what we call Bitcoin Core, right?
Gloria Zhao [4:30]: Yeah, exactly. And hopefully I’ve made it clear that the vast majority of changes are just like bug fixes or fixes of some thing not being perfect, and then a very small minority are maybe something like Taproot or something like Package Relay where you’re trying to make a protocol change. And in that case, I think Bitcoin Core is very cognizant of A) Bitcoin Core is not all of Bitcoin and so there’s a lot of community consensus that is required to make a protocol change. But also, B) Okay, since it is a majority of the nodes on the network, then we need to be cognizant of like if we make a change, how does this impact the health of the entire system? And I think you had a question that you were going to ask me that I took a note on which is like: Bitcoin Core doesn’t make decisions on behalf of Bitcoin as a whole, right? A: It’s a small minority of pull requests, but it’s mostly just making sure the code is working properly.
Stephan Livera: Yeah, and actually just to touch on that as well: could you explain for us what’s the difference between let’s say a non-controversial change that’s just like changing the color of a dialog box, versus something that is more like meaningfully, Oh okay, if something goes wrong here, it could really screw up the way the P2P networking works, or the more controversial aspects. How does that distinction work?
Gloria Zhao [6:14]: Yeah, exactly. So we have consensus rules which are the validation of blocks, so I think people might have a good understanding of like, Okay, every node on the network needs to be on the same page with respect to consensus. Another thing that would need community approval on is something like a P2P protocol change, which is how the nodes talk to each other, so if they’re not using the same protocol messages then that’s a problem. I would say those are the two on the extreme side, and then, like you said, on the other side you have the colors of the buttons. But even something like a translation string — like I said, our security bar is really really high, so we’ve had issues in the past where someone translated something in a language that the maintainers didn’t understand into, Hey, if you have a problem, send your Bitcoins to this address, for example. And that’s a security issue even though it’s at the very cosmetic level of the node software. But sorry to go back to what your question was of levels of controversy: so consensus is very much like you have to be monitoring how the community is responding to something, but those are very rare. But maybe something similar to that like: okay, so we have a scale of extremely invasive and dangerous to on the more cosmetic side. Maybe like a change to mempool policy would fall a bit closer on this end, but a change in the RPC help string would be closer to [that] end. But if we’re going to change the public-facing API, of course we need to have a proper deprecation cycle and whatnot. And so in that case, review and assessing how reviewed something is follows along that stage where: if it’s a consensus change, you’re obviously going to need a community consensus, and then if it’s an RPC change then you make sure that the downstream users are aware of something happening. But if something is like, Oh let’s change the type from a signed integer to an unsigned integer — there is no behavior change happening. Or if we’re changing the variable name because this variable name is outdated because now it does more things in the code — we need to rename it. Or something like that — those kinds of refactors are more for code health or paying down technical debt than they are user-facing, and so these are all the considerations when you’re thinking like how reviewed does something need to be in order to be incorporated.
Stephan Livera: Sure. And so for listeners who are new to this world, can you explain a little bit about the terms that you mentioned earlier: upstream and downstream. Like, what does that mean? What is dependencies? Just at a high level for new listeners.
Gloria Zhao [9:18]: Yeah, definitely. Sorry for jumping ahead on that. So every software has things that it depends on. So for example: most software doesn’t come with a computer to run it on — you need a computer with some operating system. Like, Bitcoin Core, for example, needs to be able to create files, and so it’ll ask the operating system, Hey, can I create this Blocks directory so I can store blocks. And we have some other dependencies as well: we need a C library. We need a C++ compiler because the source code is written in C++. And so these are dependencies in that — like, we write the Bitcoin Core code, but then these are black boxes or libraries that we pull in and we’re like, Okay, we rely on this working. We rely on an Internet connection to make connections to other nodes, for example. And some dependencies are, like I said, like glibc, like a C library. And some are yet another GitHub repository where you can then go and see the pull requests and the issues, and you can open and pull requests and stuff. And all of these are maintained to a different level, and all of these will also cut new releases and they’ll update things and deprecate things and we have to respond to those changes. And so we are downstream users of that upstream library, or repository, and so their changes affect us. And our changes affect downstream users such as LDK or LND which require [us] working — I assume. There’s different types of dependencies, right? There’s like literal code that you’re calling. There’s like, Oh, there needs to be a Bitcoin node running so that we can send data to it, but that’s not necessarily part of the binary. So hopefully this gives a sense of the relationship of like: you need other code. I’ve had people ask questions of like, How much code is in Bitcoin Core? How many lines of code do you look after? And that’s a very very difficult question to answer because if you think about it, really: for a Bitcoin node to work you need all of this, like this whole stack to be working properly. And @fanquake often says, The easiest way to slip a vulnerability into Bitcoin Core is probably upstream because we’re definitely not watching boost libraries as closely as we’re watching our own repository. But you always have to make a sense of like, Is this maintained or is this not?
Stephan Livera: Yeah, gotcha. And so as I understand it, that has been a thing in other ecosystems. I know people talk about like NPM and there’s like 1,000 different dependencies and one little upstream error up there can end up in your project because you’re pulling this other thing. Now, as I understand, obviously Bitcoin Core is much more, let’s say, conservative or reasoned about these things and there’s deliberate effort done to reduce dependencies or remove them where that is feasible, but just explaining the concept so people have an idea about what that is. Now the other question that probably a lot of new listeners are just thinking is, Oh well hang on, Gloria — does that mean you control Bitcoin Core? Obviously I understand that’s not really how it works, but just for the sake of listeners who are new to Bitcoin: how would you answer that question? Do you control Bitcoin Core per se?
Gloria Zhao [13:14]: No — not at all. I guess we can walk through a hypothetical situation which I don’t think is realistic at all, because everything happens in public — as you said, it’s open source. What if the six of us decided to merge a hard fork in Bitcoin Core or like do something crazy. What if we just deleted everything, for example? Well then someone can just fork and make a new copy of all the code and then maybe rally behind some reputable previous Bitcoin Core contributor or some other whatever and then just continue on. And then maybe you’d have like a day of a bit of chaos and then you just have the same code continuing on. And of course like here’s another thing is: software — we don’t have the power to auto-push updates to Bitcoin Core users. That’s not the case in other software. For example, like printers get pushed auto-updates all the time, but we don’t have the power to force your computer to download and install the new release of Bitcoin Core. And so if we merge a change today, we would then a month from now create a release and then we’d post it and then all the users would decide whether or not they’re going to download it and install it. So we have very very little control over what’s actually run on the network — well, I mean, maybe there’s some like social influence or whatever based on the trust that people have in someone who’s been doing this for several years, but like very little power over Bitcoin at all. Maybe Bitcoin Core as the GitHub repository, but again — like, you can easily make a copy. That doesn’t necessarily mean [that is] what’s running on the Bitcoin network.
Stephan Livera: And there are multiple implementations as well: there’s Libbitcoin, btcd, and some others out there, right?
Gloria Zhao [15:38]: Yeah, Rust Bitcoin.
Stephan Livera: And so then could you also just touch on — so you mentioned there are six [of you]. So if you could talk a little bit about — I guess as I understand, you might be a maintainer who is a specialist in a certain area. If you could just elaborate on that?
Gloria Zhao [15:55]: Yeah, exactly. So my scope is mostly mempool and associated functionality: so mempool the data structure, mining, mempool validation, the fee estimator, and then various surrounding areas — a bit of transaction relay, maybe. And so essentially this is the area of code that I’ve become very very familiar with in the past couple years, and so since the past 6 months to a year, I’ve been reviewing every single pull request that is open to this area, and I’ve also opened various pull requests to improve things or fix things in this area. And so as far as I can understand, the other maintainers — and maybe some other Bitcoin Core contributors — feel that I know enough to be particularly asked to review and or merge things in that particular area. And @fanquake is the build system maintainer, and Marco [Falke’s] scope is test and QA, so he’s constantly testing and fuzzing and monitoring and sanitizing and running CI and fixing CI and all those things — and he opens the most bug fixes out of all of us because he’s constantly finding those. @achow101 is responsible for the wallet. @hebasto is responsible for the GUI.
Stephan Livera: So everyone has their domain where they’re considered an expert, but that doesn’t stop you from, let’s say, working in someone else’s area — it just seems like you are, let’s say, taking carriage of this particular area that you’re assigned for.
Gloria Zhao [17:46]: Yeah it just means if there’s a PR somewhere it’s like, Okay, this is the default person to tag for review and they’ll be able to tell you like, Here’s what usually needs to happen in order for something to be merged — for example.
Stephan Livera: I see. And so also could you just touch on — I think this is another point that might be subtle and difficult for listeners who are new to this world, but as I understand: the focus with open source contribution and being a Bitcoin Core contributor — yes, there’s some subjectivity, but the aim is to be scientific and objective about how you do things and have a technical approach to things as opposed to subjective opinion about it.
Gloria Zhao [18:38]: Yeah, exactly. So in engineering there’s always trade-offs. I think Twitter tends to underestimate how much back and forth discussion is appropriate when you’re discussing various trade-offs in designing something, so what looks like arguing is really just: we’re discussing what’s the best way to do this because there are multiple ways to do something. And in that case, maybe there’s a bit of subjectivity, and maybe you prefer something to be more performant whereas someone else might be extremely conservative and say, Bitcoin should not change very much whereas someone else is like, Oh, we should be trying out new features as much as possible. People will have different preferences on the various spectrums of the space that we are exploring, but yeah it’s still like you’re objectively saying, This has a runtime performance of this, and, This requires this much memory. For example: mempool is by default 300 megabytes, and we want this to be very accessible for anyone anywhere in the world to run a Bitcoin node, right? And if someone is like, Oh let’s add this new feature and it’ll require every mempool to be 600 megabytes and then someone else might be like, Well this is very much worth it, whereas someone else is like, Well this makes running a Bitcoin node less accessible. And so then you’re exploring those two objective truths of like, Where on the spectrum can we compromise? Maybe like, What ideological value do we value more? So that’s where the discussion is usually framed around.
Stephan Livera: Yeah sure. And as I understand, there’s also a concern or consideration given for people who have built up a certain use case and now let’s say some new technology has come or a new scaling improvement has come and that might undermine someone else’s use case that they’ve already got out there. And so I guess that’s also part of where there’ll be community debates back and forth about the viability of this and which should be prioritized.
Gloria Zhao [21:11]: Yeah, exactly. And so there are many spectrums on which to consider something, but one of them is risk and how badly can we screw up Bitcoin if this is merged? And often that corresponds with utility. A consensus change — it’s very very high leverage because you’re at the bottom of the stack and you’re changing things you can build L2 protocols on that do wild amazing things based on just this one little consensus change, but if you screw this up and you’ve accidentally introduced a consensus change where now we have an inflation bug or something, that’s a bit scary, right? So like we were saying, you have different levels of review and community consensus required. Yeah, risk is always the huge I think number one priority or number one thing that people are arguing about — or discussing!
Stephan Livera: Yeah sure, and I think — maybe this is where there’s also some subjectivity that enters into it, but — at least it seems that as I’ve followed things, at least the discussion I see, it seems like people don’t really want to do things that massively destroy someone’s use case, or it seems like, generally, changes have to sort of be like a win-win-win for the most part, or otherwise they just won’t make it in. Would you say that’s fair? Or would you disagree?
Gloria Zhao [22:44]: Yeah so it’d be great if something’s — I mean, that’s very easy to argue, right?
Stephan Livera: Right, but the difficult case would be let’s say someone — it’s like mostly win-win but someone somewhere has to take a slight decrease and then it becomes a bit of a debate.
Gloria Zhao [23:04]: Yeah so it depends on like what the decrease is. So I don’t know about use cases, but I think in the scenario I mentioned where it’s like, Oh now people with tiny Raspberry Pi’s won’t be able to run Bitcoin nodes on them — I think that’s a dealbreaker, although it would maybe be perfectly reasonable to someone else and especially to other cryptocurrencies where you don’t require accessibility as much. But yeah some things I think most people would consider fine, or some people would consider absolute dealbreakers, and everyone has a different opinion.
Stephan Livera: Okay, so as you’ve mentioned, a lot of the work is around maintenance, as in: fixing bugs and just keeping the wheels turning — I think this is one of those things where maybe people online, you see online discussion, and there’s people who talk about this idea of ossification. Now I’d be curious to get your view on some of that as to what it would mean if there were no big changes to come to Bitcoin, but also contrasting that with what you’re explaining, which is that, actually, a lot of the work is just maintenance anyway, so that there is maintenance still to be done.
Gloria Zhao [24:25]: Yeah so I mean, so just if we did nothing — I mean, it would just not work anymore. So maybe some concrete examples: like we had a P2P protocol change, which is maybe a big deal, of an outer v2 message, which is required in order to send IPv6 and Tor v3 addresses of Bitcoin nodes. So if we hadn’t made that change, we would not be able to continue using Tor because Tor v2 would be deprecated and then we have no way of sending Tor v3 addresses and so you can’t run a Bitcoin node on Tor anymore. So sometimes protocol changes are required to continue functioning in this very important way — being able to run on on Tor. And so that’s maybe a really easy to grasp example, but the vast majority is literally just like — bugs are found on a regular basis. There are 400 pull requests open and a large portion of them are bugs. And there’s different types of bugs, right? Some of them are like just data races where if the threads run in a particular order — they’re scheduled a certain way on your OS — then you’ll have a crash. And this is extremely rare, but you would want to fix that — any software project would want to fix a data race. Or an integer overflow bug, for example. And these are like code quality things. There’s no such thing as bug-free code — I will assure you of that. And yeah, so like if we stopped doing anything, it would just stop working. Eventually, there would be no operating system supported. Eventually, the upstream libraries would be no longer maintained — vulnerabilities would slip in, and it would just be impossible to run Bitcoin Core. And so a great deal of maintenance work is required even if we don’t want to make any changes. And then the idea of like, Oh, how much change should we have in terms of features? Everyone has an opinion on that. I would lean towards the conservative side, just thinking about how much maintenance work is required, and if you have this great big project and you only have, let’s say, 30 regular contributors looking at it, then, gradually, you won’t have enough people to be looking for the bugs and then you’ll have more and more of them and then it’ll just stop working, or you’ll be vulnerable to attacks.
Stephan Livera: Yeah, right. So I think we could summarize it then as: Bitcoin does not exist in a vacuum — it still has to interact with operating systems, Internet protocols, whether that’s Tor v2 being deprecated in favor of Tor v3 or even I2P, which was another one where people were saying, Okay, can we use this other way of interacting? So in that sense, there’s always going to be maintenance required because there is always change happening in the rest of the technology world, and so I think that’s maybe something people on Twitter or things don’t quite grasp that element of how the system works and how the sausage is made, let’s say. And so I think the other point that’s probably interesting as well is just around getting contributors into Bitcoin. Part of that is actually from that book, Working in Public: one thing that really stuck out to me from that book was this theme that it’s always easy to get a new pull request, but actually the real bottleneck is normally on having people who are capable to do review. So could you just elaborate a bit on that in the Bitcoin Core context?
Gloria Zhao [28:38]: Yeah so we have a pretty high review bar which stems from the focus on security, and it’s all about bugs, because if you catch a bug in review, that’s way easier than catching the bug after it’s been merged and interacting with other things and whatnot. And so every pull request — it depends on the pull request — requires quite a bit more review time than time to write it. And maybe that’s not the case in other projects where it’s just managers said, Write this feature, and then you wrote it and then the manager merged it and it’s just a B2B, like 50 clients software, and nobody’s going to attack it. But in Bitcoin Core, we assume a highly adversarial environment, so if there is a bug, if there is a way for someone to crash your node, that’s huge! — That is really really bad! And that’s why review time — I would say that, depending on the pull request, you need at least twice as much review as you need time to write it. And so that’s why not every pull request gets merged, and why maybe newer contributors might take some time to acclimate to this development culture, because maybe they worked at the company that I just described. And after it’s been merged, that’s also not the end of it, because it’ll interact with other pieces of software. And then if it’s code that only one person understands and then that person leaves, then suddenly we have code that nobody understands and we don’t know how, if another pull request is opened — everything touches everything — that touches this module, we’re not sure if this is going to cause a bug. And so maintenance after something has merged is also a consideration. So yeah, that book Working in Public I think talks about this pretty extensively and I really like a lot of the language that is used there in that — maybe some people don’t want to say this, but — there are people who want to get a commit into Bitcoin Core and then be like, I’m a Bitcoin Core contributor, right? And that is not always contributing — it’s not always helpful. And in fact it may take away review time from a bug fix, for example. And I really really hope that this is not discouraging anyone from contributing to Bitcoin Core, but, on the contrary, what I’m trying to say is: the best way to contribute is to review — it’s to test. And everyone can contribute to Bitcoin in their unique way. I would say that in terms of helpfulness to accessible ratio, if you’re a newer contributor, please come and review pull requests! Or if that’s difficult, come to PR Review Club. So — sorry this is a plug: I help organize a Bitcoin Core PR Review Club. Every week we pick a PR that is open to the Bitcoin Core code base. And a PR is — you can think of it as a guided tour through some particular area of the code base or some functionality. So we’ll break down the concepts, we’ll link to pieces of code or documents that help you understand this, and then we’ll go through various questions that help build a mental model of how you can be very critical at looking at each commit and looking out for bugs and looking out for potential future problems that might happen if something’s built on top of this. So we do that every week and it’s supposed to be for beginners, so everyone is welcome. Yes — please come and contribute!
Stephan Livera: Yeah, so listeners: I’ll put the link in the show notes. And it’s one of those things where I like that it happens in public because there have been times where — just out of curiosity — I’ll just click through and read some of the discussion just to try to at least have some familiarity. I don’t quite grasp all of it, but I at least have a look at it to get a rough idea of what’s going on and what the discussion is on some of the changes of the day. And speaking of changes, we’ll chat a bit about this theme as well: this whole idea of mempool, package relay, and a little bit about that now. So maybe to back up and give the background on this, let’s start with some of the basics: so there’s this concept of a mempool. So what is a mempool? Why do we need this?
Gloria Zhao [33:32]: Okay, so I think a lot of the interesting aspects of Bitcoin is when an ideological value turns into a technical problem to solve. So the ideology that we might start out with is: anyone should be able to pay anyone in the world regardless of what country they live in, their politics, what they say on the Internet — yadda yadda. And so in Bitcoin we know: thus, we cannot have some central entity that is managing who is transacting with whom, and so we have a peer-to-peer network where each node can join and they can kind of hide behind this huge network of anyone could be a miner, anyone could be someone who relays transactions or the sender or the receiver, right? And so this permissionlessness is the technological challenge, because the idea of like, Okay, anyone should be able to join the network and send a transaction also means, Okay, this guy could be a bad guy trying to take down the network. So at the easiest level to understand is: they could be trying to censor your transaction — they just drop it, or they’re trying to spy on you — they’re trying to figure out where exactly did this transaction come from? Or — what I consider the most interesting part about mempool is: maybe they’re trying to cause you to go out of memory, or they send you data and that causes you to just stall. You’re just spinning in this infinite loop — and this is why Ethereum has Gas, right? — is what we call in computer science a denial of service attack. And the mempool, concretely, is: a cache of unconfirmed transactions — each node keeps a mempool of the transactions that are floating around and waiting to be admitted to a block. But it’s also this precious precious resource that each node has and has to protect from attackers. I don’t know if I explained this very well, but this ideological value of permissionless — anyone should be able to pay anyone — translates into the technical challenge of: How do you have each individual node participate in this distributed transaction relay network in an adversarial environment and protect itself? And so the definition of mempool is: a cache of transactions and associated logic for validating transactions — deciding exactly what’s going to go in and out of this precious 300 megabytes that you allocate for your participation in the network. And it’s also very very useful in other ways as well: it helps you speed up block validation because it is a cache, it helps you do fee estimation and all those wonderful, useful functionalities of a node, it helps you broadcast transactions with unconfirmed inputs — all those things. And I consider mempool to be one of the most fascinating parts of Bitcoin — kind of underrated, to be honest, because it’s so technologically fascinating! There’s so many interesting trade-offs, and it is where it’s like, Well it needs to be permissionless, it needs to be accessible — anyone should be able to run a Bitcoin node, and that’s where you really start looking at trade-offs that are very interesting.
Stephan Livera: Yeah, right. That’s cool! So listeners: I don’t know how familiar you are — if you’re an experienced listener, I’m sure you’re well familiar: there are these mempool visualizers. So, famously, mempool.space is probably I would say the leading one today, but blockstream.info is also a memorable one or a well-known one out there, and you can visualize the mempool and you can see, Oh look, these are all the transactions out there. And as you said, Gloria, the default is 300 megabytes, so then — obviously, there’s all these different aspects to discuss, but — that default 300 megabytes one can create conditions where some transactions can “fall out” of the mempool, but really what we’re talking about is falling out of, let’s say, a default mempool. So to put that into a concrete example, we would say: some big exchange or some big actor on the network has just dumped a whole lot of transactions into “the mempool” — but really we mean the mempools of all the nodes — such that if, let’s say, I’m a pleb and I just put my 1 sat per vByte transaction through and I’ve fallen out of the mempool — what happens to me?
Gloria Zhao [38:12]: Well, it depends. So if you are just trying to send a payment and maybe you signaled RBF, you can just make a replacement transaction and you’re okay. But yeah, so mempools have limited space because we cannot reasonably expect every node to have infinite amounts of space — and that would be an attack vector as well. And part of policy — so policy is basically a set of validation rules that we apply to these unconfirmed transactions that help us guard this precious resource and decide what goes into our mempool. So each node has their own mempool, and so usually when your mempool gets full, you’ll dump the ones with lower fee rate because those are less likely to be mined and so having those in your cache is not really going to help you validate blocks because miners are going to choose the higher fee rate ones so you’re going to want to keep those so that you have that in your cache. And so when your very low fee rate transaction drops out of most people’s mempools, that’s usually okay for a regular user that uses RBF. It’s not as okay for contracting protocols: so we have these L2 contracting protocols such as the Lightning Network where two counterparties who don’t trust each other are able to — at a very highly scalable and private way — transact with each other by not broadcasting the transactions. So they’ll sign a Bitcoin transaction with the spending conditions of, Hey, if we’re still on the same page next week, we’ll have this amount of money — if you go offline, then I’m going to broadcast this on-chain and the contract terms are still fulfilled. And that’s why we might think of this as a smart contract. And so vaults might be another example, or DLCs — discrete log contracts, where you essentially have a pre-signed transaction between untrusting parties. And they’re going to sign this transaction far before they’re going to broadcast it, because the whole idea is you’re not going to broadcast this, right? You’re just going to settle this off-chain and then only if something goes wrong do you broadcast this transaction and get it confirmed in the Bitcoin chain. And that’s why you get the same security guarantees — you’re always able to settle on-chain. However, that breaks down depending on maybe what’s in the mempool, because you signed this transaction a long time ago and you put a fee rate on it and you decided, Okay, I’m gonna broadcast in the future, and maybe we put 2 sats per vByte because the mempool is completely empty, or there’s not very many transactions in peoples’ mempools right now, so it’s very reasonable to be like, Hey, 2 sats per vByte or 5 sats per vByte. But then if you go to broadcast and — just recently, right? — there was a huge dump of several days of blocks worth of I think it was 17 sats per vByte, and so you’d just be below all these things [in priority]. And typically in these contract spending conditions it’s like, Okay someone goes offline — you have 144 blocks between this time and that time to get the transaction confirmed, otherwise the other person can redeem their coins. This is very time-sensitive, but the time at which your transaction confirms depends on the fee rate, not really on how badly you want it, right? And so you need a way to get your transaction prioritized, and then the security assumption that we always rely on — which is quite minimal — is: miners are economically rational. And so we want it to be such that, on any transaction, you’d be able to add fees, and that directly translates to: it’s going to get confirmed. But if the transaction cannot be replaced and is below the mempool minimum fee rate then you’re out of options, because — well, up until quite recently — most mempools only looked at transactions one at a time, so they’d see the really low fee rate transaction that was signed a long time ago and they’d see the child and then they’d go, Bleh [do not want]! And then the really high fee rate they’ll be like, Well, I don’t have the transaction it depends on so I’m going to throw that away too. And so now we have package validation, which is: being able to look at both transactions at the same time. And then the next step is being able to relay them together, because — essentially — if the client submits it to their node and the node has that in their mempool, they still need to propagate that across a few nodes to get to the miners. And so if you’re only sending transactions one at a time — unless there’s some very clever logic in handling orphans on the other receiving node — they’ll probably still just drop it on the ground. They need to know to accept it as a package. And so that’s where we have mempool validation changes i.e the logic of being able to look at multiple transactions at a time. And then the peer-to-peer protocol change — package relay — which is getting nodes to be able to communicate with one another about like, Hey, try these together! Trust me — it’s going to be a good fee rate!
Stephan Livera: Yeah, and so part of the complexity that we’re dealing with, as you mentioned earlier, is if somebody was malicious. Like, let’s say I’m a bad guy — I maliciously craft a package that I know will cause you to spend a lot of your CPU time, and actually at the end of it you’ll realize, Oh wait, actually Stephan was bluffing — there was nothing here. That’s one of the difficulties you’re looking at.
Gloria Zhao [44:50]: Exactly. Or another really important one that has essentially made this a very very big project is a pinning attack. So like we said, one security assumption for L2 protocols is that the transactions are going to propagate. And so if there’s a way to censor those transactions from getting to the miners, then you have an attack, right? And so a pinning attack is a type of censorship attack in transaction relay where you take advantage of the policy rules that nodes have and you find a little spot where if you have this particular configuration of transactions, then either a conflicting transaction won’t replace it, or this transaction is never going to get mined because it’s going to stay at the bottom of the mempool, or it’s just never going to make it into mempools. And so there have been a lot of RBF-related pinning attacks that we’ve been talking about a lot, and so package relay includes solving a lot of these pinning attacks. Because it won’t make very much sense if, Okay, now we can talk about packages! — but, there’s this way for attackers to always be able to pin packages.
Stephan Livera: Yeah, so as an example, let’s say we’re in a Lightning context and let’s say you and I have a Lightning nodes and we’re connected with a direct channel. Let’s say I’m an asshole or I’m a malicious guy and so I guess the idea is — you correct me if I’m getting this wrong — let’s say we have a channel open, I send some sats to you over that Lightning channel, but then maybe I notice, Oh, Gloria’s offline or something and I try to broadcast an old state and try to pin the old one so that way — I’ve heard it explained as to try and outrun the timelock, that 144 blocks. So that’s the idea, right? If I’m the malicious attacker, that’s what I could potentially do in a fee pinning attack.
Gloria Zhao [47:09]: Yeah, exactly. So they can broadcast that one. Often the way to do it is to broadcast the transaction and then you’re allowed to have up to 101 kilo virtual bytes of this transaction plus all of its descendants, and let’s say you put a low fee rate on that so it’s not a fee bump, actually. But then in order to replace this, you’re going to have to replace this transaction and all of its descendants, and so that can get quite expensive. And so we would consider that a pinning attack. And part of package relay is fixing like, Okay, let’s make sure that maybe they only have to replace up to 1000 virtual bytes or whatever it is, so that there’s a little bit less variance as to what might happen when you’re trying to broadcast your transactions.
Stephan Livera: I see. And so if we could just briefly explain RBF as well? So for listeners who might not know: What’s RBF?
Gloria Zhao [48:12]: Okay, so RBF stands for replace-by-fee, and it is a mempool policy that is implemented in Bitcoin Core and other implementations where, when we see two conflicting transactions — like, they’re double spends of the same UTXO or UTXOs — instead of just throwing out anything that conflicts and picking the first one, we will actually assess like, Hey, maybe this pays a higher fee rate and thus is more incentive-compatible for me to keep in my mempool. And that has a lot of different rules around it mostly for DDoS protection, so we don’t want to be like replacing the same one over and over again — that would be pretty poor network bandwidth spam. But there’s a lot of mostly fee rate-related rules. And so users can take advantage of the fact that nodes have this implemented in their mempool policy, and so if your transaction’s taking longer than you want it to for your confirmation target, you can “add fees” by replacing it with a higher fee rate version of the exact same transaction, and it’ll RBF its way to a miner’s block template.
Stephan Livera: Yeah. And so I think depending on who you talk to, there are some people who don’t like RBF. But on the other hand, I can see there are others maybe in the technical and developer communities who are saying, No, we actually want RBF to be like a default thing. So could you just explain a little bit around that dynamic for people?
Gloria Zhao [49:51]: Okay, yeah. So one of the rules right now is: in order to RBF a transaction — to replace a transaction — the original usually will enforce that it had a signal on it to opt in to being able to be replaced. And that was — as far as I understand — a courtesy to give everyone time to handle replacements in their logic. So for example, if you’re tracking payments by TxID, you might struggle a little bit when replacements are very common, where it kind of messes up your tracking. So you give businesses a bit of time — and Optech did a lot of advocacy around this of being like, Hey, here’s how you handle unconfirmed transactions. But at the end of the day, double spends essentially were always possible — it was always possible for you to receive a transaction with a payment and then everyone else on the network to receive a completely conflicting transaction. But I think the idea was: some people were maybe under the impression that with RBF introduced, now you can double spend, or now double spends are more easy to do or more common or I don’t know. And I think this mostly stems from a misunderstanding of how transaction really works, or maybe not a very accurate risk assessment on businesses part to bake in the risk of using unconfirmed transactions. So I think there is a use case for unconfirmed transactions — nowadays you probably want to use Lightning, but if you’re going to accept unconfirmed transactions in your business, you need to bake in the risk of there being a double spend out there. So right now it’s been 8 or 9 years since RBF was first deployed and now it’s quite common — there’s quite a few. Like, thousands of replacements that happen every day, so hopefully all the businesses have caught up, and your wallet software is handling replacements. And now there are a number of pinning attacks that stem from like, I can’t replace this because it didn’t signal replaceability, right? And Antoine has posted on the mailing list quite a few explanations of these kinds of things. And so it’s very important now for L2 security that we’re able to replace things or have some kind of policy that isn’t what we have right now! But I think apparently there is still a bit of misinformation out there about what replacing means or what full RBF might mean.
Stephan Livera: Yeah, or certain businesses who maybe rely on that kind of idea, because maybe they have a business reason to accept that risk, right? So I mean I know for example companies like Bitrefill will do this kind of thing — that’s probably the well-known example — because it’s about delivering a voucher, and from their point of view they’ll say, Well, we want to be able to give the customer their thing straight away, but here’s what we’ll do: we’ll make sure that you are not signaling RBF on your transaction and you’ve put it above a certain fee rate so that we can mitigate our risk down a little bit. But I guess this is kind of the longer-term argument which I understand you and perhaps other developers are making is that: No no! We need to move to Lightning and use more scalable — and as part of that, to make those L2 and Lightning and things safe, we need RBF more as a default, as a standard. Am I summarizing that correctly?
Gloria Zhao [53:37]: Exactly. So the first time I met Sergej [Kotliar, CEO of Bitrefill] — shout out to Sergej — we had a whole argument about this and he talked about like, Bitrefill has business logic to handle risk — as a smart business does. And so yeah he’s like, Eh, maybe we’ll be fine — but it is nice that there is this opt-in thing. But yeah, like you said, there are reasons why you might want to not wait for confirmation! And so that’s valid — I think Lightning is mature enough where that should be the implementation, behind the scenes, of accepting faster payments. And I’m not here to tell other businesses how to do their business, but I’m here to say that full RBF is not necessarily more risky because it’s always possible to have a double spend. It’s always possible that other nodes have full RBF already on and they are accepting replacements without enforcing signaling. But at the end of the day, if the community is like, No, and that comes from a solid, objective, technical reason rather than a misinformation one, then yeah — then I guess we have to stick with opt-in signaling. Yeah — totally fine: it’s just my opinion. Like, it’s been 8 years and we haven’t merged full RBF so what do you — ?
Stephan Livera: Well look: personally I — obviously if I’m in-person or I’m doing small value commerce, I’ll always prefer to use Lightning if I can because, Hey! — It’s just more convenient. But I understand there are Lightning skeptics out there and people who just say, No! I just want to use my thing on-chain. So I’m just trying to reflect at least a little bit of the thought there, but I personally do believe we’re kind of — all the small value commerce is going to be done on Lightning anyway, or let’s say other scaling methods.
Gloria Zhao [55:53]: Yeah. So one proposal that I’m cooking right now is to have another way to opt-in to RBF. And I hope nobody quotes me and says like, Oh she’s trying to strong-arm full RBF, because [I’m] looking at many alternatives and would not move forward without good community support and everyone understanding each other’s systems, or each other’s constraints. So just putting that out there.
Stephan Livera: Yeah, sure sure. And so let’s go back to the broader topic around the mempool and packaging transactions together. I think what we’re talking about — as I read you, as I understand you — this is about the long-term functioning and scalability of Bitcoin, right? So, high-level: today a typical block may have something like 2,000, maybe 3,000 transactions in it. Every 10 minutes. 6 blocks an hour. But let’s say we were to move into a world where many many people are using Lightning and they’ve got their channel opens and closes and all of this — that’s where this thing would become a lot more important, as I understand it. Because then if you are in a scenario where you opened these channels 2 or 3 years ago back when the fees were low and fees are consistently high and now you’re in a scenario where, for your security, you are dependent on getting your channel closed or let’s say someone’s trying to cheat you or you need to get your justice transaction or your penalty closed transaction — that’s where this is more important. Is that how you’re seeing it?
Gloria Zhao [57:35]: Yes. Security is economics, right? So right now I don’t think it’s ever happened — these types of attacks that we’ve described. And it’s probably because there’s not much value in these Lightning channels. If you’re able to steal an HTLC output — what’s that, like 20 sats? But in the future, when you have more money at stake, that bounty is higher. And if conditions change, like you said, you can have much fuller mempools or a lot more channels open. So if there’s like 50,000 channels that you need to try to close at the same time, like this attacker does it at a huge scale — there’s not enough block space for all of those channels to be closed within a day, right? And so this is all just trying to be forward-thinking, because protocol development takes a really long time, and these kinds of security assumptions already don’t hold true today or are at risk of not holding true today. But that’s the thing to focus on, not the fact that, Oh these attacks aren’t happening, we can be complacent — No! These security assumptions need to be solid, otherwise you don’t have the same security model on Lightning as you do on L1 Bitcoin. And so this is the remaining interface layer that we need to make sure is bridged properly. So this attack is maybe more possible and more profitable in the future and we need to look ahead to that.
Stephan Livera: Right. Okay so in terms of where we’re at today with the different components — so just refresh my memory: one’s called package relay and the other is called mempool accept was it, or — ?
Gloria Zhao [59:41]: Yeah Package Mempool Accept. So the crux of package relay is having safe validation logic that is resistant to pinning attacks, is resistant to other types of censorship, that can handle arbitrary sets of transactions. Because when you’re looking at N transactions, that can be very different from looking at one transaction, and so just figuring out all those things. And then we’re fixing a lot of the RBF issues along the way, because the goal, again, is to make something that works. And so that’s Package Mempool Accept — all the mempool logic. And then Package Relay is just: What’s the most bandwidth-efficient way for two nodes to talk about packages where we’re not significantly increasing the amount of network bandwidth that we’re using on the network? So for example: we want it to be such that one node only ever downloads a transaction once. If in package relay they end up downloading the same transaction multiple times, that’s bandwidth waste. Same thing as like: each peer-to-peer connection should only talk about the announcement of a transaction once. And if Package Relay means that you’re gonna announce every transaction once and then every transaction in the package once, then that is also quite bandwidth-wasteful. And so the discussion right now is like, Okay, how do we make this such that we can discuss new information without using extra bandwidth — as win-win as possible, as we’ve discussed. And so that’s gone through a couple iterations.
Stephan Livera: I see. Yeah I think it’s fascinating stuff! And also if you could explain one other part: I was just reading in your e-mail it’s got WTxID-based relay. So what what is a WTxID? And why is it done on these spaces?
Gloria Zhao [1:01:51]: Right, yeah. So WTxID is as opposed to TxID. So pre-SegWit we had a TxID where you hash all the parts of the transaction. And post-SegWit you need a new way of hashing these transactions because the witness data can change even if the inputs and outputs and version and whatnot don’t! And so you might have two transactions that have the exact same TxID but different witnesses. And actually, one transaction could be valid and the other one could not because the witness data is the signatures and stuff, right? And so this is quite significant in peer-to-peer, so we moved to WTxID-based relay because there was a censorship possibility. So here’s how transaction relay works: I’ll announce the hash of a transaction to you first and then if you already have it then you’ll drop it, and this achieves the goal of not downloading the same transactions over and over again — that’s a waste of bandwidth. And so if you don’t have it, you’ll ask like, Hey, get data of this thing that you announced and then I’ll send you the transaction. But if we’re using TxID-based relay, I can censor a transaction by sending you the TxID of that transaction and then I send you an invalid version of that transaction and then you’re like, Oh, this is wrong — I’m gonna throw it away. And we also have a rejected filter, so for transactions that we’ve already seen — well, it’ll roll block-to-block, but — if it was invalid then we’ll throw it away and we’ll also cache that we rejected this within the last minute or so. But that’s very dangerous because if you’re only using TxIDs then you’ve just censored the actual transaction by sending the invalid one first, and so it’s very important to use WTxID-based relay, which is: it includes the witness so that it’s not ambiguous. It’s like, Okay, well you sent me the invalid one — well, this is a completely new transaction. And the one use case left for TxID-based relay is orphan handling. So an orphan transaction — from the perspective of a node — spends inputs. And an input contains a TxID and the index of the output vector — but it’s by TxID, not by WTxID. And we can’t change this because then it’s a hard fork. So when we receive a transaction and it refers to an input of a transaction that we don’t know about, we can’t very well ask for the WTxID of that transaction because we only know the TxID. So then we’ll go and ask and we’ll be like, Hey, there’s this TxID — this transaction refers to it. I don’t have it!Clearly, you must have this transaction. Can you send it over? And so you can’t really get rid of that because otherwise you would never be able to handle orphan transactions. And that can happen at any time because we’ll have unconfirmed transactions that rely on each-other and there’s no network propagation guarantees because you have multiple peers you’re downloading transactions from, and on a peer-to-peer connection it’ll always announce the parent before the child so you won’t have this backwards dependency error. But let’s say two nodes announce both to you and you request it from both and the guy who has the child is just faster — they have a faster Internet connection, so they sent you the child and now nothing really went wrong per se but you’ve received them in the wrong order so you need to be able to handle the situation where you just got them out of order. And so you can’t just delete TxID-based relay because we still have a use case for it. But, with Package Relay we can get rid of this, because now we’re able to talk about transactions as packages and we can say like, Oh, can you send me just the WTxIDs of all of the ancestors of this transaction? And then I’ll pick the ones that I don’t have and then I’ll request those from you. And so that’s one other benefit of Package Relay, is being able to handle orphans without TxID-based relay, in addition to all of the wonderful fee-bumping improvements.
Stephan Livera: Yeah. And so what about from a backwards compatibility point of view? So let’s say I’m a laggard. I don’t upgrade to the latest Bitcoin Core, but you’re on the latest stuff because you’re advanced. What happens then when my node is talking to your node? Do we just drop back based on versions and you have to speak to me in the old language because I’m a laggard and I can’t speak the new language of Package Relay?
Gloria Zhao [1:07:09]: Yeah, so we just won’t relay packages because when we’re doing our version handshake I’ll be like, Hey, I know how to relay packages! And then you’ll be like, I don’t. I’ll be like, All right. Fair enough — we won’t relay packages. That’s fine. And then just from an implementation details standpoint — you know, I think about this — and it’s like, Okay, well maybe I’ll have a zero fee transaction with a high fee parent. So we’ll have some logic so that I also won’t announce those to you because I know you’re not gonna take them. Or I’m not gonna announce the high fee child to you because I know you’re not gonna take the pair.
Stephan Livera: Because I won’t take the package. and the hope then is: over time, enough of the network upgrades over to the new system and then, Okay, everyone can speak Package Relay and everyone is in the happy path, let’s say.
Gloria Zhao [1:08:04]: Yeah. And I think once we get to the point where maybe like 10%-20% of the nodes do Package Relay, you should probably be able to reliably propagate zero fee parents with high fee children — yay!
Stephan Livera: Gotcha. And so that may also open up possibilities around what some of the scaling and Lightning and other protocols do because now they can take it to the bank now because they can say, Oh now that we’ve got Package Relay out there, now we can design our protocol in such a way that we can more reliably RBF.
Gloria Zhao [1:08:41]: Yes, exactly. So I think this would save quite a bit on fees for Lightning because they can put zero fees on their commitment transactions. Because right now, essentially, I think usually you put like at least 10 sats per vByte or higher because you always want to overestimate a little bit just in case the mempools get congested. But anytime you overestimated, you’re overpaying on fees. But with Package Relay, if they’re like, Okay we’ll just add fees when we broadcast, then you don’t ever need to overestimate — you just put zero on the commitment transaction, you have an anchor output, and then you add the fees at broadcast time! And that’s just much more [efficient]. And then, hopefully, you don’t have unused anchor output UTXOs floating around. So yeah, hopefully this is pretty good.
Stephan Livera: And yeah so that opens up possibilities there. I’m curious in terms of non-Lightning things: do you see any implications there or any benefits — wins there?
Gloria Zhao [1:09:47]: Yeah so I mean this will just make CPFP more reliable in general, and Lightning is not the only user of CPFP. Like I said, orphan handling should be better — we should have fewer orphans if we’re able to download packages at a time and not store stuff. But yeah, this is I think something that has always been talked about as an improvement to Bitcoin Core before Lightning existed — like, CPFP was quite a few years ago, and at the time they were already like, Oh, well now we should be relaying transactions as packages because we have these dependency relationships and fee rates should be assessed at a package level rather than an individual level. But it’s just: there’s so many moving parts to consider and it’s become more and more important, and so people are prioritizing it more. But maybe back in the day it’s like, Well nobody’s using CPFP, the mempool’s never congested — like, this is not a priority, right? Like I said, there’s always many bugs to fix!
Stephan Livera: Yeah of course. And I think it’s interesting because in some sense some of the stuff you’re working on, it may not be relevant for years to come, right? We don’t know exactly when, but we want to have it when it comes because then it’ll be much more comfortable to deal with it because then we’ll have less people who are, let’s say, getting cheated out of money on their Lightning channel close, and it may make the user experience a lot nicer for people as opposed to them having a janky experience where like, Okay I thought I was on Lightning but then I got yoinked because there was some exchange who dumped all these transactions in the mempool and then they just took up all the capacity and now I got — you know?
Gloria Zhao [1:11:40]: Yeah. And if you consider how long it takes to deploy something like this, you have several years of research and development and design, and then after it gets merged you need to release it, and then gradually nodes adopt it and then you have a reasonable portion of the network using it so that you’ll be able to propagate a package — like, this can take many years. And so if something’s going to be a problem in 3 years we have to fix it now — we have to be working on it now.
Stephan Livera: Yeah interesting. Okay cool! So I think those are probably the key questions I had. Was there any other key things you wanted to hit while we’re here?
Gloria Zhao [1:12:19]: Not particularly. I already got to plug PR Review Club, so I hope to see people there!
Stephan Livera: Okay, well let’s do that then! So listeners: make sure you check out Gloria’s work. I’ll put the links in the show notes — and check out the PR Review Club. Is there a website or anywhere best for people to go to follow?
Gloria Zhao: bitcoincore.reviews
Stephan Livera: Fantastic. Okay Gloria, well thank you for joining me again.
Gloria Zhao: Thank you for having me! This was fun.