Lightning Multi Part Payments
Speakers: Rusty Russell
Date: December 16, 2019
Transcript By: Stephan Livera
Tags: Multipath payments
Media: https://stephanlivera.com/download-episode/1685/134.mp3
podcast: https://stephanlivera.com/episode/134/
Stephan Livera: Rusty, welcome back to the show.
Rusty Russell: Great to be on again Stephan.
Stephan Livera: So Rusty, I hear there’s been a lot of development around multi-part payment and you’ve got the new c-lightning coming out. Do you want to just give us an update there?
Rusty Russell: Yeah, for sure. So on the, so there’s two sides to this. One is kind of the spec side and the development and interoperability of multi-part payment stuff. And the other side is what’s happening to c-lightning. So c-lightning is obviously my baby. I started at Blockstream. We have two other devs at Blockstream who work on it. And we have like a lot of community contributors as well. And we tend to do a first release candidate on the 10th of every second month. So we don’t define which time zones, we get a bit of slippage there. But basically two days ago we released rc1. Christian Decker was release captain for this release, so it was his call. And that unfortunately didn’t have multi-part payments in it. You know, he made the call that, nope.
Rusty Russell: We hadn’t had interoperability testing. It wasn’t finalized in the spec. And so we chose to have to like let it pass and then I read my email a couple of days ago. I woke up in the morning and T-bast, Bastian from ACINQ had actually, they’d completed their implementation and he had successfully done interoperability testing with the c-lightning implementation that was running on my test node. To make sure it all work together. You actually, discovered a couple of UX bugs too, so I filed those for him and then we got those fixed as well. But importantly that it worked together. And that’s like the final checkbox that we needed to make it final in the spec, once it’s final in the spec, it can go live on the network. So I then, basically begged Christian, we kind of slipped that in at the last minute.
Rusty Russell: And he has most of that in now. So the next c-lightning release, which will be kind of out this week we’ll actually have multi-part payments support. So perhaps we should step backwards and ask all these questions. What is multi-part payments support? You know, what’s this spec that you’re talking about, how did this kind of happen? So at the moment, in lightning, if I send you an invoice and I want you to pay me some money, you have to basically pay it in one big chunk, right? So you have to find, you know, you’re trying to pay me like, you know 10,000 satoshis you have to find a path that will allow you to pay 10,000 satoshi through every hop to get to me, right? And if you can’t do that, you can’t pay me. And it actually works pretty well for small amounts, but for bigger amounts it can be a real problem.
Rusty Russell: And in particular, in the case where you’ve got multiple channels, which is great for routability, but you don’t have all the liquidity in one channel, so there are payments you simply can’t make. It’s not that you don’t have the money, it’s that you don’t have it all in one place. Right. You know, and you can try to rebalance and stuff like that, but that kind of sucks. So, you know, it drives people towards sitting at one big channel because it’s more effective than having lots of little ones. But for robustness, it’s much better to have lots of little ones. Right? So multi-part payment was an idea proposed prior to the last November by Lightning Labs. And they had this scheme they called AMP. Cause we love lightning acronyms. Yeah, that’s fantastic. So atomic multipath payments in their scheme, basically you send all these separate parts and the recipient, the final recipient wouljd all these parts together.
Rusty Russell: They could combine the secrets that were in those to get the preimage that they needed to redeem all the parts. Right, and it was a really interesting scheme. But the problem with it from my point of view is that there’s no invoice flow. So in an invoice flow, I’m basically giving you a promise that says, Hey, if you pay this invoice and this is like a signed promise, like a contracts say, Hey, I will give you the preimage for this thing if you pay this, which means that if you can share the premed for this, this hash thing, you, it sure shows that somebody paid the invoice, right? So that gives you an ability to go, hold on, clearly this has been paid. I can prove that it has been paid. Where’s, where’s my stuff? Right? You know, you’re supposed to ship me a pizza or whatever it was.
Rusty Russell: Right? and I consider that kind of important. You know, in a decentralized network where there’s no middle man this cryptographic receipt technology is actually key. So this is something I really wanted to preserve. And Anthony Towns pointed out that actually, because we have this property, you can do a much simpler multi-part payment. You simply send all the pieces and rely on the fact that the vendor isn’t going to give you the receipt. They’re not going to cash it in until they’ve got enough. Right? I mean if you say you want me five bucks for this, well, you know, you’re not going to give me the receipt to say that I’ve paid unless I actually give you the amount of money you expect. Right? So this makes it a much simpler scheme where I just send you all the pieces with the little flag saying, by the way, I know this is not everything you wanted, but it’s coming and you wait for like up to 60 seconds to see if you get the other pieces.
Rusty Russell: If you get them all, great, you atomically accept all of them together. And I get what I want. I get the receipt. Now, this is conceptually simpler to implement. And you can basically, it turns out you can build the more advanced, the secret based AMP on top of a kind of a base AMP kind of layer. So this became base AMP and we decided to drop the puns. And in the spec it’s called multi-part payments. Because you know at some point readability is important. And having people know the names rather than just being able to, multi-part payments is kind of, you know fairly obviously. And you don’t have to like define what that is, whereas AMP you always had to kind of tell people and let them in on the secret of what it stood for.
Rusty Russell: So so what we’ve got at the moment is base AMP and that has benefits of simplicity. And it looks like at this stage there is so obviously ACINQ are testing and implementation in eclair because they have the interoperability with us, c-lightning have an implementation. And I just heard from on IRC from Conner that they also a few days ago, merged into master their implementation of base AMP. And they were doing some testing as well at the moment. So what status that will be in the people’s next releases is still up to the individual projects, but it looks like the next release of every single of the three major implementations will support base AMP. And that’s incredibly exciting. On the C-lighting side we’re really good with receiving. Like, you know, you can just transparently receive base AMP.
Rusty Russell: If it comes in multiple parts, it’ll wait for the other pieces and it’ll work. If you want to send one, you have to construct it manually, kind of out of match sticks and snot, right? There’s enough there that you could in theory create a payment. There’s enough to test but the normal pay command still doesn’t understand multi-part payments. So it’s important that we got the infrastructure out there. And the next releases of course will enhance the heuristics on when we should use multi-part payments. How we should use it. You know, what happens if part of it fails, you try to split that again and do more parts and things like that. And the most obvious case, right, is where, I’m trying to pay you, but I’m trying to pay you more than a single channel of mine will handle.
Rusty Russell: Obviously I should do multi-part payments. But since our pay command is actually a plugin, it’s actually really easy for third parties to implement that if they wanted to before we get to it.
Stephan Livera: Right.
Rusty Russell: So it is an incredibly exciting development. It’s a really good time and it’s an exciting time. Everyone’s like public champagne corks right now. Because you know, this has really come together. It’s been a long process getting the spec together. So the way the spec works is we kind of, somebody builds a proposal, we thrash it out then everyone tries to implement it. Then we kind of revisit the spec again because inevitably you come up with things go, hold on, we should have done it this way cause it’s simpler. And then there’s like, when everyone agrees that the spec is correct, we’re all good, but there’s another gateway and that is that two independent implementations have to interoperate and make sure they work together. Because we’ve had cases in the past where we’ve kind of, gone, “Oh, well it’s obvious how we do it”. And it turns out we were wrong, right? To be able imagine implemented differently in incompatible ways. So we’ve gotten stricter over time with that requirement and that requirement was met a couple of days ago. And so now basically we’re ready to merge into the spec and it’s a first class citizen. in lightning. So
Stephan Livera: Yeah, that’s pretty cool. So I guess from a user perspective, they might’ve seen it like, “Oh, I tried to route these payments and they didn’t work.” But then hopefully going forward, now that we have, well, let’s assume two releases forward, right? And MPP is now operating at all of the major lightning implementations. And then ideally the corresponding wallets that are, interacting with those lightning daemons will know how to speak the language of MPP, let’s say and be able to construct the payment in such a way that more of our payments will now go through without needing us to have one fat channel. But you might have 10 small channels and you split up that payment across all of your different 10 channels, let’s say, and route it to your desired recipient.
Rusty Russell: Yup, exactly. So it’s important that as many people as possible, like there’s two stages, right? You’ve got to support receive first before anyone could support send, right? And that’s why this relates to the priority in c-lightning was to get receive, done, right? Make sure we can receive them right. Because if everyone can receive them, then you can upgrade to send, right? If the end user doesn’t understand, the person you try to send money to who doesn’t understand multi-part payments, it doesn’t matter how clever your wallet is, you’re still stuck. So yeah, getting the receive out there first, even though, the user doesn’t see any benefit until both sides support it. Right? But this breaks the ground. It means we’ve got there. Now there’s an interesting thing here in order to do this we built on what we now call TLV style, modern style payments.
Rusty Russell: So up to this point we had a very fixed format in what you would send and that’s now called legacy. And there were a whole heap of things we wanted to do. One, of the reasons this took so long is we wanted to really tear apart that format. Use the extension field we had and basically come up with a new one, which is more generic and does this. And so in order to use multi-part payments, you need to use this modern TLV style format. And at some point we will actually make it compulsory and we’ll get rid of the legacy stuff all together. Obviously that’s like a few, probably at least 12 months away before we see an implementation do that. But we have all the steps in place to do that. One of the things that it adds is this thing that we currently call a payment secret.
Rusty Russell: Now that’s actually a bit of a misnomer, but we haven’t been able to come up with better a name, so in the spec its still called a payment secret. But this basically solves a problem that we have at the moment and that if you present me with an invoice and I try to pay it, if one of the intermediary nodes, guesses this is the end point payment. Obviously they can’t see exactly where it’s going, but they might be able to guess and so they can implement their own payment and actually pay that invoice. And if it succeeds, then they can go, great. Well, okay. Obviously I can collect the incoming money because I have the the pre image that I need. But now I’ve de-anonymized the payment, right? And this is a potential attack for anyone who’s on that payment line, right. Now we don’t know of any implementations doing this at the moment, but it’s certainly a theoretical problem. This problem gets worse if you have an amount-less invoice. So most invoices say you have to pay this amount, but it’s actually optional. They can say, just just pay me some money. Right now in this case I’m trying to pay you know, 100,000 satoshis for example, and somebody probes with a single satoshi payment halfway through. Now, if you accept any amount, cause you’re just accepting donations, for example, they’ll still get the pre image that they need, but they’ll only have paid out like one satoshi or one milli satoshi.
Stephan Livera: and they can just claim the rest.
Rusty Russell: They claim the whole amount. And I will never know that you didn’t receive the full amount that I expected you to receive right? Now in order, to work around this, we have this thing called a payment secret.
Rusty Russell: So when you send me the invoice, it contains a secret and you will then only accept payments that have that secret in them. All right. So if somebody tries to do a probe but they don’t know the secret, obviously and you’ll just pretend that you don’t know what they’re talking about. Right. So it means they can’t de anonymize the payments and it means that they can’t do this thing where they can steal funds, if there’s an un amounted payment that’s accepted in there. Of course that requires the modern format. It doesn’t work in the legacy format. There’s nowhere useful to put the secret. So that’s another reason to try to push the upgrade as fast as possible because once everyone’s upgraded, we can make this secret compulsory and we can close this kind of loophole.
Stephan Livera: Sorry, just to clarify, with the payment secret, is that exclusively for MPP payments or just any lightning payment?
Rusty Russell: It’s actually any lightning payment. So the way it’s implemented is every payment is now technically an MPP payment. Like they’re pretending to be multi-part just, there could only be one part. Right? if every MPP payment, Hey, this is the total amount that I’m trying to pay you, and obviously if this payments that you’ve received covers the total amount, well then, well yeah, it’s you’re finished. Right? So the format for whether it’s multi part or single part in the modern parlance is basically the same. And if you use a multiple payment, you have to use a secret, right? So we’re assuming that if wallets upgrade to use multi-part payments, which they want to do, they will also support the payment secrets in invoices. So that will hopefully move the ecosystem really fast towards accepting all these great things. And then we can deprecate the old legacy stuff and, the, cases where you didn’t understand the secret in the invoice. But for now, we still need to allow it.
Stephan Livera: Right. And one more question, maybe I’m not getting it quite clearly. What’s the difference then between the payment secret and then the pre image?
Rusty Russell: This is where the name payment secret is pretty bad. The payment secret is basically just some junk that you’ve put in the invoice and it proves that the person paying has seen the invoice. Right. so You’ve put this random nonce in the invoice. And then when I pay it, I present that nonce to you back again. You go cool. Okay, well you’ve seen the invoice whereas someone just in the middle, they can see the they see the payment hash cause they need to, that’s part of the contract that they’re exchanging with each peer. But, they’ve never seen the invoice so they don’t know what the payment secret is and it’s encrypted inside the onion. So the middleman can’t see what the secret is. So, in a modern world where, you’re requiring the secret because everyone’s upgraded you, you won’t accept a payment from anyone other than the person who’s seen the actual invoice.
Stephan Livera: Gotcha. So if I’ve got you correct then, like the typical model where it’s the pre image and the payment hash, the pre image is what lets the person collect the payment. But in this new model, like we’ve still got that there, but now we’re adding this additional component, which is the payment secret, which is basically proving that you knew the invoice. Is that right?
Rusty Russell: Yeah exactly. That’s right. And it does, I mean, this is a bit of a work around once we’ve got, you know, Schnorr and, you know all that through, then we can have decorrelation. We have a whole heap of other cool things that give us these properties we want, but we didn’t want to wait that long. And given that we needed to change up the format for multi-part payment, it seems sensible to kind of put this in now. And it really is trivial. Like, you look at the invoice, you get the extra field, you go great. I’ll just stash that where I’m supposed to in the modern format and send it through. And the fact that your invoice presents that field kind of implies that you understand the modern format and all those things. So.
Stephan Livera: Cool.
Rusty Russell: So, yeah, it was pretty much a free win on the way through to multi-part payments.
Stephan Livera: Oh, and Rusty, would you mind breaking down payment decorrelation? What is that?
Rusty Russell: Right. Okay. So this is a current weakness in both the legacy and the base multi-part payments. And that fundamentally the way lightning works is, I’m saying, you give me the thing that hashes to this and I’ll give you some money. And that’s actually the contract of the, smart contract on the Bitcoin blockchain, so it’s enforceable. But of course, you know, so if, if I’m trying to pay you and I’m going through Alice and Bob, I say to Alice, Hey, I’ll pay you, a hundred thousand satoshis if you give me the pre image to the thing that hashes to this, right? Alice then turns around to Bob and goes, okay, I will pay you, slightly less than that satoshis if you give me the pre image that hashes to this, right?
Rusty Russell: But it’s the same question, which means that if two people on the route are talking to each other or they’re the same person they can trivially tell this is the same payment going through. Right? You know, we go to these lengths to try to encrypt things and use onion routing, but if you’ve got, two people on the route who are cooperating with each other, they can trivially tell that this is the same payment going through. Right? Because it’s asking for the same preimage. So that’s, that’s called route correlation and basically they can trivially correlate the different hops. Now with base AMP you may be making multiple payments, but they’re all asking the same question. You’re asking for the same preimage which means that you still have correlation even though you split it into multiple parts.
Rusty Russell: If I see two of the parts, I can trivially tell this is a multipart payment of the same payment. Okay. And that’s not good. If we have Schnorr signatures then instead of using hashes and pre images, we can basically use points or keys and private keys or secret keys. And then we can actually change these and do it in such a way that you can’t correlate anymore. But in the meantime, if you use the original OG AMP proposal where you present the secrets you can also use decorrelation in that way. So the disadvantage is that you don’t get the power of invoices, but the advantage is that you can completely decorrelate the payments and that they can basically be asking for completely separate preimages because you get to make them up yourself.
Rusty Russell: So that gives you the advantage of decorrelation, which is a much more private way of making payments. Now in practice, the lightning network is probably not broad enough that, that you get a huge amount of privacy anyway, right? In order to get a high degree of privacy on a network, you need to have a lot of different options, right? So, if basically if I’m the only way people can make payments to Alice, I can pretty much tell that you’re making a payment to Alice. Right. similarly with economic concerns, I can go, hold on. You’re probably not sending a payment to this person cause you would’ve gone around me. The only reason to go through me is if you’re getting to get to Alice and things like that. So at the moment decorrelation is probably not the greatest problem with privacy on the network. But certainly, it’s a low hanging fruit. Something we’ve known since the beginning as something we want to get rid of. So, you know we’re one soft fork away. I hope for maybe a couple of soft forks away from being able to do that. And then we will get full decorrelation and in the meantime we will have other approaches to allow you to do this.
Stephan Livera: Yeah. So just again, replaying a little bit of my understanding there part of that. There is, if you are along the route, You may, Depending on how the payment goes, you might have some information about who it’s for. And also if you see, potentially you can see the, I guess if you tried to probe the network and infer what is the balance in the channels and then try to see based on the movement in that balance of the channels that because the lightning network right now is not huge, there might only be one way that you could have routed a certain payment, therefore, you know, Oh, okay, that payment must’ve come from this guy. Is that?
Rusty Russell: Yeah, exactly. You know, if the topology is limited enough, you can work it down to like the three people who are likely being paid or the other way. You go it probably must’ve come from one of these three. Right. and that’s not really good anonymity and worse. Of course, if they’re retrying payments, each payment will look the same too. So you can tell that they’re retrying. Right, they’ll still be asking the same question. Okay. Give me the preimage for this. Right. So there are ways that the network isn’t as private as we would like at the moment. There was a good Claudia gave a really good talk at.
Stephan Livera: Claudia Sanchez.
Rusty Russell: Yeah. So and she’s like, no, that’s, it’s a really hard problem to get around this. It’s actually something I’m really interested in working on in the next release as an aside is basically making our routing worse because fees are low in the lightning network.
Rusty Russell: There’s a lot of tolerance for doing really sloppy routing. And if your routing looks a lot like a random walk, it gets a lot harder to tell. Oh, well of course you’re going from here to here cause you were taking the cheapest route, right. If you’re prepared to pay an extra like fraction of a percent of your payment you can get a lot more wild with the routes you take and they can look a lot more like random noise and it gets a lot harder for you to tell. Well, of course, you can’t make assumptions like, well, of course they wouldn’t have routed through me if that were going there. Right. Suddenly it’s like, well, God knows what they’re doing right there. They’re basically doing a random walk through the network that happens to pass through the points that they want.
Rusty Russell: And that’s where we want to end up, I think. So, that’s a whole exciting area of research. But you know, once you’ve got that perfect, then you’ve still got this, this correlation problem that if someone sees the same, if you go through two adversarial nodes they can be talking to each other, or someone can be gathering the data. Well, of course that’s the same payment. They’re asking the same question. So, there’s a lot to, as excited as we are about the state of the lightning network, we inevitably look up to the horizon and go, there’s all this cool stuff coming. That will take us to the next level even beyond what we are at now.
Stephan Livera: Yeah. Right. And so I guess first question on my mind is how economical would it be for somebody to try that sort of adversarial node attack? Like, because theoretically they would have to interpose themselves across the network into lots of different routes so that they can try that decorrelation attack. Right. And we don’t necessarily know how much that would cost for them to do that. And how much of the lighting network they could theoretically decorrelate, right?
Rusty Russell: Yeah. LNBig could do it today, right. So LNBig is this group of nodes that are extremely well funded that effectively provide an ad hoc backbone to the lightning network. Right? They’re really well funded. They’re really well connected. There’s like lots of them. And so they end up, a lot of payments go through them. And they could certainly do a lot of analysis on payments if they wanted to. So, to some extent that this problem gets harder for an adversary as the network grows, particularly as we start seeing more private routes, right? So not every channel is advertised. And there’s a couple of proposals to make private channels even harder to probe for. So it’s, they’re much harder to find in the network. And those things will also make the job of doing this kind of analysis more difficult, which is definitely the aim. Right? We definitely want, this to become more difficult. But it is probably, it’s far too easy, I would say to do today. And it doesn’t necessarily almost saying, LNbig is bad or anything. I think they’re great, but you wouldn’t even have to be LNBig. It could be somebody who’s actually got access to their data, for example, if you’ve got access to logs of LNBig, you would also be able to do,
Stephan Livera: Right. And that you were also touching on one of my other questions, which is around this question of public channels versus private channels. Now currently as I understand, so your node builds its own little model of what is the network graph and obviously the public channels, it knows about those. But the private channels, it doesn’t necessarily know about those. But then as I understand, then there might be, my understanding is there’s routing hints and they can disclose what the private channels are. Can you give us some background on how to think about, public versus private channels there?
Rusty Russell: Yeah, so public channels are basically, okay. So at the moment, private channels are anything you don’t announce, we have a network to say, Hey, here’s, here’s all these channels you can use. Right. Which is kind of required for routing. So you know depending on your implementation, I mean eclair doesn’t, when it opens a channel it assumes it’s not gonna be used widely for routing. And so it does not make it public by default. Other implementations c-lighting for example makes it, makes channels public by default. And then you have to say, no, I don’t want this to be public. So there’s this gossip network. Everyone talks about their channels and says what’s happening? The private channels are not represented in that network, but if you have a private channel, you can obviously use it yourself because you know about it.
Rusty Russell: But if you want to receive a payment, you have to tell the other end somehow that this channel exists. And the way you do that is you put it in the invoice, right? You say, Hey, by the way, here’s the channel you can use to reach me. So you know, they’re private in one sense but you do disclose them to the person paying you in a lot of cases and unfortunately at the moment when you disclose them, you tell them exactly where the on chain footprint of that channel is. They learn a lot of information about it. There’s a good proposal to basically randomize those. So you’ll give it, you’ll talk to your peer, you have a private channel with you go cool, give me a temporary ID, like a fake ID for the channel and it’ll give you one.
Rusty Russell: And then you use that in your invoice and you can change that whenever you want. So that will allow another level of obfuscation. So although you’re telling people you have a private channel, you’re no longer telling them, you know, the person paying you no longer telling them the capacity or anything like that. So the other thing that happens is that a channel on the lightning network is a two of two payment to both parties. And that leaves a specific imprint on the blockchain. And you know again, the LN Conf (The Lightning Conference) Christian Decker did a talk on basically doing that analysis and he goes, look, he can much see how many private channels there are. Just looking at that and certainly when they are closed you can tell that they were a private channel. There were two of two, they looked exactly like this.
Rusty Russell: Okay. That pretty straight forward. So that’s the only thing that would get in a soft fork with taproot is that in the mutual close case you wouldn’t be able to tell difference between a normal Bitcoin spend and a lightning channel. So again, that makes private channels just that much more private and it means we’ll no longer have any idea of how many private channels there are on the network. So while private channels are currently at the moment used for things where you probably don’t want to route, in the future, it may be that you open a whole heap of private channels and you selectively disclose them to it through receipts and things like that. There is a reason to disclose private channels to people. And that is that if I have a private channel with you at the moment and I send a payment out, you can be pretty sure that I’m probably, I’m probably running eclair or something that does private channels by default.
Rusty Russell: I’m probably not routing other people’s payments. Right? I’m probably sending it myself, which is obviously a disclosure that I don’t want to make. So in some cases if you disclose private channels, it makes that heuristic a lot more muddy. Right? Then you’re not sure that a payment coming through a private channel is necessarily from me. Right? So as private channels get more private, we may also a bit counter intuitively see this case where the people are actually just selectively disclosing private channels in order to gain deniability over their use.
Stephan Livera: Okay. So then when we’re setting up our channels, does that mean we should set up some that are public and some that are private?
Rusty Russell: So your question is when we create channels, should we set up some that are public and some that are private? The answer that really is not yet when privacy technology has gotten further in the lightning network, I’d say that’s actually going to be a really interesting heuristic to do, to start just gratuitously making some otherwise useful channels.
Rusty Russell: When you have multiple channels making some of them private. There are reasons that you might want to do that, you might want to keep some channels for yourself and use others for routing most of the time. And then occasionally do that. We’re definitely gonna see more more complex heuristics around use of channels and things like that. We’re seeing this evolution at the moment, of course but using some private channels and there’s even this idea of having kind of a local gossip network. So even if you have private channels, you still might share them with a couple of neighbors and maybe they share them with one level of neighbors out just to gain this deniability. So it’s private in the sense that I’m not going to prove to you, I really have a channel, but I will tell you that maybe I can route for you if you really want.
Rusty Russell: And that means that you could maybe use my private channels every so often. And if an implementation was known to do that, you can no longer make this assumption that a payment coming through a private channel is necessarily from that person, which I think is actually a huge win for the network up with something like AMP. You also get the ability to use those even and not rely on them solely, right. Cause you can send part of the payment through like a local channel. That is perhaps private and you can mix and match a lot more. And obviously once we get payment decorrelation in there as well this becomes a much more robust system where you can spread things all over the place. And yeah, I’m definitely looking forward to, to that.
Rusty Russell: We are seeing more researchers looking hard at the, the, the properties of the lightning network and the privacy properties. And that definitely is starting this treadmill of privacy where, we look at exploits, we look at ways that people can identify nodes and try to figure out what they’re doing and then then kind of we respond to them. So yeah, I certainly have no doubt that this is going to keep us busy for the next several years at least. On top of all the technical innovations that are going on is trying to make sure that this privacy war is something that we try to stay ahead of.
Stephan Livera: So Rusty, is there a tension between routing success and privacy on the lightning network?
Rusty Russell: Okay. Is there a tension between routing success and privacy on the network? I think the more, there’s more a tension between. Okay. So in some sense, yes there is. You can, for example, if you do this random walk thing I was talking about where you know, you make your routes much less predictable. That almost certainly implies they’re longer and the longer it is, the more chance there is for failure. Right? So you know, if you’ve got twice the length of route, you’ve got twice the number of nodes that you’re hoping don’t fail. So perhaps your success rate drops to some extent. The challenge in the lightning network is to make it so reliable that the difference is between 99.99% and 99.95%. Small enough difference that nobody cares, right? Privacy is great, when nobody really has to ever disable it because it always just works.
Rusty Russell: And they can’t even measure the cost of the privacy. So I’m hoping that there isn’t this point where we have to start going, well, crap this is terrible for reliability. I need to undo things. And fundamentally, multi-part payments is about increasing reliability. So I’m hoping that gives us enough of a reliability improvement that it’s no longer a problem. We’ve seen over the, I mean, Christian Decker’s talk again, talked about the problem of stuck payments. This is, we’re getting a lot better over time in the network. We’re seeing a lot fewer stuck payments. We’re seeing a lot higher rates of success. The usual reason that a payment doesn’t succeed at the moment is a lack of capacity on one of the hops. And of course, base AMP gives us a way around that. So I expect this trend of the lightning network becoming more robust and reliable to continue.
Stephan Livera: That’s great. And also on this idea of using MPP multi-part payments, it could also help stop that attack of a person interposing themselves in the network and figuring out merely from the change in channel balances because now that movement may be only one part of the payment and not all of the payment. Right?
Rusty Russell: Yeah, exactly. So there is this attack where basically you let a payment go through, a payment’s going through, node Mallory and Mallory decides, I wonder if it’s going to go, he obviously knows the next hop, but I wonder if it’s going through this hop afterward so it does a probe to figure out what the capacity is. You know, it tries a few payments. OK cool they’ve got about this much. They let the payment through and then they probe it again. Aha. The capacity dropped by almost exactly the amount that I expected. Right now I can be pretty sure that that part of the payment did actually go through this link. That’s a very hard thing to combat. The only way to combat it really is to put a false floor in.
Rusty Russell: So you have heuristics where you pretend you have slightly less capacity than you do. Depending on, where things are being asked from and over time you revise that. But fundamentally that’s a difficult thing to to combat. It’s still possible in an MPP world because you can tell that an individual shard went that way. Potentially it’s worse actually because if you see multiple shards, then you get multiple chances to kind of do this probing and see what the result was. What we will see, I think over time is more obfuscation of exactly how much channel capacity is available. Now at the moment nobody really does this obfuscation. So you can probe down to the millisatoshi and exactly figure out now, there’s a chance that multiple payments go through at the same time and it messes your heuristic cause something unrelated caused the capacity to change.
Rusty Russell: But I expect that we will get more sophisticated and we will start, making up channel capacity changes just in order to try to defeat some of these heuristics. And the smaller the payments are the less critical it is that we represent channel balance. Exactly. So these heuristics actually go together. Interestingly, there’s a proposal that I have on the spec where we would actually do some of this work for people and say, Hey, this is approximately how much capacity I have left. Now this wasn’t useful up to this point. But with multi-part payment it was, it becomes important when a payment fails, I know just tells you I couldn’t make this payment, right? Transient, there was a transient error and then you’re like, well maybe if I split it up, it’ll work the second time.
Rusty Russell: Right? so you split it in two parts and you try again and it fails again. Do you split it again and you know, keep working your way down. That is effectively doing a capacity probe, right? Only we’re making you do a lot more work for it. So if we return a hint to you to say, Oh, by the way, you’re like three orders of magnitude out and you know, even that number could be fuzzed, you know, it could be very approximate. You have an idea, should you bother to split the payment or should you just try somewhere else as well. So we’ll probably end up with these hints because people will effectively end up doing these probes for capacity anyway. So on the one side we’ll make it a little bit easier for people to find out approximately what capacity is left. But on the other hand we will probably be increasingly active in obfuscating the capacity amounts that we return to people so that these kinds of probing attacks become more difficult. But that’s still an area of active research.
Stephan Livera: And on that it also matters how many pieces the payment gets split up into. Can you shed some light on how you’re thinking about MPP and how many pieces the payment will get split into?
Rusty Russell: Right. Okay. So this is this is a really good question. How do you actually use MPP? Like, at the moment as I said in c-lightning it’s completely manual. You’d manually divide up the pieces and you can send the pieces and it’s great for testing, but it’s completely useless, for real world. Obviously then what you want is your pay command to have heuristics to figure out when to split and how to split. In the next release, you know, the first versions will probably be quite conservative. They’ll try the full payment first and if that works great. If it doesn’t, they’ll then look at splitting it. You want to split, it’s nice to split it into uneven pieces. And one reason to actually split is that if someone doesn’t see both pieces, then you’ve obscured the amount much more, right?
Rusty Russell: People tend to ask for money in round amounts. Like, you know, give me 100,000 satoshis, or a certain amount of USD, right? And you can see that payment going through pretty easily. And if you knew how much I’ve paid, you can make some guesses. You know, paying this much on the Blockstream store, I can pretty much guess what you bought. Right? So or I can guess, “Hold on this looks like a payment for a sticker at the Blockstream store. Therefore I’m pretty sure where you’re going, cause I’m pretty sure you’re paying to the Blockstream store, right?” If you split it in two pieces and you split it arbitrarily not exactly in half, then obviously you’re denying the information to people who are in the middle who don’t see all the pieces. So there’s an argument for doing it just aggressively.
Rusty Russell: Just always use multi-part payments when you can and split it up arbitrarily. But the first implementation, we’ll probably just have it as a fallback mode. So if it doesn’t work the first time it will split into two parts and try again and split again and again. There is a limitation here in that there’s a timeout. If one piece gets accepted, the end node will hold that for at least 60 seconds and it’s specified in the specification. It has to have some time out, must be at least 60 seconds. C-lightning has chosen 70, because it’s more than 60, and it’s not ridiculous amounts. So you, at some point you have to stop trying, right? You have to kind of try to get them all through. And then you will get a specific failure back if you don’t manage it in time.
Rusty Russell: So you will have these heuristics on what you try. But at the moment we’re probably just gonna split it into two kind of random pieces, somewhere between a quarter, two to three quarters each and then, and then try to send those in parallel if the first one doesn’t work. And how far we go down the rabbit hole is a good question. Right? Do you really want to be splitting it into like a hundred separate pieces? Or do you want to give off at some point before that, you know? Okay, well this is getting silly. The other thing is that because there is a slight fee associated with each payment, splitting into more pieces means more fees, right? Because every node charges a fee. That’s a base plus percentage. The percentage doesn’t matter how many places you split into, but obviously you’re paying the base multiple times, once for each piece.
Rusty Russell: Now at the moment, those base numbers are so small, nobody really cares. And if you’re paying a really tiny payment so, that’s a large percentage of your total payment, you didn’t need to split it anyway, right? Because it’s so small it was going to go through it, right? There’s no point splitting a 500 milli-satoshi payment into parts except for experimentation purposes because you know the chances that somebody could route it if only it was less than 500 million satoshis is pretty much zero on the network today, right? If 500 milli satoshis won’t go through, then one milli satoshi go through. So there are going to be a whole heap of heuristics that come around this and a lot of this, we can only decide once you’ve actually deployed it and got some real information back. And that’s one of the areas where I’m sure the other implementations will cooperate and share a lot of data about how this is working and what heuristics work well.
Speaker 2: I think it might be useful to see the use of things like on chain fallback as well. So there was recently some chatter amongst the BTCPayServer team about whether they should present the lightning invoice and then also include a fallback Bitcoin address, wo it’s all in one. And as I understand, I think it’s BOLT-11 has a fallback, Bitcoin on chain address. So in the lightning world, would we try a single payment or a multi-part payment and then try again with another route and then if that doesn’t work, fall back to Bitcoin on-chain?
Rusty Russell: Yeah, this is, this is something that’s been in BOLT-11 since the beginning and that is the ability to have one or more fallback addresses. You can, in fact they’re specified in BOLT-11 in preference order. So you can say, well look, here’s a Bech32 address. But you know, just in case you don’t understand that, here’s the second fallback address, which is like a P2SH or something, right? So BOLT-11 can specify multiple fallback addresses and obviously it’s up to the client to go, well, hold on, I was trying to pay you 500 milli satoshis that doesn’t even make sense on chain or have some heuristic that’s okay, right? You really want to pay this. Okay, we’ll go on on chain c-lightning at the moment will not fall back to on chain without interaction. You’d have to do that manually.
Rusty Russell: But you certainly could. And again, I mean pay is a plugin so someone could write a smarter pay plugin that does those heuristics for you and says, well, okay, so it’s only gonna cost us like 1%. Sure it’s going to take longer. Cause you have to wait for confirms and things. But it’s worth sending that. The alternative is to use the Bitcoin style URL, which basically says here’s a Bitcoin address. And as a fallback, here’s the lightning payment, right? So you can kind of do it both ways. The lightning BOLT-11 payment can contain a Bitcoin address or a Bitcoin URL can contain like a lightning invoice as well. So people have implemented both ways depending on where they’re coming from. Right. If they’re coming from the Bitcoin world, they tend to add the lightning URL or sorry, the lightning invoice, into the Bitcoin URL. And if they are on the lightning side, then they use the full backup mechanism that’s built into BOLT-11.
Speaker 2: Awesome. And also we are seeing a bit more adoption around LNURL. Now you and I, we spoke about this at the lightning conference in person, but just for the listeners, can you share your thoughts on LNURL?
Rusty Russell: Oh yeah, look, LNURL is great. So and I was just listening to your podcast with the Zebedee.
Stephan Livera: Chris Moss, Mandelduck
Rusty Russell: Chris Moss. Yep. And talking about LNURL. So this is fresh. I thought. So you know, we, we spec’ed up to BOLT-11 as far as the lightning effort, we’re obviously gonna get an invoice, whatever else. But as he points out, there’s a whole piece missing there, right? Which is like, well, okay, but how do I prompt your side to send me an invoice so I can pay it? And stuff like that. And LNURL has basically found all these things that you need to get from, Hey, the lightning guys implemented this. That’s kinda cool, to I’m actually trying to use this in a real web dev kind of way.
Rusty Russell: And so LNURL this grab bag of techniques basically. It’s a format, but it also does all these cool things like, Hey, this is how you open a channel. So it’s kind of this grab bag of things that’s kind of like all the missing pieces that we haven’t done or haven’t done yet inside the main spec. So future avenues for the spec is to basically steal stuff from LNURL cool. Let’s adopt that. But LNURL by itself, is a very powerful spec that increasingly is becoming like a de facto standard that everyone should implement. Implementing it properly and c-lightning is in fact on my infinitely long to do list. Because I think it’s a great innovation. One of the things that I dislike about LNURL is that it makes web requests and you know ideally we have this nice anonymizing, a network that we were going to pay the lightning payments through.
Rusty Russell: It would be great to be able to tunnel some of these requests themselves through the lightning network. And so we’ve started discussions on basically how we would do this. So some of the stuff that’s in LNURL probably, you know, in 12 months time or whatever will start to become lightning messages themselves and that solves the problem of going, Oh, well, you know, it was all great and anonymous that you were paying this until you used LNURL and basically did a web request direct to this server. And then you, what IP address you were coming from. You know, and now in theory you could do it over tour and everything else. But in practice we found people don’t. So you know, so LNURL for me is like this. It’s both a great indication of stuff that we should look at.
Rusty Russell: Like obviously people requiring it and we need it, but also, it just shows how things have changed. And like, certainly the last 12 months we’ve seen this explosion of developers and development on top of the lightning network. I mean, those of us who are deep in the weeds in the lightning network, like, Oh, we’ve got so much work to do, we’ve got to do all this stuff. But some people one layer up already find it useful and are already, looking at stuff that we’re doing and go, Hey, here’s stuff that we can add and you know, and building these layers on top, which is incredibly exciting to see. Right. So I, you know, I love what LNURL are doing and I hope they keep going with more cool features.
Stephan Livera: Fantastic. And recently there were some comments you made around slow HTLCs. And I think yourself and Christian have been critical of that. And as I under, the main criticism is that it locks up liquidity across the network. So what’s your view on slow or long lived HTLC’s?
Rusty Russell: Yeah. So, long lived HTLC’s are a problem for the network. They’re a problem because, they can tie up liquidity and they attribute it to weaponize you can create, these long lived, HTLC’s that tie up liquidity through the network and then, at some point which could be, you know, depending on how long, what the maximum duration people allow HTLC’s to last for, which varies from node to node. You know, you’d hold it up for 24 hours and then just drop it. Right. And that attack is completely costless because if you don’t actually pay the HTLC, if it fails, you don’t pay anything. Right. So in the long run, right, the network is going to have to develop mechanisms against spam and this is one of the most obvious forms of spam this is like liquidity spam.
Rusty Russell: And the problem is you can’t tell the difference between someone using this legitimately cause they really were trying to make a payment and someone using it illegitimately because they just kind of, they kind of like a bit of a yeah. They’re a bad actor, right? And particularly in the lightning network because we try really hard not to de anonymize, right? We really try hard to make this thing where you can’t tell where stuff’s coming from. And if we don’t solve this problem and bad actors can just run free through the network, then what nodes will end up doing simply through self defense is that they will have to start deanonymising the connections, like cool, I will forward this payment for you, but just first tell me where it’s going and who it’s from, right? Which completely undermines the network.
Rusty Russell: But if it’s your choice is that, or turn your lightning node off, that is exactly what people will do. So we do need to solve this problem. And there are various proposals that are being made. But the important takeaway for people is that in the longterm there is going to be a limit on how long you will be allowed to hold an HTLC, for right now today, things are pretty wild and free on the network and that’s fantastic. You know, we’re all experimenting, that’s great. But it is important people that people realize that if you’re building a business model that is based upon the idea that you will hold people’s payment for a week before redeeming it, the network is going to defend itself at some point against that behavior and it will stop working. So I’ve tried to, make sure that, that, that people have heard this message and that look, and it’s absolutely fine to do today, but just be aware that, that will become something that people weaponize and as like collateral damage the defenses that the network introduces against that will prevent you from doing it.
Stephan Livera: Right. And it would also be from your point of view here that fees will not compensate for this.
Rusty Russell: Yeah. So we deliberately chose a model where fees are paid on success because that’s incentive aligning. I will pay you if you successfully forward this payment. The downside of that is that if you don’t, you won’t get paid at all and that becomes something of a problem because, I can spam you with a whole heap of payments that will never succeed. Right. And you will not get any money from it. I don’t gain much from it, but potentially I could though I could, if I’m competing with you for liquidity, I can basically spam away all your liquidity and drive up your CPU costs and everything else. So that people are forced to use my routes, right? So in any kind of unregulated network, you will end up with these effects.
Rusty Russell: So I see, and we’ve had this discussion several times. At some point in the future there probably will be some kind of upfront payment requirement. And we’ve had a whole lot of discussions about what form that would take. But it seems like the only real way for us to solve this is to have a two component system. You have like an upfront payment that you’ve got to pay before. I’ll even try and then you have like a post pay on success now to be incentive compatible. The post pay on success still has to dominate, right? Because I want you to try to deliver it, not just go “Nope, sorry I couldn’t thank you. I’ll take your money.” And also turns out that doing this in a way that doesn’t leak information as far as like how long the route is, is also incredibly difficult.
Rusty Russell: And I’ve had some really good conversations with Anthony Towns on different cryptographic techniques that he thinks we could possibly use to kind of square the circle and get this working. But you know, the takeaway here is that at some point in the future of the lighting network, there will probably be some form of upfront payment and, you know, it’ll be tiny. You know, it’ll just be a token amount so that if you’re spamming, at least you’ll be paying me.
Stephan Livera: Great. And are there any other things you wanted to point out with the upcoming c-lightning release 0.8.0?
Rusty Russell: Yes. So one of the things that we’ve gotten, this releases obviously, you know, c-lightning about in 2019, the story of c-lightning was really, we’re three people at Blockstream.
Rusty Russell: Three. I mean, I’ve got to say Christian and Lisa are incredibly good devs. And we’ve got, community contributors as well and that’s fantastic, but we’re still a very small team and we realized that, we couldn’t do everything. So see, lightning became this kind of foundation on which we built all these plugins. So we added these hooks so you could do these things that you needed to do. You need to implement your business logic or you want to do something custom or boutique. We didn’t have the resources to meet everyone’s needs. So we went, okay, well let’s create a plugin infrastructure so that, you can make your own customizations, you can do whatever, you know boutique things you need to do for your use case. And really, you know, it’s the latest 0.8 release that’s really come to fruition.
Rusty Russell: There were so many great hooks that you can use. If you’re doing research on different onion formats and things, you can now inject and receive arbitrary data that you want. So, that’s really been the key takeaway. But if we look back over 2019 in the lightning ecosystem, it feels to me like we’ve been building infrastructure this year. What the major things, I can point to a whole range of things. But you know, all of them are about, you know, making a solid foundation and making, you know, getting more people to understand the lightning network and that whole layer of people on top, right? You build the light network, but then you need people, app developers to integrate it and they need to learn, you know, what techniques and get that knowledge about the best methods and practices for doing things so that, so that knowledge, is being spread further and further.
Rusty Russell: So more people can come along to and go cool, I have a great idea. I want to add lightning to it. What do I do? And they don’t have to kind of, you know understand HTLC’s, read the lightning paper and work their way up. They can basically take these toolkits and these techniques other people have developed and build on that. And that has been the story for me of 2019 is that this is really about laying foundations and getting us to that point where you know, more and more people can build useful stuff on top of lightning. And that’s what makes me incredibly optimistic about the stuff that we’re going to see in 2020 because I feel like, as we close out 2019, we’ve really moved things up a notch as far as lightning goes. You know, it’s now a network that you can use for real stuff and you know, it’s more a lot more stable than it was 12 months ago as far as like the success of payments and just, ease of running your own node.
Rusty Russell: And the ease of like, your selection of wallets and everything else has really kind of stepped up a lot closer to that kind of production quality level. That I think we’d always hoped. Of course, you know, it took longer than we’d hoped as always. But I really feel that 2019 was getting our ducks in a row and getting everything organized and it feels like, we’re wrapping up with this multi-part payment stuff at the end of 2019 and 2020 is just going to be an amazing year for lightning.
Stephan Livera: Yeah. So I guess on that, what are you most excited about for 2020, for example, MPP becoming more mainstream. Anything else?
Rusty Russell: Yeah, so getting rid of legacy payments would be fantastic one, because it cleans up the spec, the space currently, I’ve got all these cases, Oh, if you want your legacy, right, we can just remove all that stuff.
Rusty Russell: And, if you want to read about what we used to do, you’ll read like the, the old git versions of stuff and the stack itself will become a much much neater document. Also in the implementations we get to pull out that stuff. It’ll be much, much nicer. From my point of view. I think the excitement, I mean, you know, there’s a whole other things coming down the pipe that we still, we still want we’ve obviously got eltoo potentially once we’ve got this soft fork in, we can do those kinds of tricks. And there’s a whole, that opens the door to a whole new realm of lightning. But increasingly I think people will be insulated from that because they’ll be one layer up. We’re seeing a lot of innovation in wallets, you know, a lot more robust deployments.
Rusty Russell: And so even though the technology might be improving underneath, it’ll be incremental for them at some point. You know, it is getting so much easier to use lightning that, while I think that will make it easier, more robust again from their point of view, they’re almost insulated from, that layer of layer of technology. So, you know I’m a, I’m a low level person. You know, I work mainly on the spec and on deep implementations and we argue over bits and bytes on the wire and core protocol stuff. And I’m still excited about what’s going to happen there. But really what we’re going to see is, innovation like both in the applications people are creating and at the wallet level, like integration, both payments and receiving payments and that, I expect to be the big story in 2020.
Stephan Livera: Actually just on the question of wallets with c-lightning, you’ve got Spark which works as a remote control for your c-lightning node. And I also understand that there’s Zeus by Evan Kaloudis. He’s also working on having that to support c-lightning. Are there any others that you’re aware of that you’re looking at?
Rusty Russell: So they’re the two main ones. You know, I really want to see in 2020, I want to see like better hardware wallets, support for lightning at least for your off chain funds. We can do that in theory today in c-lightning so that well c-lightning will control your, sorry. So c-lightning will control your your channels and, all the balance there, but it w the internal wallet won’t hold any Bitcoin, all the Bitcoin goes to and from your hardware wallet, right? That’s obviously a nice thing to have and a nice compromise between having a fully integrated hardware wallet that does all the validation of, the lightning protocol stuff and the current state where you basically transfer Bitcoin into the internal wallet inside your lightning implementation and then use it from there.
Rusty Russell: Right. So that’s definitely something that I want to see. I’m happy with the state of wallets today. Everyone, everyone’s running a wallet. That is fantastic. In the c-lightning side, I guess given my background is as Linux, kernel dev and server side, we have a wallet implementation. It’s especially functional. It’s kind of cool. But we’re more on the the server and the industrial side, right? So I am delighted to see everyone innovating with their wallets and making it so much easier to use. And, you know, the Phoenix stuff that’s just been released by ACINQ, it looks incredibly exciting. I love what, what Jack’s doing with Zap as well. I love to see all this innovation, but you know I feel our plates full just doing like the service side.
Rusty Russell: And I’m going to let them figure out all the tricks and tips on the lightning wallet side. I mean, to be fair lightning wallets are evolving so much faster than bitcoin wallets did, right. It took a long time for us to get bitcoin wallets that didn’t suck. And it seems like lightning is kind of that path has already been greased by the successive learning of bitcoin wallets. And so the lightning wallets seem to be really catching up really and, getting that user experience down really, really quickly. So that’s incredibly exciting to see. You know, and perhaps it’s a little bit like at one stage everyone used to write their own window manager for Linux. That was the thing everyone did. At the moment everyone’s writing a lightning wallet. So there’ll probably be some kind of die off and we’ll end up with like a few major ones. But the innovation that’s happening is incredibly exciting.
Stephan Livera: Yeah, it’s really fascinating to see. And like you, I’m definitely excited to see what happens in 2020 so I think that’s going to do it for this episode. But do you have any shout outs that you’d like the listeners to check out or follow?
Rusty Russell: Yeah so I did a series of Twitter like livestreams on the BOLTs which I will be continuing. I got up to BOLT nine, I need to do 10, 11. I think that’s kind of under appreciated. The stuff that underlies this learning stuff is all the implementers coming together and basically trying to spec it out and thrash out what’s gonna happen. And it gives you an insight into what’s coming and implementations cause it goes into the spec first usually, and then it gets rolled out. So I’m really excited about the spec stuff, but it’s a bit hidden from people. So there will be more of those. You can follow me on Twitter and I will definitely tweet out there. But on that, I do want to do a shout out to to the other implementers, right? So particularly particularly Lightning Labs and at ACINQ who have been incredibly generous with their time. And you know, as we engage in this mutual search for truth, right, where everyone is trying so hard to make sure that things are understandable and the best that we can possibly make them. And it’s honestly an honor to work with those devs and try to keep up with all the great ideas that they have. So it wouldn’t be fair to talk about lightning without talking about the fantastic engineers that I get to work with on this, all this stuff as well as of course, the c-lightning devs. But the, the teams altogether are both, incredibly bright and very talented, but also just genuinely nice and very friendly to each other. So I’m really enjoying the cooperation and competition, that we have together.
Stephan Livera: Thanks for joining me, Rusty.
Rusty Russell: Thank you, Stephan. Always wonderful.