Lightning Protocol
Speakers: Alex Bosworth
Date: October 22, 2018
Transcript By: Michael Folkson
Tags: Lightning
Media: https://www.youtube.com/watch?v=1R5DNUcCYRg
The Lightning Protocol, an Application Design Perspective
Slides: https://lightningresidency.com/assets/presentations/Bosworth_The_Lightning_Protocol.pdf
Intro
I’m talking about the protocol design and also going through the BOLTs. What I’m going to talk about is also about the BOLTs. The BOLTs are like the spec for Lightning and there’s a bunch of them.
BOLT 00: Build Apps
What I try to do is go through the BOLTs and look at only the parts of the BOLTs, the protocol design, that you as an app developer would need to care about. The most important thing that you as an app developer want to think about is the HTLC. Everything that you’re going to be doing when you’re receiving money, when you’re sending money, it all revolves around this HTLC concept. So that’s kind of like a payment. All the stuff about channels and the channel graph, those are all things that are going to limit you. Ideally you would never have to worry about channels, you would never have to worry about the graph, you would only be thinking about HTLCs, about committing HTLCs to other people or receiving HTLCs from other people. That’s kind of like the universe. You want to be thinking about HTLCs.
BOLT 00: HTLC Lifecycle
It’s not really talked about too much if you just look at the BOLTs but HTLCs have this lifecycle that you need to be thinking about. What happens when you’re sending a payment to somebody? The first step is that you are going to lock some money to your peer and you’re going to say “I’m locking this money to you and after a certain timeout it will come back to me but hopefully it doesn’t come back to me. Hopefully the secret of the HTLC is revealed and the payment makes its way to its destination.” So step 1 is the funds are locked to your peer. Then you get into the waiting time. Once you’ve locked it to your peer you’re kind of in a position where you’re just like “Please peer. I hope that this works because it’s going out somewhere. I gave you some hints about where to send it but it’s at arm’s length now. Now I have this timeout.” If we arrive here and we stop and the peer never gets back to us we’re in a bad situation because all we can do is wait, wait for the timelock. Hopefully that doesn’t happen. Hopefully we have a great peer and a great route that we’ve sent out on and they’re going to tell us “Hey. This payment that you tried to send, this HTLC that we tried to send across the network, it failed.” If that happens, that’s actually pretty cool because now you can try somebody else. Or maybe they tell you that the HTLC succeeded and they’re going to tell you by telling you about the secret that corresponds to the HTLC. They’re going to say “Hey look. This payment succeeded, let’s move forward, your job is done here.” Those are the two ideal cases. Offchain we resolve the failure or offchain we resolve the success. But there’s another situation that you have to be thinking about which is what happens if we are waiting, waiting, waiting and they’re not telling it to us offchain? So then we have this situation where we’re going to go onchain. Offchain isn’t the only way things can resolve. Things can still resolve onchain in a success or failure somewhere later. You need to be thinking about all these different states of the HTLC and really try to keep it so that it stay in 3 or 4, it doesn’t get to 5 or 6 and you don’t get stuck in 2 which is waiting. You want smoothness and that’s going to be the ideal case for writing your application.
BOLT 01: The Protocol
Going back to the overall protocol, the way to think about the Lightning protocol is that it’s a lot different from the chain because the Lightning protocol was designed to be modified and adapted over time. The chain is designed to be set in stone forever. There’s a lot of things if you go through the BOLTs where there’s descriptions about how upgrades will happen and there has already been upgrades that have happened. This is a changing system and the only people who really need to care about changes are you and your peers. Another thing that’s in BOLT 01 that’s kind of like an assumption that you need to be thinking about for your app is you want to be maintaining a server and for as long as you’re doing commerce you’re going to have to be online. That’s just an assumption of the whole system. At least intermittently online but ideally if you’re writing an app or a service you’re going to have to be online all the time. Another thing about the protocol is that even though Bitcoin is mentioned a lot in the protocol, people call things millisatoshis and the whole spec is filled with references to Bitcoin, actually there is no real constraint that it is on Bitcoin other than Bitcoin is the best. Including if you look at the port. In Bitcoin we have a different port for testnet and a different port for mainnet but in Lightning there is just one port even for Litecoin. The overall grand vision of this is that as an app developer you should just forget all about other chains. The only thing that really matters is your peers. In the future we’re going to have this routing network that can translate different types of payments to you and you don’t even have to care. Somebody who has their own wallet of their own coin that they love, they can send that to the routing network and they can say “I want you to translate that into a different type of coin” and then you’ll receive the coin that you have connections with with your peers.
BOLT 02: Peer Protocol
This was covered already. You have this constraint which is that in order for HTLCs to be flowing you need these channels. If you think about it, the whole graph part of things is not even strictly necessary in terms of you don’t have to actually specify which channel to go from which one to which one. In fact in lnd, when you receive a packet that says “use this channel to send to this channel”, lnd will ignore that. It will just say “Oh I have this set of channels and somebody told me that this peer would be able to figure out the next step.” It will just choose a channel at random even if you told that peer which channel to use. This is something where it’s like the channel is a necessary evil. The other tricky thing about the channels is these are going to be random people you’re not making a legal contract with that they’re going to stay around for you. The best strategy for these channels is you’re going to have a bunch of them. You’re going to have redundancy and not only redundancy as in you’ll have a couple of channels, you’ll also hopefully have channels that are spread out through the network. If you’re receiving money you want to be spreading out your channels to all the different types of people who would want to be sending you money. Another thing that is interesting about the peer protocol that is different than Bitcoin or normal chain transactions is that you have no idea ever who sent you the money. You can know which peer sent you the money but you don’t know where that originated because people are swapping secrets. I’m swapping a secret to him, he’s swapping a secret to him. This is could even take place outside of the Lightning Network. I could tell you the secret in exchange for some money and we could come up with any kind of way to swap these secrets.
BOLT 03: On-Chain
Like I said before, you want to avoid this resolution state where the HTLC goes to the chain. That’s like death, you’re going to pay a lot in fees, you’re going to disrupt your channels. You might even want to avoid taking money if it’s going to mean your channel dies. This is also makes for a semi-custodial aspect of Lightning for very small payments. The sweet spot for Lightning payments is actually not the tiniest of tiny payments. Lightning can go now to millisatoshis, like 1 thousandth of 1 satoshi. Anything from the 1 millisatoshi to the 999 millisatoshis is essentially just like a nice understanding that we have together. There’s no way to enforce that on the chain. Even if you go above to 1 satoshi where it is enforceable on the chain, what are the incentives to enforce them on the chain? Now I’m going to have to be paying some fees and I’m going to be having to destroy a channel, maybe I’m going to be locking up a bunch of funds. The rest of the funds will be also constrained by other timelocks so you wouldn’t only be thinking about should I take this penny and will this penny be locked up for a week, it could be shall I take this penny and $1500 are locked up for a week. You have to be thinking that you really don’t want to go to the chain because you’ll be subject to these timelocks. Even in the best case, just in normal operation, timelocks will be a new experience if you’re an existing Bitcoin developer. If you have an existing experience with Bitcoin, timelocks will become a part of your regular life. Some portion of your funds are just sitting because some channel just didn’t close in the right way. Some random peer decided to go offline and you closed out on them. There’ll be a portion of your wallet which is giving you presents every few days where your money is returning back to you.
Q - What length of time do you recommend for these locks?
A - Currently lnd is very conservative because we’re missing some of the tools like outsource reach watching so we use 2 weeks at the maximum if you put in $1500. We scale it down if you aren’t risking as much money. I think the lowest is like a day. It’s from a day to 2 weeks.
BOLT 04: The Onion
BOLT 4 talks about how this routing happens, this routing idea where you’re not really giving away too much information about how the payments are flowing. One thing that is interesting that a lot of app developers run into is that you would expect that you would be able to send a payment to somebody in Lightning. But you can’t unless they ask for the payment because they need to generate this invoice and they need to generate the secret of the HTLC and you need to lock your payments to that secret. There’s another way to do this payment that can work on Lightning and can still be user initiated. The onion is creating layers of encryption that you pass along the route when you’re sending to your destination. Each person opens his layer of the onion and he uses his key to decrypt it and he gets a little message for himself. It can say “the next person you’ve got to hand it to is going to be this person”. You can actually more information in those layers. It can be a little bit open-ended. One message that you could be attaching in there, you could attach a final message to the final person and say “By the way here’s the secret, here’s the preimage to the HTLC. Guess what? You’ve got some money locked corresponding to that secret and so you can go ahead and take it.” That’s work in progress for lnd. In general, that’ll be a cool feature that can hopefully drive a new wave of…. On Y’alls I’m missing this feature so if somebody getting rewards for their articles I don’t have any way to give it to them. I need to have them come back to me and say “I need to generate an invoice and I’m going to withdraw some of my rewards from my articles.” In the future you could just say “I’m going to keep pushing it out to you and as soon as you come online you’ll get some of it.” Another thing to think about with onion is that the farther along you are within the onion routes, the longer the timelocks are becoming. Each hub has this thing called the CLTV delta and so it’s giving itself time to deal with its relationship with its peer. The issue is that you can have like 20 hops long and each hop could maybe be adding 1 day. If we go back to the chain case, that could mean that if you’re the 19th hop, you could be waiting like 3 weeks for the payment to be resolved. If you go back to the idea of how it gets resolved, that doesn’t mean that it’s going to be 3 weeks at timeout, that means it’s going to be 3 weeks to know if it even succeeded. It could still succeed after that. There’s another aspect that you maybe want to randomize these numbers a bit because it kind of reveals a little bit how deep you are within the onion, based on what the timelock is at the moment. If you have a very short timelock the chances are you’re more likely that you’re early in the route rather than if you’re at the very long timeout, you’re late in the route.
Q - …
A - You can put arbitrary data in these onion packets so it is possible. You would never really know where it came from. They could say… they could put a message in there.
One cool thing about these packets is that we can put in back and forth messaging. Just like you were saying, we could actually create APIs using these onion packets that didn’t need any interface outside of the Lightning Network. You could say instead of a REST API you could give a pubkey. You could put in the first packet “give me all of your commands that you will allow me to do”. I could do something like “what’s the weather like?”. I could say “give me the weather” and I could send a packet and only you as the final hop when you’ve unwrapped your portion of the onion would see this “what is the weather?” request. You could send back a response because they wouldn’t even need to be addressable. You can send back an error response and you can overload some of the data that is coming in here to give results. You could just say “Here’s the pubkey, here’s the weather API”. These are all regular payments. For each call of the API you’d need to include some routing fees so that people would deliver your message but you could have back and forth messaging.
Q - What’s the advantage of using the onion routed messages….?
A - You’d get the advantage of using the same network really. It is like an extra feature on top of that Sphinx sent. That send where it is a unilateral send so I’m going to send you some money. If you’re already set up to do that, if you’re already set up to send people money through the network, you can attach metadata to this.
Q - If you’re actually going to propose that, I would propose that we have payloads of success as well because the only way to return data from a payment currently is by having an error out and thus not transferring any fees and not paying for the capacity you’re using. That might be a good use case to have payloads of success as well.
A - I think that’s an open issue because it might be a problem that failure is too useful on the network. We might even have to start to charge for failure regardless because if people start to find failure useful they will be spamming the network with tonnes of failures and it could be a problem.
Q - …
A - You can pad it with the rest of the hops. It doesn’t just have to be that one layer. You can say next layer, next layer, next layer, you can unlock those.
I just think in general this is not something that you can do today but once it is unlocked I think there’s a lot of cool apps that could come out of that.
BOLT 05: End States
So BOLT 5. Hopefully this is not related to you as an app developer which is what happens when the channels close. We want to avoid that, we never want to close the channels, we especially don’t want HTLCs to go onchain. I would say that there’s been too much emphasis on these breach conditions. As an app developer, I’ve been running Y’alls pretty consistently for maybe a year now, maybe a bit less than year on mainnet. I’ve never had a breach attempt, never. That’s never happened to me. I’ve been watching other people running their nodes. Breach attempts are a complicated way to attack things. There’s a lot of simpler ways to attack things, there’s a lot of other ways you can be annoying. I would say there is too much emphasis on how do I deal with breaches or how do I maintain redundancies. Not the highest priority thing. You can go ahead and build an app and don’t freak out about breaches. Especially if you do other things right like be a little cautious in your channel selection, limit the scope of an attack to make the attacker have to work more. He would have to become a good member of the routing network and then suddenly he would have to turn bad in a way that you could actually prove. I would de-emphasize the problem of breaches. The thing I would worry about is chain fees. How am I going to lose money by running an app? The thing you probably lose money is, maybe not so much now that chain fees are low but every time you get a channel incoming to you, you have this UTXO burden. Let’s say the guy wants to make a channel to Y’alls and he wants to pay for an article. Then he maybe pays for three articles and then he closes the channel. Now I have a UTXO of 3 cents. What am I going to do with that? That’s negative money to me. Every time you’re looking at your channels you should think that there’s this UTXO cost associated with them. We’re not living in a perfect world, we’ll probably never live in a perfect world where you can just always, for every single channel, be perfectly able to push out funds to the other side. That’s the more likely case where you will start to pay money or start to lose money to UTXO consolidation costs. This is something you can’t control. Some people worry that when someone opens up a channel with them and they force close that they’re having to pay money. That’s not how it works. There’s an initiator concept where if somebody opens up a channel to you and they force close, they pay for that force close. There’s the other side to that which is that if you’ve gotten any money, if you’ve credited them for something, you have this balance that you’re going to have to pay some money to get that balance. Another thing to think about for end states is how we’re going to handle backups. If you’re running in production with these services, none of the implementations except for Eclair have any backups. c-lightning and lnd do not have backup support. Eclair has limited backup support. Ideally you have a distributed database. That’s what we really would like. Every time a HTLC came in you would send it to one datacenter and another data center. Once it was secure in those two data centers then you would move your state forward. You could lose a data center and your application would be fine. The path to get there will be pretty long. The intermediate step will be something more like Eclair’s. The way that Eclair’s will work is let’s say you lose your state. What happens if you lose your state? Normally in the current mode for most people you’ll lose all your money. There’s this thing called static backups and those will be simple backups like a copy of the commitment transaction, what’s in flight at the current moment. If you just retain those, if you come back online you’re still missing a lot of data about the channel, the channel can no longer be operable, it’s not enough just to have the final set of data, you need to have a whole set of other parts of data about the channel. At least you’ll be able to close out all your channels and get your money back. There’s a part of the protocol where even if you’ve lost your states as long as you’ve retained your seed and you can connect back to the existing peers, they will remind you of what the last state of the transaction is, assuming they’re not evil and malicious people.
BOLT 06: Left Unsaid
BOLT 6 is a removed BOLT. That’s kind of like a wildcard BOLT because that used to be the IRC way to bootstrap the network just like Bitcoin used to have a IRC way to bootstrap the network. Just like Bitcoin we moved to DNS so BOLT 6 no longer exists. I just want to go over some things that are usually left unsaid instead of talking about IRC. Just to reiterate this point, a payment in Lightning is either super fast or you get into the bad case where it’s super slow, like it can take weeks. It’s not just weeks to timeout, it can be weeks to resolve at all like maybe you’re still going to send the money. That’s the worrying case as an app developer.
Q - …
A - It’s not about the mining fee, it’s about the timelocks, that’s why there’s this time.
One thing to think about also is that especially in the formative stages of Lightning, this can be a costly system, especially if you’re using a lot of money. People haven’t really put all that much money into Lightning. If you try to make a huge payment like you’re paying $300 you might be surprised that you actually start to pay much more than you’d pay onchain. That’s because capacity is limited in the system, it’s a limited resource and you’re starting to exhaust people’s channels. If you exhaust somebody’s channel they should make you pay for it. Although right now they don’t as much as they probably should. If you use my node I’ll make you pay for it. Backups, sharding and distributed stuff is a way off but that’s the ideal future. There’s a tricky thing that apps can do which is what if I’m offering a service and I want to charge a bit more but I don’t want to raise my prices, I can put some routing nodes between me and the network and have them charge fees. Then I can tell you it’s those guys charging fees not me charging the fees. That’s probably something the wallet developers would have to think about. This is something that I hope is improved in the wallet apps which is surfacing fees to users and making it clear what is going on because there will be fees especially at the high end. There’s even this timelock thing that is something to think about as an app developer for wallets. How do you show this dichotomy of timing?
BOLT 07: The Grid
BOLT 7 is the graph. The idea of the graph is from an app developer’s perspective like your ISP. You connect to the graph and the graph takes care of the job of connecting you to everybody else. There will be two types of nodes really. There will be the type of node who is doing their job like I’m doing my wallet job, I’m on the phone, I’m paying people. Or I’m doing my job accepting money as a merchant. There will be another type of node that is going to be on all the time and my job is to convince everybody else on the network that they should use me to send payments around through because I’m connecting everybody. Of course you can have some kind of overlap like a mobile phone could technically route and a merchant could be also a router but in general there’s these two roles. Eventually we want to distinguish the roles using the private flag system. We kind of want to say that all the channels that are public channels, those are ones that are signing up to be great routers and if you’re not signing up to be a great router you should never make a public channel because what’s the point of that? Why should I event tell the entire world through this gossip network the state of my public channel if I don’t really want you to use that. I’m not trying to make the case to you that I’m going to be a great router. Although in the current system I would still recommend public channels, it’s just something in the future that would be good to go to private channels. I’ve done some measurements on the overall graph and about 90% people I would say are not using the public channels correctly in this mode where it says I’m a great router. About 90% people are turning their laptops off or they’re making all sorts of stupid channels that are like $1 or they are going on and off all the time or going off for long periods of time. Ideally we start to refine the network into this kind of routing idea of network. That will also be great for mobile clients because instead of having to download a huge graph and then parse it and figure out who in this graph is doing a good job, they can download a more concentrated graph of just those who are interested in routing.
BOLT 08: Your Public Key
Everybody has a public key, that’s pretty cool. When you send to a merchant, they’re going to have a consistent identity so you’re going to be able to see that you’re sending to the same person. That can apply to users, users can have this identity. We can say we’re now going to use this node identity which is a node key but you have to be a little bit careful because sometimes people are going to use custodial nodes and then you’ll think “oh he signed it with that key” but actually it’s a custodian creating that payment request so we haven’t completely solved the decentralized identity.
BOLT 09: Features so Far
BOLT 9 is like the way we’re going to upgrade the protocol. What are some upgrades that we’ve already had? We’ve had this concept of people helping each other keep track of what is the current state. lnd was a little behind on that but that’s kind of the static backup concept and that’s a new concept. We haven’t had a global upgrade but hopefully we will iterate on this and there’s not the same restrictions that the chain has where if you change one thing we fall into a totally split network and everything is destroyed.
BOLT 10: DNS Bootstrap
Just to reiterate the point about the public nodes should be good nodes. It is interesting how you connect to the network because similar to Bitcoin, it’s written into the spec that when you connect to one of these DNS servers that tells you about the network, it is being non-judgmental. It’s just telling you something random and you should never make a judgement but actually in Bitcoin it is a little bit different because in Bitcoin your peers can be anybody. You really don’t need to give a crap about your peers other than you hope you’re not being eclipse attacked and you’re not getting totally bad peers that are going to hide stuff from you. In Lightning you’re more dependent on them, you need good peers. Otherwise they could just go offline, waste your money. The DNS thing is only one layer of how you’re going to connect to the network. You’re going to need some other way to think about who should I really be peers with and not just totally random people.
BOLT 11: Give Me Money
This final one, BOLT 11 is super cool. That’s the way you’re going to give out invoices so it’s going to be the replacement for addresses. It’s really cool because it has so many features involved in it that are useful for app development. The big downside I guess is because it has so many features, they’re these giant strings. An address you could reasonably look at it, count and read every character but an invoice is massive so it’s better for copy and pasting. You can put in the expiration time, you don’t have to just show it on the page, that can be integrated into the wallet, how long you’re giving somebody to pay. You sign your description of what you’re paying so other people can say “Oh look, he ripped me off. He signed this and I paid him but he never gave it to me.” You can put in a fallback, we don’t have to go to Lightning right away. We can say “if the payment is super huge, let’s go to chain or it’s convenient for you, you can suggest which channels be to used with the private routing hints”. It’s extensible, you can make up your own fields, you can add other things to it.
Read the Docs
That’s it. There’s also all sorts of docs on these. A lot of these are for protocol design so you can also look at Node.JS libraries that I put together where I go through all the different things you can do in lnd.
Q&A
Q - …
A - There’s two ways to attack routing. One way is to make your client super smart so your client is using machine learning and becoming a genius on how to pick a path through the network. The other way to attack routing is to make it so that the routing nodes have good incentives and it’s very easy and they’re becoming a great network. You don’t have to be smart on the client side. You can be pretty dumb and connect to any routing node that is somewhat credible and they’ll be a reasonable choice. We’ll attack it from both sides, we’ll try to make the nodes better and we’ll try to make the client better.
Q - With the example you gave, just having 3 cents in a channel. Would it be worth in an example like that to suggest that they don’t open channels with your service because you know it’s dealing with small amounts?
A - I should’ve said that because that’s the solution. The solution is don’t allow people to open small channels to you, put a limit on there. People don’t really all have the right ideas on how to use Lightning so maybe they’re opening a channel with you because they think they’re going to pay the least in fees and they’re not thinking this is a routed network and they should just be connecting broadly to the overall network. You can enforce that by setting minimums and in the future as a merchant, as an app developer you would probably say no channels. I wouldn’t take it as read that you will always be able to open a channel with whoever you’re sending to. In the future the app developer might become more in charge, he might only have private channels and he would connect to the network and the customer would connect to the network. Then it’s the routing node’s job to determine how big do I want the channels to be, how much UTXO consolidation cost do I want to have, that kind of thing. At the current time the easy fix is to use a minimum channel size.
Q - Is it possible….
A - Currently there’s a problem if you split the payment because it’s like what if one payment goes through and the other payment doesn’t go through? That’s a weird scenario right? It would also require the other side to create two invoices, that’s one way you could do it. It is possible from a peer to a peer if we have multiple channels. Let’s say I have two channels of $100 and I want to send $200. As long as I’m sending through the same peer, I’ll locking money to the same secret so they have a pretty good incentive to forward on a $200 if I’ve locked in both channels $100 and $100. lnd doesn’t currently have the logic to support that. That could happen without any magical crypto stuff. It’s just like “oh I realize I have these $200 and I could pay $200 to get it”. Then in the future we have this thing AMP that we’re working on at Lightning Labs. The receiver will have to create multiple different shards and then put together so that the payment will only succeed or fail atomically across multiple paths. It will become much easier if we have a Schnorr upgrade on the Bitcoin network. Although we have ways that it could work with ECDSA, I wouldn’t say that is easy to implement.
Q - Can you explain why is Schnorr important for AMP?
A - We have to do all these proofs that we wouldn’t have to do with Schnorr but I’m not sure I could explain the technical details.
Q - You mentioned milli-satoshi. How would they be settled?
A - Right now it’s just a trust based system. I don’t really care. You’re sending me 1/1000 of 1/100 of a cent so I’ll just give that one to you. In the future, Tadge Dryja has proposed this idea where you could actually give somebody a percentage chance. I could give you a millionth chance of one satoshi or something like that. Assuming you trust math and you’re a reasonable person, you would say “ok a chance of $1 is just as good as $1 multiplied by the chance”. So that’s a way we could solve it but currently it’s trust based.