Lightning 101 for Exchanges: Private Key Management
Speakers: Chris Stewart
Date: October 19, 2019
Transcript By: Michael Folkson
Tags: Lightning
Media: https://www.youtube.com/watch?v=14GKUfxfPMU&feature=youtu.be&t=1201
The Lightning Conference Day 1 (Stage 2)
https://twitter.com/kanzure/status/1212894908375322625
Slides: https://docs.google.com/presentation/d/1_-FF0U2AXuhBxEzW9J_IrYxvRi1SS2MYwJl0QeIcqbI/edit#slide=id.p
Intro
Hi everybody. My name is Chris Stewart. I’m the Founder of a company called Suredbits. We monetize APIs with Lightning but we strongly believe that the quickest way to widespread Lightning adoption is through exchanges. As other people have said at this conference most cryptocurrency economic activity does happen through exchanges right now. The logical thing for me to really start thinking about with my background is how can exchanges use Lightning safely. To take the easiest topic to break this down to, I thought private key management is something that everybody understands. Everybody understands that it is really important. Exchanges act as a custodian on the behalf of their customers usually. Maybe the only thing Bitcoiners hate more than soft monetary policy and three letter government agencies is Bitcoin exchanges losing their money. We’re going to talk about that today.
Goal
What is the goal of this talk? My goal is to convey to you guys what keys associated with Lightning can be kept in cold storage. Lightning is a very interactive protocol. That means that some keys have to be hot but not all of them have to be hot. This allows exchanges to minimize the risk associated with having all this capital on a Lightning node. Again, at a high level we just want to get as many funds and keys as possible away from our hot Lightning node.
Onchain Private Key Management
Let’s just start with the basic 101 private key management for an onchain hot wallet. Usually what exchanges do is they have their common hot wallet, cold wallet paradigm. The cold wallet stores 95% of the exchange’s funds. The hot wallet stores like 5% of the exchange’s funds. The hot wallet is used for instantaneous withdrawals. There is a lot more risk associated with operating this hot wallet because these keys are touching the internet. There’s a long industrious history of exchanges being hacked in the Bitcoin space. Exchanges are still getting hacked. An early large exchange got hacked earlier this year. This is not a solved problem by any means. It seems like the best way that we’ve kind of learned as a community to minimize this risk is just keeping keys as far away from the internet as possible. There’s some really deep rooted irony there but that’s what we’ve found to be effective as an industry.
Lightning Wallet
Let’s compare what an onchain hot wallet looks like compared to a Lightning wallet in terms of the security considerations. First off, for our Lightning wallet you need an onchain wallet to fund the channels. However, once the channels are open you can actually operate without this onchain hot wallet. Lightning has some cool flags that can be used in the protocol itself to specify cold storage keys for when a channel is shut down. You can directly put those funds back into a cold wallet if you so desire. That’s a simple way to start mitigating this risk. When channels are closed you actually close to a cold key. You need the ability to generate keys to follow the Lightning protocol. We are going to talk more about this later. There are a lot of keys associated with Lightning and we are going to go through a little marathon session illustrating what all these keys are. It is important to note, again I want to hammer this point home. Only a subset of these keys need to be hot, not all of these keys need to be hot to have normal channel operations on Lightning.
What are we going to go through
What are we going to go through? We are going to have some bullet points up here. We are going to first off say what is this key for? What is the intended operation for this key on Lightning? What purpose does it serve? Did the Lightning developers just add it because it is fun to add keys? No that’s not the case. What are the keys for? Must it be hot or can it be cold? It is a really important distinction here because again we are all about minimizing risk associated with your Lightning wallet. What can be stolen if this key is compromised? This is another thing that is a very subtle point with the Lightning protocol. There’s actually two different security considerations here, whether the person that is trying to steal money from you is your counterparty in the Lightning channel or if the person trying to steal money from you is a third party and you have a counterparty that is friendly with you. These are two different things to think about when you are operating Lightning. Depending on the key this security profile does change.
Payment lifecycle with Lightning
A very quick thing talking about the payment life cycle of Lightning. These are the messages that you need to go through to actually execute a Lightning transaction. It is really not that important. The two things I really want to point out to you guys are these Commitment_signed messages. Those are the messages that contain digital signatures in them to authorize a new Lightning transaction. That is where the signing actually happens in Lightning and these signatures are relayed back and forth. Remember this is happening fairly fast. As you can see to execute a payment, the As and Bs behind these things stand for Alice and Bob so if Alice wants to pay Bob you can see what the flow of messages is there. Again, Commitment_signed that’s the big message where all of these digital signatures live. We’re on 1 out of 6 secrets associated with Lightning.
Funding Private Key
We are going to start with the onchain funding private key. This is the output that controls the funds actually on the blockchain. It is one of the two signatures that are in the funding output. You and your counterparty are both locked into this funding output. You use this output as collateral for your Lightning channel. This controls all of the money in the channel in a very real sense because it is the collateral behind the channel. It must be hot. That is an important note about this private key, it must be hot. It has to have a digital signature every time, one of those Commitment_signed messages that I just talked about earlier is issued. Now what we’re going to do is talk about if your counterparty gets this key, the person in the channel with you, and if a third party, a random attacker on the internet gets this key what they can do with it. If your counterparty gets access to this key it is game over. If your counterparty is malicious, is not your friend, it wants to steal money from you actively, if they compromise this key they can make a digital signature and then they have the other key. They spend all the money in the output and you’re done. If a third party gets access to this key though they do need some collusion with your counterparty to steal funds. This is going to be a theme that I’m going to be talking about here. Your counterparty in the Lightning channel can actually be a very good friend to you if it is somebody you know and trust. When thinking about this from a security perspective that is a very important point to make. Again this is one example. If a third party gets access to this key and if your counterparty likes you and you know them you can mitigate risk here. If not you’re kind of screwed.
First Per Commitment Secret
This is the second secret associated with the Lightning protocol. It is the per commitment secret and this is changed every time a Lightning transaction happens. You rotate this secret so it is constantly changing every time something happens in the Lightning channel. It is used to generate the two outputs on your commitment transaction so you need it in a very real sense to be able to access these outputs in case a commitment transaction goes to the chain. Again with this key it must be hot. Every time a Lightning transaction happens you need to be changing this thing and then revoking it after a state transition happens. This something that you have to have actively at hand and make sure you can access it whenever your peer wants to make a transaction or you do. If your peer finds this secret they can steal all the money in the commitment transaction. They can derive revocation_priv_key. Another example. If you trust your peer and you screw up you can mitigate risk here because they won’t actually try to steal money from you. However if your peer is just a stranger on the internet, profit seeking, just wants to make money, this is a case where they can take money from you directly. If this is exposed before you revoke it you’re done. If a third party compromises this key only they can’t steal anything. They need collusion with your peer. They have the other secret needed to derive revocation_priv_key. Another case of thinking about who your counterparty is on Lightning and how it can save your butt if something does go bad.
Revocation Basepoint Secret
Ok we’re at 3 out of 6 guys, we’re halfway there. So revocation basepoint secret. This is the secret needed to claim your peer’s funds if they try to cheat you on the Lightning protocol. This is for the case that your peer is trying to be malicious. They want to publish an older state in the Lightning channel that gives them more money. This is the secret that you need to punish them in that case and go claim their money on the chain. This is the first instance of a key that can be put in cold storage. There is a caveat to this cold storage but it can be put there. In the Lightning protocol there is this thing called to_self_delay which is a timelock you can configure with your peer to say how long they need to wait before they can take money from an output that they put on the chain. You can adjust to_self_delay. Say if you have a cold storage key, you’ve put your revocation basepoint secret in cold storage, you may want to have a longer to_self_delay to make sure you can access this key before this timelock expires. To give you a little bit more time, be a little bit more cautious from a security perspective take advantage of this feature of Lightning. Now thinking about if your counterparty gets access to this key they can take money in their to_local output immediately. This just gives them the ability to circumvent the locktime. That means they no longer have this to_self_delay. If your peer does compromise this key you effectively have no way to punish them. That means they can go to a previous state in your Lightning channel that is most advantageous to them and take that money. If a third party gets access to this key along with the counterparty publishing a revoked state which is another qualifier if your counterparty is nice and you’re friendly with them, the third party can take funds in this to_local output but they need collusion from your counterparty again to take advantage of this vulnerability on Lightning.
Payment basepoint secret
4 out of 6, we’re getting close. This next one is the payment basepoint secret and this is the secret that allows you to claim money from the to_remote output on your peer’s commitment transaction. What that output is, your peer for some reason need to go to the blockchain because they’ve had something wonky happen. This is the private key that controls your hard earned money that your peer rightly owes you. This is another example of a key that could be cold. You could put this in cold storage, you actually have no spending conditions on this so you can think of this as a normal hot wallet key that you’d be operating in your exchange’s hot wallet. This is definitely a key that should be put in cold storage to start mitigating some of this risk associated with operating a Lightning node. If your peer gets access to this key all funds can be taken from the to_remote output as your peer also knows the per_commitment_secret. Again if you guys can’t tell I really think it is important to be really thinking about who you’re peering with on Lightning. If an attacker gets access to this they must collude with your peer to take funds from the to_remote output, namely that your peer needs to give over this per commitment secret. It is kind of a reoccurring theme here, thinking about who you are peering with.
Delayed Payment Basepoint Secret
Ok, 5 out of 6. If you can’t tell there’s a lot of keys. The delayed payment basepoint secret. This is the secret that allows you to claim money on your to_local output on your own commitment transaction. This is for the case where you’re in a wonky state personally and you’re like “I’m not sure what is going on, I just want to go to the blockchain and make sure I can claim my money.” You are encumbered by a locktime that keeps you from claiming the money right away. You need to wait until this to_self_delay timelock expires. Assuming that you haven’t published a revoked state you can then claim this money after the timelock does expire. Another example of a key that can be cold, it does not need to be around any networking stuff at all. Again that is 3 out of 6 keys that can be cold if anyone is counting. If an attacker gets access to this secret along with the per commitment secret they can claim your to_local output in the channel after the to_self_delay timelock. This is another example of needing this per commitment secret to claim money in this channel.
HTLC Basepoint Secret
This is the last one and this is also the most complicated one. This is the script that controls the HTLC in the Lightning channel. To actually transact on Lightning there’s these HTLCs flying around, each one of these HTLCs has a very unique script attached to it with some complicated spending conditions. This is the private key that controls those complicated spending conditions along with some other things. It is hard to reason about and it took us a while to work through this. This is the secret that is needed to sign for HTLCs in the case that they end up being dropped to the chain or being used in Lightning. It is a refund for the case where the payment preimage is not revealed. That’s how you end up refunding yourself in case there is a payment preimage that never comes through. Then claiming funds in the case that the payment preimage is revealed. So in both cases, the success and failure case you need to have access to this key. This must be hot, you need access to this fairly frequently, every time you make a Lightning transaction. When offering a HTLC, what it means to offer a HTLC is you’re starting a transaction. You want to begin a transaction on Lightning. When you’re doing that, when you’re beginning a transaction on Lightning an attacker can claim the funds with access to the htlc_basepoint_secret and the per_commitment_secret if the commitment transaction is onchain after the CLTV expiry thing. I’m not going to go into all of the little details there but those are the conditions that you need to be able to access money. There is another concept of receiving a HTLC. That means somebody wants to route money through you. In the case where you’re receiving a HTLC or somebody wants to route money through you the attacker can take your money if they have the htlc_basepoint_secret AND the payment preimage, so you’ve had to have found the payment preimage somewhere because that’s your job when you’re trying to route things AND access to the per_commitment_secret.
The future
So that’s a lot of stuff, there’s a lot of nuance to this, there’s a lot of new ways to lose money with Lightning and some of the old ways are still applicable if you are an exchange operating a normal hot wallet. The future, this is where I think we should be trying to go as a community. Thinking about this again from a security perspective, what would make security professionals’ jobs easier so we can see wider Lightning adoption and we can articulate this without a bunch of slides up here. As channels grow in size more stringent security procedures are going to be needed around key management. I don’t think this is a controversial point. I think everybody is in agreement here, we’d all like to see Lightning grow. I think HTLC size is going to grow, the more money being transferred over Lightning. That’s going to make people take the security story a lot more seriously. If you go talk to exchanges the very first thing they are going to ask you about is how do I protect my customers’ money and that is what they should be asking. That is a very important question and they shouldn’t be putting any products into their stack that they think is going to lose their customers’ money. Another bullet point, reducing the complexity of key management. I don’t know if you guys managed to follow me through this presentation but one thing I do want to try to get across to you guys is that it is complex. This is not straightforward stuff. I don’t think there has been a lot of literature published on this, I don’t think there’s been enough brainpower that has gone into this stuff and I’m trying to be part of the solution to at least get people to think about it and hopefully we can optimize for a solution. To credit the Lightning protocol developers they do realize this is a problem. One feature that’s coming down the pipeline is this notion of an optional static remote key. That means that one of the keys that I talked about in this presentation will no longer be dynamic, it will be just static and won’t change as transactions happen on Lightning. This is the last bullet point here, I think is very important too. We need to think about if at all possible minimizing interaction in the Lightning protocol. If I go back here, this payment lifecycle slide here. All these messages need to go back and forth for Lightning and that’s 5,6 messages here to make a single Lightning transaction which is kind of crazy. I think inherent to having interaction on any protocol means that private keys must be hot because you need to lower the latency requirements for accessing those private keys. So we as a community need to be thinking about how to minimize that interaction to make sure that security professionals feel more comfortable integrating Lightning. That’s pretty much all I’ve got for you guys. If you guys enjoyed this, I have a company called Suredbits. We monetize APIs with Lightning. We write about this kind of stuff fairly regularly on our blog posts. We have a Lightning 101 for exchanges series. Follow us on Twitter and that’s my personal Twitter if you guys get into this stuff, I try to tweet about it fairly frequently.
Q&A
Q - Great talk Chris. So the threat that you have to consider I think in all of these cases is that the peer you’ve opened a channel with attacks you right? So the private key management part is actually critical but there’s that whole other part of trying to decide which peers you open channels with. In the exchange case if you’re opening channels with other exchanges perhaps this is less of a worry because there isn’t going to be a hacker within the exchange you wouldn’t think trying to steal funds on behalf of that exchange?
A - I agree with you there. I was trying to be private key centric with this talk. I think that is probably its own separate talk. If there’s interest I can dig up details there. A key thing to note though is that people do eventually want to get their money out of exchanges. Somebody is going to have to peer with the outside world. When I talk with the exchanges, it’s like “You probably should be talking with Lightning related companies that understand this stuff better and use them as a proxy to try to minimize some of your risk.” That’s my current thinking. I don’t think I disagree with what you said, I think that’s fair.