Scaling Bitcoin Exchanges With Lightning And Lightning User Profiles
Speakers: Philip Glazman
Date: October 2, 2021
Transcript By: Stephan Livera
Media: https://www.youtube.com/watch?v=utH_obtgeLY
podcast: https://stephanlivera.com/episode/308/
Stephan Livera:
Philip, welcome to the show.
Philip Glazman:
Hi, Stephan. Really excited to be here.
Stephan Livera:
So Philip, you’re at River, and River has a lot of really cool, interesting stuff going on technologically, and I’m sure you can tell us a little bit about that. Do you want to tell us a little bit about your background and how you got into the more technical side of Bitcoin?
Philip Glazman:
Yeah, absolutely. So I’ve always been interested in Bitcoin. My rabbit hole was being libertarian, reading about Hayek, learning about Bitcoin that way. And then I went to school for Computer Science and figured that Bitcoin would be the best way to leverage that degree. It’s an intersection of two different passions of both technology and also Bitcoin. After school, I moved to Europe and I worked at Bitmain at btc.com for about a year working on their wallet team. And after that, I moved to San Francisco and met the two founders of River, Alex and Andrew, and I joined River Financial. And it’s been a blast ever since, and we’ve been growing and selling Bitcoin to new people ever since.
Stephan Livera:
Fantastic. And so one thing that I like about River is that you guys are very Lightning-heavy, if you will. That’s an interesting approach out there and I’m sure there’ll be various arguments pro and against and all of that. I’d love to hear a little bit about how and why you went about that approach of actually really taking Lightning very seriously from day one? Because that was the impression I got.
Philip Glazman:
Yeah, absolutely. So I would say River is Bitcoin-heavy. For those that don’t know, River’s a simple and secure platform to buy, sell and use Bitcoin. River.com is the brokerage, and what we offer is a white glove service for people that want to buy Bitcoin, store their Bitcoin with us, or just even use Bitcoin. And Lightning has been this opportunity to use our hyper-focus on the Bitcoin technology and offer a novel way to extend the use case of Bitcoin. What we found is that a lot of people do really like to use the Lightning Network, and I think our user experience has made it easy in order to both deposit and withdraw Lightning. And there are a lot of different ways and a lot of different reasons people use Lightning on our platform, and they’re all equally exciting.
Stephan Livera:
Awesome. So you guys have been running the Lightning node for something like two years now, is it? Can you tell us a little bit about your experiences there?
Philip Glazman:
Yeah. We soft-launched in October 2019 and we’ve been running Lightning since day one. We always knew that this was going to be part of the future—it’s always been integral to our product suite. When we launched, we had the ability to both deposit and withdraw Lightning, and our architecture has since matured a lot as has the network itself. In the beginning we were running a couple of Lightning nodes, structuring it in such a way that people can easily send and receive small amounts of money. We’ve also had a pretty hard limit with regards to how much you can send and receive in the beginning—that was quite small. But as reliability has improved throughout the whole network, we’ve been able to increase these limits. And we’ve seen a lot of growth around both how many people are using the Lightning Network on our platform, but also the amount of money that’s going through the network.
Stephan Livera:
Yeah, it really has come such a long way because I remember in 2019, you might’ve gotten more payment failures and MPP (multipath payment) was not popular then, or I think it only came in at the end of that year. So that’s just one example. Was that also similar to the experience you had in terms of being able to route larger payments—or even smaller payments?
Philip Glazman:
Yeah, it’s two-fold where both (1) the network at the time was really immature with regards to its participants and also its liquidity, and the other part was that (2) there wasn’t really enterprise tooling built out yet. It’s one thing to run a node on a Raspberry Pi and connect with different nodes as a hobbyist, and it’s completely different to run Lightning Network as an enterprise. They are completely different requirements, there’s security requirements, there is observability requirements. And when we launched there really wasn’t enough tooling built out yet, and it was onto us internally to really build a custom system, a custom set of tools that allow us to both detect failures, diagnose these failures and try to improve the situation. I think us having a very limited limit with regards to both sending and receiving in the beginning and increasing it over time, it’s really helped [the] user experience. I think one thing that really stood out to us is that we really believe in Bitcoin and we want to show that—the Lightning Network though at the time was nascent—we want to show people that this technology works and provide the best experience possible. With that said, we were always really, really upset whenever a payment failure happened. And we always rushed to build tools to make sure that we can improve the situation. Those payment failures are really sparse nowadays. I only really see payment failure if it’s reaching a node at the edge of the graph and it’s some private destination that doesn’t have inbound liquidity. And once we notice something like that, we’ll speak with the client and help them navigate the space. Client education has always been a really big part of introducing Lightning to our clients. But yeah it’s gone a long way. It’s pretty remarkable what two years can do to a technology. If you look at the graph of public channels and the amount of Bitcoin locked in it, it’s just like a hockey stick. And these payment failures have all for the most part been resolved. And the type of large payments we see today, it’s just unthinkable back two years ago.
Stephan Livera:
Yeah. And so obviously without doxxing and stuff, do you have any rough numbers you could share or anything that you’re seeing, like do you regularly see $1,000 payments, $5,000 payments? Or what kind of numbers do you see?
Philip Glazman:
It really depends on the type of user. I recently wrote a blog post at our River Engineering blog about scaling Lightning Network at River. And in the blog I identified three different types of users: you have the consumers that use Lightning as a payments platform, you have traders that use Lightning as a fast settlement platform, and then you have the hobbyists that are really quick to jump on new trends, use us as a swap service, use us as a wallet as an entry point into the network. And all these different types of users have varying levels of amounts that they send and receive. It honestly varies to the people that try it out for the first time and they just want to send 2 sats and see what it feels like to have an instant payment network, and then it’s people that actually are moving large sizes—tens of thousands of dollars—without any hiccup at all. And it’s really amazing to see.
Stephan Livera:
Yeah. It’s interesting to see the spelling out there of the different categories of customer. So I can imagine someone who’s totally new, just a retail customer who wants to buy some Bitcoin, and now they just want to play around with paying with Lightning and they just want to buy something on Bitrefill or Blockstream store or something. What’s the education journey like for that customer? Because I presume some of them don’t really understand Bitcoin on-chain versus Lightning. What’s the education process like there?
Philip Glazman:
So the Bitcoin team at River works really closely with Client Services at River, and it’s been a journey to educate clients on what exactly Lightning Network is and how it’s different from on-chain. But even stepping back from there, a lot of our users were first-time buyers, right? And so it’s their first time into Lightning, but it’s also the first time into using Bitcoin. And they have to understand the nuances of addresses, how self-custody works, what are blocks, what are payment confirmations. And then once they’re finished learning the basics of the Bitcoin base layer, then they have to learn about Lightning Network and what are nodes, and what’s a pre-image, and what’s an invoice, and why is there an expiry, and why did the route fail? And there are all these different questions that people often ask when they start using these things. Since day one, the thesis we had at River was both the base layer and Lightning Network—it’s just a payments rail. And really at the end of the day, the person that is withdrawing doesn’t so much care about which layer they’re using as much as: did the payment go to the recipient? Was it the right amount? Is there some sort of proof that it went through? And in our user experience, we just present a simple input form that lets people paste either an address, any sort of address, Bech 32, whatever, or a Bolt 11 invoice. We handle the parsing and provide the final information before they send the payment. We don’t bifurcate between a Lightning Bitcoin or on-chain Bitcoin. It’s just Bitcoin as a balance in the user account. So the user doesn’t really have to worry too much about how much Lightning Bitcoin I have or on-chain. It’s just Bitcoin. And the benefit of that is that if you just use Bitcoin on-chain, you don’t have to worry about Lightning and you can just continue to use Bitcoin as you usually do. But if you’re a power user or [it’s your] first time using the Lightning Network, it’s like this hidden unlock and you’ve discovered a way to send Bitcoin very instantly, fast, and cheap. And what we’ve done to improve the education around this is two things: one is we have River Learn, which is a public education resource that people can use to learn more about the technical and economic narratives around Bitcoin as a whole, and then we also have a knowledge base that lets people poke around and ask questions and go through step-by-step on what it’s like to withdraw. But I think what really helped us in helping people understand the Lightning Network is our Client Services team. They’re always on call. We pride ourselves in having a phone number that’s reachable. And so a client could always call and ask questions about: What is the Lighting Network? How do I use it? And this kind of white glove service has helped us develop a really close relationship to the client, but also really understand their questions.
Stephan Livera:
Yeah. So it’s a mix, it’s a combo of things. It’s the user experience of how you use River in terms of scanning a Bitcoin address or a Lightning invoice, the River Learn, and then the customer service team and that aspect of hands-on service. In this case, for the customers who are using Lightning, you’ve got to pass that Bitcoin address or Lightning invoice and then from that, make the payment for them. At that point, as part of those things, in terms of making the customer have a nice experience, you’re doing things like probing the network to check the route and see, Okay, what are the fees? And to see, Okay, what’s my success probability in trying to attempt this payment and things like that, right?
Philip Glazman:
Yeah, that’s right. So under the hood when somebody enters a Bolt 11 invoice, what we do is we probe the network—for two reasons: one is we want to make sure that we could at least attempt to pay this destination. The other part is that like on-chain, there are fees in Lightning Network. There is a base fee for routing a payment that each routing node advertises. And then there is a proportional fee that’s related to the amount of money that one is sending on the Lightning Network. And us probing the network allows people to have no surprises and [be] able to see an explicit fee before they send. So on that final confirmation page, they’re able to gauge, Is this payment worthy of sending for this amount of sats? And yeah, there [are] also these very interesting edge cases. Why charge a fee, even though the fees are so low, right? Well, there are all these different [and] interesting attacks somebody could do. There [are] fee siphoning attacks that are happening now—I saw a recent post on Reddit about this, somebody siphoning money from exchanges. Generally speaking, if you’re sending a payment, you have to pay some sort of fee and you have to make sure that the person that’s sending the payment is the one paying the fee or else you have some sort of DDoS or fee-siphoning attack. And what I mean by a fee-siphoning attack is that a malicious user could set up a route where one of the routing nodes is along the route and they can charge a very high fee and take that fee and pocket it for themselves. And this isn’t really a genuine fee for routing payments, but it’s more of a way to scalp both clients as well as exchanges from their money.
Stephan Livera:
Yeah. So essentially they would set up their own little route and make sure that they’re the only way to get to this destination and set up a super high fee and basically just stuff fees out of the exchanges that way. So that is a very interesting idea because then as an exchange technical team, how would you even stop that? Because, here’s the thing: you don’t know whether it’s legitimate or whether it’s actually this guy trying to siphon the fees?
Philip Glazman:
Yeah. For the first part, by showing the fee itself to the user is a good step as a sanity check that a human can take and see that, Okay, is this fee rational or not? And they can make that decision. On the other hand, we’re the ones building the routes at the end of the day, right? Because Lightning Network is source-based routing, we’re the ones constructing the routes and we decide the way that the payment is going to flow through the network. And the first step that’s more naive to minimize this sort of attack is to have a well-connected routing node. So be very central, have large channels with different nodes, and make sure that you’re not limiting yourself to one section of the graph through this one person that might be malicious. The other angle of it that’s a little bit more complex is: how do you identify this? And over time, if you’re able to identify a node that is leveraging these high fees, you just have to essentially build a route around them, or just notice that there is this sort of experience that a client is having and suggest some sort of remedy. Because at the end of the day, they’re paying. They might be the one doing the attack, but more likely it’s that they’re the ones that are being scammed from a high fee. And so it’s just a matter of working with them and figuring out, Okay, you have Bitcoin, how do you get this Bitcoin to your recipient, and how do you do it in a very cheap and honest way?
Stephan Livera:
Yeah. Clever. If that attacker was going to get really clever about it, maybe they would set up their node in such a way that it only has one channel from their own person and rejects channels from anybody else, that you can’t even route another way. Anyway, that’s an interesting aside, but I’d also love to chat a little bit and you mentioned the profiles as well, the user profile. So you might have the consumer profile, and as an example, obviously, as [are] many of us in Bitcoin, we’re stackers, we are accumulators. So I’ve wondered: does it make sense for a retail consumer to stack on Lightning or is it actually better for them to try to batch up their purchases and do one withdraw on-chain? What are your thoughts on that? And do you have any tips for people out there?
Philip Glazman:
Yeah. Well, Lightning is unfairly cheap compared to on-chain. Well, right now while we’re talking, the on-chain fees are 1 sat per vByte so it’s a pretty good day to send money around. But it’s not always like that, right? You have days where you can have $30 fees and over time, Bitcoin on-chain is going to become more expensive to accommodate for a security budget, and there’s going to be a sustained fee market. Reasonably speaking, it’s going to be a lot harder to send small amounts of payments on the base chain. So Lightning Network comes in as the scaling solution and a way to send money instantaneously for really cheap fees. What I see people do is that people go and earn sats a lot of different ways through Lightning Network. There’s a lot of different services out there that let you accumulate sats either through gaming or purchases, or—the list goes on. I think in the past year there’s been an explosion of different products like spend on your cash and earn sats back—which is pretty amazing. And sometimes these values that the person has on a platform are really small, right? And so if they were just limited to on-chain, they would be stuck, right? That value would be stuck. It’s almost like buying a Starbucks gift card, buying a single coffee, and having that 50 cents stuck there forever, and no way to take that 50 cents out. And all these little chunks of change the different platforms have and Web 2.0 builds out, and there’s really no way to take that value out. With Lightning, it’s different because if a platform supports Lightning, you’re able to take your small chunks of change out and accumulate it, move it over to a platform like River and build up your balance that way. And over time, perhaps you can use the on-chain layer and send a big UTXO out, but just the ability to move small amounts of money is really empowering, and it gives you this option to exit at all times from these platforms. So stackers that are earning Bitcoin everywhere on these platforms on Lightning, they always have this ability to exit, which is really empowering.
Stephan Livera:
Yeah. And look, I broadly with you there, but I think the one point I would challenge is just this idea that let’s say you are an accumulator and you’re not spending down. If you’re trying to receive over the Lightning Network, then you might just run into problems of your channel resizing. So you would then end up having to pay a channel fee opens every time you’re stacking, especially if you’re talking about small amounts unless somebody has already opened a big channel to you, because then you’re in that weird spot of needing more inbound liquidity and maybe having to open another whole channel, right?
Philip Glazman:
Yeah. This is a really interesting point and I think that’s where this dichotomy of custodial Lightning and non-custodial Lightning comes in. I mean, we are in a privileged situation where we try to earn our client’s trust, having best-in-class security, and making sure that we can operate custodial Lightning. But in non-custodial Lightning—and what I mean by that is people that are running their own nodes or have their own wallet on their phone and they use a proxy service in front of it—the situation is completely different, right? Because then you have to maybe worry about channel management, you have to worry about inbound liquidity, and it might not make economic sense to withdraw at all times. Back in 2019 I think, the non-custodial Lightning was falling behind in some ways. It wasn’t as easy. There was a lot of payment failures. And I think there was a lot of frustration around that. I think today it’s a completely different story. You have a really, really good set of wallets out there that are non-custodial and allow you to receive Lightning using like turbo channels, on the fly channels, the ability to receive Lightning payments without really worrying about the on-chain footprint and just paying perhaps a small fee in order to receive that payment and letting the proxy service handle the routing for you. So with that said, it really depends where your money is going. I think that because there is this opportunity to [essentially] become your own payments hub, it’s really empowering, right? So the user is able to do it if they want. Sometimes people will opt for the lazy way—and that’s fine—and send money to custodial services. And I think that’s completely fine. And likely more than not, most Lightning Bitcoin will be in custodial services, just like we see most on-chain Bitcoin being in custodial services. That doesn’t mean that Bitcoin is trending towards centralization, it just means that over time businesses have gotten better at earning people’s trust, building better security practices, and at the end of the day the clients can always withdraw. And now they have two options: they have Lightning and Bitcoin. So I think we’re getting into a better spot here.
Stephan Livera:
Yeah, of course. And what I’m referring to is more just like: in certain circumstances, maybe at very small amounts, that customer might end up getting rinsed in terms of having to—even as an example, if they were using Phoenix and Phoenix automatically opens the channel. But every time, if they’re just stacking, they don’t have any inbound capacity so guess what, they’re going to make a new channel—you’re paying another on-chain fee. And so in that way, people who are trying to “stack on Lightning” might end up just paying a lot of on-chain fees if they’re not clever about how they do it. Now, on the other hand, if you are Lightning-native and you earn Lightning and you spend Lightning, well then it sort of balances out because then you would actually have inbound capacity and so on. So that’s just an interesting example that I think is worth thinking about and talking about. And I’m curious, actually, if that happens for you as an exchange? Because you might have a lot more people let’s say withdrawing Lightning or depositing Lightning, and then does that cause issues for you on [the] channel balancing side?
Philip Glazman:
Yeah. And this goes back to our point of the different types of clients that are using Lightning. There’s going to be clients that only withdraw, and it’s going to push liquidity the other direction. There’s going to be people that—like traders, for example, will probably have [a] more balanced liquidity use case where they’re sending to an exchange, they’re going to do a trade, they’re going to settle, and they’re going to bring that money back. Then there’s hobbyists and it’s really a mixed bag—you don’t really know. The thing for us that has helped us a lot is observability. So just really monitoring our capacity with our peers, our failure rates, which direction is liquidity going. And it’s never this constant thing where, Okay, this peer has been the best for the past three months. It’s always changing, and there new exchanges and new platforms that pop up and they become the more central routing node. And a lot of money is shifting in one direction, and it’s something that we always have to monitor and stay on top of. So for us, observability has been critical in really understanding the direction that money is moving. So it does happen that we’ll have an imbalanced channel from time to time. Sometimes that’s intentional, because if we know that this peer is always going to be sending payments to us? Sure, we want to throw all our sats on their side in order to receive payments. Sometimes we know that this peer is going to be receiving money from us, so a lot of our capacity is outbound. And we use a lot of different variety of both open source and our internal tooling to rebalance channels on the fly in order to streamline the experience for the end user.
Stephan Livera:
On the topic of rebalancing, are you using a circular rebalance? Or are you maybe also doing that approach I’ve heard where some people will actually differentially set fees on channels if they wanted to try to use that to entice someone to help rebalance the channel for them. Is that something you’re looking at?
Philip Glazman:
On both items, yes, and there’s also swapping services that we use. So there are three different ways that we go about this, is that we leverage a lot on Lightning Lab’s loop service, and this allows you to loop in, loop out, for the aim of changing the directional liquidity of channels. And this has been really, really useful for us, especially getting started. Well, actually, in fact, when we were first getting started [it was] really a situation of reaching out to different people and asking for liquidity and returning a channel in response over Signal and Telegram. And it is still like that today, and that’s something that I think would be worth talking about. But we do use loop, we use circular rebalancing whenever it makes sense fee-wise, sometimes swapping is kind of expensive. Sometimes circular rebalancing makes a lot of sense for us if we have the right channels set up for that. And we have started to experiment with using different fee rates for different channels in order to incentivize the routes that people build to use different routes, but it’s sort of early for us to really do this. I think this has been really successful for a lot of people and it’s something that we’re definitely taking note of. And these fees—I hope one day that we could accumulate a lot of fees and give back to our clients in some way. But yeah, I think this fee rate approach, it’s going to be [a] really, really desirable mechanism for signaling liquidity.
Stephan Livera:
Yeah. That’s really interesting to think about and just to see the way that it evolves over time. And you mentioned earlier—I mean there’s a few ideas we can go into here. One is this idea that River itself is being used as a swap service by some customers. They might send in, as an example, they might’ve earned some sats on Lightning, shot them over the Lighting Network into their River custodial account and then withdrawn them out as maybe they need US dollars or maybe they need Bitcoin on-chain. And that’s one example of people who might use River as a swap service, while at the same time you might yourself use swap services, right?
Philip Glazman:
Yeah. It’s it’s pretty funny. We’re using different swap services, and people [are] using us as a swap service. I guess it’s pretty convenient for the clients who use us for swapping and it’s probably worth finding [out] what that is for people that don’t know. But if you have Lightning Bitcoin, perhaps you want to get a UTXO, right? And so what people will do is they’ll create invoices in our platform and send Lightning Bitcoin there and because we don’t differentiate they can withdraw on-chain Bitcoin. And vice versa, if you have a UTXO Bitcoin you deposit to River and now you have Lightning and you can use Lightning different ways there. Some of the hobbyist groups, they’re very inventive. There’s a lot of people that also rebalance their own channels using us. So they’ll try to nudge us to create a certain route in order to rebalance a channel on their end. It is pretty great.
Stephan Livera:
They’re pushing their fees onto you guys, though!
Philip Glazman:
Well, they’re still paying the fee. But with regards to swapping, of course us having a lot of small UTXOs is not desirable. So we take an active UTXO management approach. All of our deposits go to this cold wallet and we always consolidate at low fee periods. We use batching. We use native SegWit. And so we created [an] approach where we’re offering this service to people that use River this certain way, but I don’t think we’re getting hurt in any way by doing this.
Stephan Livera:
Right. It’s cool in that way. You mentioned the Pleb routing node operators, there’s a bunch of Pleb communities. I think probably the largest and most well-known one is Plebnet, and then there’s many others out there. Obviously without doxxing, can you tell us a little bit about this user profile? I think that might be interesting for listeners to hear about.
Philip Glazman:
Yeah. they’re really awesome. I had a chance to meet a few of the plebs. They’re great. I mean they kind of fit into this hobbyist customer profile where they’re taking charge, they understand what Bitcoin is, and they want to participate in the Lightning Network on their own terms. And they’ve become a pretty central force in the public graph on routing through this self-organized community. There’s not much to say other than that. It’s really, really nice that it’s not just exchanges that are in the Lightning Network. It’s just everyone, right? It’s anyone that can run from a Raspberry Pi to a gaming computer or server, right? So it’s like everyone’s participating in this. Everyone is helping route, everyone’s earning some sort of yield on it by providing this public good. So it’s [an] amazing testament to what Lightning Network is. And to sum it up, it really is this politically neutral payments network that anyone can participate in, and it’s very different from other L1 or other L2 that really don’t have this property of free entry and free exit.
Stephan Livera:
Yeah. A lot of those groups have Telegram chat groups and they’re all sharing tips and tricks. I’ve seen other concepts like this idea of having a ring of fire and different ideas that are in play there to try to have a node that actually gets them routing because they’re addicted to this idea of, Oh, I’m going to make money on the Lightning Network. Even if maybe not everyone is actually profitable, but it’s a hobby for many people. And the other big profile that you mentioned is traders. So I am curious to hear a little bit about this and if you could give us your thoughts on this whole idea of Lightning versus Liquid for interexchange transfers, or is it Lightning and Liquid? How are you thinking about that?
Philip Glazman:
Yeah. So the traders, they used different exchanges in the past year. We’ve seen a lot of exchanges become onboarded with Lightning. Bitfinex really stands out [and] has done a really excellent job at this. There are new upstarts like LN markets and [inaudible] that are really using the Lightning Network in novel ways in order to create a trading experience that people want. And I guess in the beginning of Lightning it was doubtful on the ability to trade with large amounts. If you’re a trader, you’re moving size and you can’t be sending 10 sats and going 100x on an exchange and praying to God that you make money. So it’s changed a lot today. There are legitimate people and traders using Lightning Network as a settlement layer in order to move money around. It’s really desirable, it’s fast, it’s cheap, and it’s really easy to enter and exit very quickly. So we’ve seen a lot of people using Lighting Network that way, and again, that forced us to understand this user profile and balance their channels a certain way in order to accommodate for this user experience. Liquid stands out—it’s another really excellent L2 sidechain that lets people trade between exchanges with large sizes. I think what stands out is their technology is really interesting—having more of an interesting privacy technology that is really useful for traders. But for us personally, Lightning Network has been this politically neutral platform that I think is really different from a federated multisig. So we’re still thinking about how we fit in with Liquid, but we’ve seen a lot of demand more so on Lightning because of its emergent properties and its politically neutral stance. So this may change, but I think there’s a huge driving force behind Lightning than there is on Liquid, but it’s really interesting to see the different use cases. In some ways, Liquid is really great if you’re a trader and [it] might continue to be the premier place in order to settle between exchanges, but it really is a matter of adoption and liquidity and seeing how how things develop there.
Stephan Livera:
Yeah. The way I’m thinking about it [is]: obviously Lightning is faster, but Liquid doesn’t have that aspect of needing the channels to be in the right configuration. And in the case of Lightning, for the traders who are moving large amounts, they basically would need private channels between big exchanges, essentially, to facilitate what they’re doing. And I wonder then if it’s precisely at the moment that they need to be able to move their Lightning funds over, that channel would get exhausted because everyone’s trying to go the same way. So I’ve always wondered about that, and I’m curious whether that almost cuts against the case in some small way, although maybe it still makes sense overall to have it and to do it. But is that something you would see, or you don’t see that really as a concern or as a common sticking point for this idea of Lightning interexchange?
Philip Glazman:
Yeah. I think there are two points there that are worth discussing: there’s the privacy angle and then there’s the capital efficiency. So private channels are not actually private, they’re just not gossipped. If you’re intelligent enough to monitor the network, you can see when private channels are created and when they’re closed. So all that means is that they’re not gossiped. But the privacy context of Lighting Network is really attractive, where messages are wrapped in onions like TOR. And when you pass along this message, the person that receives the message has no idea what’s inside, they just know that I got this from one party and I’m supposed to pass it to another. So there’s a really nice privacy benefit there. Liquid has a different privacy context where using confidential transactions is really attractive and you’re able to look at a transaction on-chain but not really see the the amounts there. So that’s a nice feature. I think in some ways it’s really similar. I don’t think your privacy is diminished using one or the other. But with Lightning, you’re right, it’s very capital intensive. So you need to commit a certain amount of capital for a Lightning channel. And in order to send and receive, you need to be worried about the directional liquidity of your channel. It’s a lot easier to get bootstrapped if you’re using a custodial service—of course. And if you’re a non-custodial trader, you have a whole set of challenges and problems. I think there are a lot of easier tools nowadays to make this easier. So Lightning Labs has this pool service that lets you rent a Lightning channel, essentially—you can rent a rather large Lightning channel for some sort of interest rate, and you’re able to use their Lightning channel for your trades. If you’re a savvy trader, you could figure out, “[Does] this interest rate make sense? [Are] the gains I’m about to make on this 100x leverage worth it?” I think there’s a lot of novel ways that traders could use Lightning Network in a non-custodial way today that didn’t exist a couple of years ago. Generally speaking, I think with Lightning, one of the more existential questions that is worth asking is, [What is] the capital efficiency that different routing nodes deploy in the network? Is the Bitcoin here that I’ve locked into this channel—at risk, mind you, because there’s a security context, it’s a hot wallet—is it worth that routing fee? Versus lending it out or using a whole slew of different services. So that’s going to be a very interesting question that we’re going to see over time. And this question is going to develop into a more mature fee market on the Lightning Network, both for routing and also just for the maintenance of channels. A lot of people right now are using Lightning Network as a public good. There are a lot of hobbyists and they want the network to succeed, but fundamentally of course it can only succeed if the financial incentives are there. So there’s almost definitely going to be some sort of push in order to create the right financial incentives in order to route.
Stephan Livera:
Yeah, very interesting considerations. And you’re right [on] the capital aspect. You’re you’re tying up capital in a specific direction in the private channels network, if you will—unannounced channels, let’s call it, as another way to put it—so that’s the concern that people might have, but Hey, we’ll see what happens. Obviously it’s a market and people are going to try things and see what kind of experience they can give for the customers or for the traders and see what wins. I’m excited to see how that plays out. I’m curious, in terms of offering a Lightning wallet, has there been any mention from regulators about like, Oh, are you paying ____? I mean, it’s not like there’s darknet markets operating on the Lightning Network, at least that I know of today. But has that been a question asked or is that a future potential aspect? Because as you mentioned earlier, it’s meant to be politically neutral, but I wonder in the future, could that change if regulators or the government legislators [get involved]. Maybe that’s an angle there?
Philip Glazman:
Yeah. I’ll preface that I’m not a legal scholar or I don’t talk too much to regulators, but what I could share is that what we’re seeing now is a lot of larger players in the ecosystem want to adopt Lightning Network, but they need to have a way to get the check boxes filled out by compliance departments in their businesses. And I don’t think Lightning Network is something radically different from—essentially, I think it neatly fits into existing regulations, most likely. It’s just a matter of figuring out the risk profile with how you send money and receive money. Just like anything else. And so what’s likely to happen is this better tooling with regards to developing risk profiles and how you send and receive money. But this is a big question that a lot of people have to answer, especially at these larger institutions as they develop it. A lot of the same laws that apply to knowing your customer will apply to Lightning Network most likely, but that doesn’t diminish Lightning Network’s ability to operate. It is more inflicted upon businesses in order to comply with reasonable regulations. But people could always use Lightning Network however they see fit because it is a technology.
Stephan Livera:
Yeah. And I also wanted to talk a little bit about the IT, technical aspects of it as well—obviously as you’re very familiar with [it]. What is it like running larger Lightning nodes and infrastructure, and maybe you could tell us a little bit about the focus of running on-premise hardware as opposed to the hosted node approach?
Philip Glazman:
Yeah. We take pride in not using the cloud. Most other people, when they’re setting up their business, they will subscribe to G Cloud, AWS. We took a different approach. We went to the data center, got hardware, and run all of our critical services on this hardware. Why do we do that? Because we know exactly how many people have access to this hardware. And if you were to ask Google or Amazon, How many people have access to my G Cloud account?
Stephan Livera:
Because it might be like third party, third party, third party—all these contractors—and they wouldn’t able to figure it out.
Philip Glazman:
Yeah, it’s a mess. And we’ve seen recently, there was this Capital One issue with AWS instances being pwned by an insider. It’s really scary. So I think our decision to self-host has been really useful. It creates a lot of peace of mind for both us and our clients. But there’s still a bunch of challenges, because unlike a cold wallet where the keys are never in a server, Lightning keys are generally hot, and the Lightning state machine is very complex. There are a bunch of different keys that are involved in both creating a channel, updating a channel, the relocation key. There’s a lot of moving parts there. I think if anyone’s listening the Suredbits blog post on Lightning Network for exchanges is an excellent starting point to understand the security awareness around the different keys. And there’s all these different projects like /dev/random, the Lightning Center project. There [are] initiatives by both LND and C-Lightning on how to really secure the different keys. And when I speak with different exchanges it’s always top of mind where both the security and operational security of running a Lightning node is pretty intense. It’s not like anything else in crypto. I think so far we’ve taken a really good conservative approach to securing funds. Just top of mind, we have a limit on how much money we are willing to put into this, essentially, hot wallet. And we abide by that limit, and we’ve taken a lot of steps in mitigating the level of hotness it is. So a cold wallet is an air-gapped machine that contains keys and is able to sign. Some of the keys in the Lightning Network state machine can be cold. And we’ve leveraged HSMs in order to keep those keys cold and still have reasonable high availability in order to sign. There are still some other keys that are still in the node that we’ve slowly been trying to rip out. And at least for us, a lot of it is also driven by community efforts. There is amazing work being done by LND—we run a lot of LND nodes because of our code base being written in Go, [and] relationship with Lightning Labs. And they’ve taken a really good architectural decision about making sure that you’re able to do some remote signing. So a Lighting node could run as it is, but it could be a watch-only node, and you can have the signing being done in your own custody setup. So essentially you can have these HSMs be completely cold or warm depending on the high availability nature of the signers and allow different requests between the Lightning node and the HSMs [for them] to have. Today it’s a little bit different. That’s the ideal state. Today, we rely on HSMs for some of the keys, but there’s still a lot of ways to go. There’s a lot of networking things we’ve done to make sure that our networking is locked down. There is container security, there’s hardware security. It really is a rabbit hole with regards to securing Lightning nodes.
Stephan Livera:
Yeah, that’s interesting, hey. And your blog post also mentions this idea of exposing only one node out to the network and having some in the background. Could you explain a little bit about that idea?
Philip Glazman:
So at a high level, our architecture design is that we have a routing node—and this is a publicly available routing node—and we have several end nodes. And these end nodes can be spun up and destroyed at will. And what connects these end nodes with the routing node are these unannounced channels—private channels. And we only expose one routing node to the network just for the ability of having a really central routing node limit the amount of capital locked up. So far the architecture has made a lot of sense. I’ve noticed a lot of exchanges spinning up multiple routing nodes for different reasons. I think for us what’s going to happen is that because we have these limits on our routing nodes and the amount of money we’re willing to be comfortable putting into this system, it’s likely we might have multiple routing nodes. But it doesn’t mean that there is segregated security or extra security if you have another routing node, especially if they’re running on the same host machine or the same hardware. There’s still a lot of tooling with regards to building a really good routing node architecture there. Routing node—to explain to people that are unfamiliar—this node essentially handles the relationships to different peers and manages the liquidity between different nodes. The end nodes are just creating invoices and building the routes, so they’re pretty dumb in that regard. They have capital locked in with the routing node, but we’re able to spin them up and tear them out, tear them down, and have different destination pubkeys and such. And the real reason is just that it gives us more high availability around that layer.
Stephan Livera:
Yeah, interesting stuff. You mentioned also this channel acceptor policy. And I’m curious if you could spell out a little bit there—I presume some of this will be public and some is private, it’s obviously whatever you can say. What’s the policy there, if someone wants to open channels with you guys?
Philip Glazman:
Yeah. I wish we could connect with everyone that wants to connect with us, but at the end of the day, because our routing node has a function to route, we have to make sure that we have the best peers. We only want to be connected with peers that have high uptime, have really good visibility on the public graph, are able to route, have very low payment failures, and are able to create channels that are very sizable. And if we were to accept everyones’ channel open requests it would probably diminish our ability to route successfully to people. So what we do is we have an allow list of public keys—identities of different nodes in the network that we deem as really great routers. We occasionally add to this list whenever we have requests. A client that wants to connect with us will be added to this list. This just allows us to route better, but it also prevents us from having zombie channels. These are channels that somebody made one day, thought they were going to use the Lightning Network, and then their power cord or their Rasberry Pi got disconnected so they’re offline. We don’t want those because that diminishes our ability to route. Also, if we have trusted peers, other peers that are either clients or exchanges, it helps us in our backup ability. So using the data loss prevention protocol, I think is what it’s called, the set of channel backups. The ability to—if we go offline—we’re able to come back online and ask our very trusted peers on the last commitment state. This probably isn’t possible if it’s just randos on the Internet. The other thing is there are definitely unknown attack vectors that we might not know about. And just keeping a more closed system creates a lot of peace of mind.
Stephan Livera:
Yeah, totally. And I think those are totally fair considerations. I also wanted to bring up the point around comms channels with Lightning peers. So obviously the Lightning Network itself can be used for messaging, but the problem in some cases might be, That peer has gone offline, as an example. Maybe in the future there’ll be more that evolves out here. As an example, they might be saying, Oh, hey, we’ve got a 1 Bitcoin channel, but I’d like to up to 2, are you cool with that? And things like that. So where do you see the future of communications between Lightning peers going?
Philip Glazman:
Always to whatever is easier. It has been Signal and it has been Telegram, it has been Twitter. It’s really informal. And that’s what’s kind of cool, right? You have organizations like Plebnet where it’s just people chatting and opening channels with one another. Today it’s pretty informal. I hope it stays that way. But as institutions enter the Lightning Network space and as businesses adopt it, there’s going to have to be some sort of SLA (service-level agreement) between different exchanges. And it’s likely there’s going to be some sort of formal communication structure between different parties. That’s where I think it’s going, but as of today, if you’re just getting started with Lighting Network, it’s really reasonable just to message people if you have their Telegram or Signal and talk to them and ask them for some inbound to some channel and then you’ll return with a new channel. This camaraderie works really well. At least from my experience, I notice that [when] somebody goes down, I’ll just send them a message on Signal: “Hey, is everything okay?” And that’s happened to us too, where people message us to make sure that we’re okay. So that’s the type of informal communication structure and it will probably last for quite a while. It’s fun.
Stephan Livera:
Yeah. I see. And I guess a lot of this is just because Lightning is so young, right? As we look at it today it’s maybe just under 16,000 Lightning nodes out there and around 73,000 channels. But let’s say that number goes 10x. Well then maybe it might be harder at that point. So yeah, it’s certainly interesting to see where that goes. And also that idea around SLAs is definitely an interesting one because I can imagine—so for listeners who aren’t familiar, SLA is service level agreement—and basically an IT company might agree, Okay, we’ll provide X amount of uptime—99.999% uptime as an example. So I wonder what would a Lightning SLA look like? Okay, this number of payments, this level of no payment failures, or what percentage, and this amount of liquidity, and this amount of uptime—I guess that’s what it might look like as the industry professionalizes.
Philip Glazman:
Yeah, I think so too. There’s going to be a level of maturity and it’s warranted with the level of money entering the system. It will be very curious to see what it looks like. The community is still super vibrant and the suits haven’t entered yet. There’s going to be, over time, some level of maturity around certain things. And that’s good because at the end of the day, people are going to get better reliability and routing out of it.
Stephan Livera:
Excellent. All right. Well, I think that’s just about all the time we’ve got for today, but Philip, if you’ve got any closing thoughts for listeners, and of course tell people where they can find you online?
Philip Glazman:
Yeah. Super fun chatting with you. I’ve been listening to the podcast for quite a while. Long time listener, first time caller. You can find me on Twitter by my name. I’d recommend people to check out the River blog if you’re first getting started with Lightning. There’s so many resources. Lightning Labs has a builder’s dock. There’s a ton of different GitHubs of awesome Lightning links and just delving into it is really, really fun. I think with any new technology, it’s one thing to talk about it, and then it’s another to actually use it and try it. So hopefully I was convincing enough [for you] to use the Lightning Network, just go ahead and download a non-custodial wallet and try sending money around. Then go ahead and build your Lightning node and connect with peers. And it’s really an awesome opportunity in human history to have this ability to freely exchange value. So take advantage of it.
Stephan Livera:
Fantastic. Well, Philip, I’ve enjoyed chatting with you as well. Thank you for joining me today.
Philip Glazman:
Thanks!