https://twitter.com/kanzure/status/1136924010955104257

How much should Bitcoin Core do, and how much should other libraries do? Andrew Chow wrote the wonderful HWI tool. Right now we have a pull request to support external signers. The HWI script can talk to most major hardware wallets because it has all the drivers built in now, and it can get keys from it, and sign arbitrary transactions. That's roughly what it does. It's kind of manual, though. You have to enter some python commands; call some Bitcoin Core RPC to get the result back in; so I wrote some convenience RPC methods for Bitcoin Core that lets you do the same things with fewer commands. So now you can do enumeratesigners and bitcoin-cli fetchkeys and it will do all the dance to get all the keys off the device and put them on the wallet. The public keys, not the private keys. Yes, the private keys would be bitcoin-cli footgun. Then there's a command to display address. Any address you want, then that RPC method would look in the wallet, find the actual address, get the keys out, tell the signer, get the derivation path so you don't need to manually find out the derivation path. Also there's signtransaction which does all the magic of composing a-- it's essentially the same as sendmany or createpsbt. You give it a bunch of destination addresses; it will create a PSBT, send to the device, wait for the device to sign it, and then broadcast it.

Ideally we should not have as many commands, everything should occur internally. Imagine you had a perfect descriptor wallet. What would the delta be? The convenience method could be removed from the pull request and moved separately. The one that is in there a lower-level one is processpsbt. So there's a signerprocesspsbt which does what processpsbt does. The only thing the signer send convenience thing does is that plus a few other things. I could remove that from the pull request.

Related to the wallet re-architecture thing, it sounds like you have jammed in the hardware wallets. Well, it's signer agnostic. You could use this with hardware wallets but later maybe also multisig. We are going to have native descriptor scriptpubkey manager, and then an external scriptpubkey manager. The manager is responsible for signing. It will be responsible for signing.

I still think the API is not ideal for mass adoption. I once proposed, not as a BIP, but in some forum, about a url scheme to communication between watchonly and signing devices. I think the current model will not work with mobile wallets, right because the HWI script is not something that can be transported; so they would use the RPC commands I guess. The GUI would need a way to display addresses from a device, and also a way to sign things. There needs to be a class to communicate between the signer and the GUI.

Why not work on an API that is completely agnostic of vendor? There is a software wallet, Bitcoin Core, and a hardware wallet like the three that HWI represents. Couldn't it be fully flexible? The URL scheme can be ideal because Bitcoin Core could call https://sign/ or something. This requires a running server. One idea was to have json-rpc. Something needs to listen for that url scheme. So this could work in both directions. Doesn't this require the user to initiate some actions? You can't enforce which apps handle which URIs. If you install a piece of malware, you can redirect URIs and not your intended... well, having malware on the machine is pretty arbitrary. GUIs launch a specific application; is there a URI scheme where the right kind of application can respond to that? A arbitrary wallet can listen to the URI scheme.

Having a separate python script is from a code review perspective kind of easier, because we can keep that separate from the main project at least from a while. Otherwise you would have to include USB drivers. The general format that we communicate with, there's a message signtransaction that takes some arguments, and we can standardize the protocol but not the connection method for now. Right now it's just executing another command with a bunch of arguments. I wanted a scheme that works for the whole ecosystem. We need to make a model where all wallets generically know how to talk with all hardware scheme. URI schemes could work for this. With HWI, it's all the same commands for every device except Trezor. Roughly. So all the same commands are available. Whenever you do something, regardless of the device, you could use approximately the same commands every time. The only thing with HWI that is super bitcoin specific is whatever the getkeypool because that uses the importmulti command which is easier. I added a pull request to HWI for get descriptors which gets descriptors in an array which is super universal.

When I look into the URI thing, I found the only way for two apps to communicate which covers all major devices, is URI scheme. You could add to HWI or some other tool, some URI handler that maps the URI to those commands and then it keeps a note about how to respond to where it came from. Every command that you call in HWI could have an optional callback trigger.

If you have multiple hardware wallets, becaues you are using multisig, and you want to choose a particular hardware wallet. It would be solvable with some extra steps. You pass a fingerprint to HWI of which device you want. Bitcoin Core knows about the different hardware wallets because when you import ... Bitcoin Core wallet has master fingerprint entries, and then you can query HWI by master fingerprint. It says "use a device for this fingerprint" and then it will use it.

Having a protocol that can be extended to do other features that hardware wallets don't currently do, like consensus rules, is really important. It must also be a universal transportable protocol that works for all software wallets and all hardware wallets. A bigger project would be to expand libwally to have all the USB drivers and all the stuff in it; you take all the python stuff in HWI and we turn it into C. Say I am a watchonly wallet, I can make a PSBT, I have an address I want to verify- very high level on the transport layer, that is transportable on all devices. The messaging happens to come in through a command but the messaging could come in through json-rpc or a URI scheme. We should do this URI scheme now, not later. Before it's set in stone, we should make something that is more platform independent.

There's a nice separation between receiving the commands from the command line and actually executing them with all the arguments. It is easily portable to a wrapper to just put a wrapper around the commands. Someone could make a pull request and add a json-rpc server, or a URI handler. Or just layer the transport layer from the content you're moving around. Someone added cli.py that takes input from the command line, and then it calls functions that are agnostic to the fact that they were called from the command line, and those functions return a json object. I think that's already there. I would like to see a json-rpc server, but the problem with it is that anything on the system can call it, in particular ... yes, you could do some authentication. You can use a cookie. This seems preventable.

I am always worried about the initialization process. Say if you talk with UI hardware wallet extensions, you get more to normal people who aren't used to using PSBT. What about the hardware initialization? Device setup, yeah. Using Trezor, you need to use Chrome to initialize Trezor to use it with Bitcoin Core. Can you only use Bitcoin Core? Trezor yes, Ledger no. Ledger setup is unrelated to your computer which is nice. The problem with setup is that for some devices there's interaction for setup. It's completely interactive which means if you want to integrate it into Bitcoin Core then you would have to have something that handles interaction, something device specific. With other devices, like Ledgers, setup happens completely independent of your computer. You plug it into power, not necessarily a USB port, and you can setup the device. This difference makes setup rather annoying. When you download Bitcoin Core, start it, and it prompts you whether you want to make a new wallet, and then it would ask do you want to use a hardware wallet, and then it magically enumerates all the hardware wallets, you select the wallet you want, boom the keys are in there, and when it signs a transaction it realizes it needs to ask for an external device and then prompts you. Given an already initialized device, the setup from Bitcoin Core to register the device is pretty easy. If we can make setup somewhat generic, but yeah every device has its own process.

My third concern, sorry for bringing only concerns and not answers, the whole plugin model. I saw that with electrum it's a single code base maintained by ThomasV so whenever there's a bug in one of those plugins, like Trezor found a bug in its own plugin within electrum, they have no control to fix it. It's a layer removed of responsibility. They have no control about the release cycle. If it's crucial maybe they call ThomasV but it's not reliable. The driver should be the responsibility of the vendors, which we violate with HWI. Right now you have that responsibility. The reason why HWI does that is to remove the need for external dependencies. I went through Ledger and Trezor git repos and cloned them and then deleted unrelated files. I wanted this so that the libraries were the dependencies so when they update a driver, you would automatically have the updated driver from the vendor. The problem is that they add a bunch of other dependencies like requests and other dependencies to do network fetching and other dangerous stuff. Trezor if you're not careful will just automatically call their servers to fetch previous transactions.

I think HWI is violating a few layers. These hardware wallets wanted to interact with Bitcoin Core but they didn't know how. We should provide an API, not HWI. They were willing to write this stuff, but now we have done this. I think it would be worth asking the hardware wallet manufacturers to provide a stripped-down version of their own libraries. Or they could provide an HWI-equivalent interface. We have HWI as a python command with certain inputs and outputs, and any vendor that can produce inputs and outputs that's the driver. So this can define the interface. It seems like hardware wallet vendors are hesitant to do anything. The HWI drivers are pretty thin anyway, which they are stuck with if the vendors don't do anything. The driver is like maybe 100-200 lines.

With a URI scheme, all the vendors could write a single application for iOS or desktop and the API would work across all platforms. We need to live with the fact that because of the USB security threats we can't have a plugin model in Bitcoin Core. Then there's the question of HDI and storage device and everything, it's too complicated.

The Nano X has bluetooth support... so you could have a driver that communicates over bluetooth. For iOS you need it, right? You can't register USB dongles or something. You can, you just need a made-for-Apple program and use their facilities and data and a lot of money. Hardware wallets can do that. So the vendor has to provide an app that meets the interface requirements that you want. The vendor is responsible for providing it. But it's annoying now because you have a hardware wallet, and your computer, and now it needs to go through an iOS app but you could also just plug it into USB to the computer.

This HWI can be expanded to a URI scheme or json-rpc scheme. The library is flexible enough to just add that. For the way we do it now, I don't think we need it. For usability point of view, imagining how this would look with the GUI. You have to tell users to go down this thing, and you have to point through the HWI script and that seems tedious. I think that's fine for first.... Bitcoin Core could just once you create a wallet, it could use a hardware wallet, it could use URI schemes, it doesn't need to install anything. I like this more than a json-rpc server because it's not running all the time. I don't intend on making an installer for HWI because that's hard. We could add HWI to gitian output or whatever so that it's distributed with Bitcoin Core releases. There are determinstic-ish builds for HWI binaries except on MacOSX. Electrum said they couldn't get deterministic builds for MacOSX either. For MacOSX, not from MacOSX. With the python all in one binaries, you can only, the existing libraries to make them only work-- only build for the device you're currently using. So to build MacOSX binaries, you need to have a mac and it will just build it there. One possibility of this is that we could have gitian or whatever we're using down the line, to package these HWI binaries with the installers. So then, when Bitcoin Core calls out to it, it knows where it wil lbe, and when users use it, they don't have to go and download something. Does this make us the vendor? We're already shipping HWI just not together. We would like hardware vendors to provide this thing themselves and their own installers. Currently ytou need to install it, either way you need to install it.. right now drivers for hardware wallets or something. You need to install the bitcoin app on the Ledger device; but I think it stops doxxing you now, you can go to the Ledger manager and install the bitcoin app and you don't need to open it so it doesn't need to know your xpubs. Long-term, it should just be UI install vendor-specific application that needs to be installed anyway for backups, firmware upgrades, then you install Bitcoin Core, and they talk together. I think the only thing that would work is URL schemes and GUI only. Any progress on URI schemes for giving specific instructions? You can register an application, but you can't say call Bitcoin Core with a certain item in there and hten pick the right app.

I could just imagine for URI schemes, I could call bitcoin-sign:// and each application would pickup that url scheme. If there are many, then your oeprating system should ask which one you want to use. I could also enumerate over that, then I could call in the URL scheme I could pass data, like callback on some way to get data, or pass direct data. You can add callbacks and other things. Device can listen based on master fingerprint, but you might not want the fingerprint to be registered in the URI handler registry.

Someone should look and figure out if this is going to work or not, and if not, then find the most portable solution. Maybe there's too much friction. Applications on macosx can register URI schemes and URI handlers. The idea of having the URI be the fingerprint for the device could be possible and then you would definitely get the right application. Applications need to know this at install time. For iOS app, you have to do it at shipping time. If you share a link on Android, it's exactly what the operating system does.

I added an external signer class in Bitcoin Core and it currently calls commands but it would be trivial to instead of calling commands to open a URI. Being aware of layers, there's a transport layer, and the whole protocol. I think most of this discussion is we should investigate other ways of standardizing the interface. Maybe it would be worth it to talk to hardware vendors.

I think people think of hardware wallets in terms of what Bitcoin Core can do today. PSBT was quite recent. What about extensions like lightning-specific wallet devices? Have we really thought this through? PSBT doesn't work very well on low-memory devices. PSBT gets very big. This was one of my concerns, because on low-memory devices you parse data and try to process it and throw away what we don't need in order to verify... the hardware wallet vendor driver app should take PSBT and do something efficient, that's what Trezor does, no that's what all of them do. The only hardware wallet that takes a PSBT is Cold Card. I've been using it without an sd card. Do they store it in RAM and go over it multiple times? I think they have a large amount of RAM. A few megabytes would be nice. The cold card has enough RAM to just, have 256 kb of PSBT in memory and that's how much they have allocated to holding PSBTs in memory. You could also parse over the wire. The signer only needs to hold in its head the script code and a couple other things. Mostly, it's pass a PSBT to whatever software driver. But you could also look for keys, and derivation of bip32 keys.

Do the vendors like PSBT? I have heard no feedback from most. I've had a lot of complaints from Trezor... and Cold Car basically said "this is great". Trezor wanted protobuf. I had posted it on the bitcoin-dev mailing list. The trezor people responded to that. I think Sanders had asked Ledger out-of-band and then Cold Card contacted me out of band as well. There are other vendors in Asia that do hardware wallets, have you got in contact with any of them? I haven't gotten feedback from anyone else but I haven't really asked.

We have a common format that vendors can build on and use for their wallets.

gwillen has an open pull request for offline-qt where you put it in a mode where you click "send", fill it out, "send" and it pops up a wizard for PSBT that has different, and you copy the PSBT and then you get all the coin control of the GUI but in the end you can either save to disk in binary or copy it in base64 and then go to your offline machine running Bitcoin Core as well and then do it. So today, you could use that instead of using wallet createfundpsbt or whatever. It's another slight UX step that I would appreciate. Needs a way to collect payment plans and then load the plans first before going into gwillen's wizard, in my opinion.

The external signer class should be available to the GUI and then the GUI can be aware of hardware devices. I think we need to end the session, but one more question. I could continue with my pull request 15382, one is to wait for descriptor wallets and another is to not wait. The downside of waiting is that we need to warn the user to only use one address type. The way our wallet works right now is that when you import keys, it creates an omnibus of every type of address, and that means the derivation paths are incompatible with other wallets. Not that it wouldn't work, but you wouldn't be able to get your coins back if you open it with a normal software wallet. This could just be a warning though. Maybe not waiting for the descriptor wallet, but maybe waiting for the re-boxing of the wallet architecture. I don't think the box gets in the way of the ... it's better abstraction. The wallet architecture-- there will be an issue describing it. You could read the transcript from yesterday on wallet architecture and boxes too. I am just using importmulti indirectly. When we get this abstraction into Bitcoin Core, it would allow for this device only supports X address type or whatever, and then it would also make it easier to have within that class, the drivers but like the-- the wallet hardware and wallet-specific things for calling out, instead of having those being separate. Right now what I find is that I'm adding a lot of code to the RPC that I don't want to be in the RPC. Some of the code I know where it needs to go; we've been taking signrawtransaction and creating their own files for that. Especially the import stuff, that's still a lot of code in the RPC, and your box might add some abstractions to that, so yeah I guess I could wait for your box but not the descriptor box. Yeah you don't have to wait for the descriptor box. I have one other pull request which is a prerequisite, it's not a big pull request, I would like some code review, but it does use boost-process and that is giant. So there's other libraries for it... Different boost versions will break compatibility. If you're not using the wallet then you don't need this. This would be a requirement on.... if we use the walletnotify stuff, we can only call commands and not get stuff back. So that's a problem, you need boost-process, or maybe some other library that does the same thing. I looked at this a while back, and by the way they broke compatibility on a new version of boost-process. It's very large. It's not available on all platforms, either. The default boost version is on various distros but not this. The alternative is some other library. What would you recommend, like IPC including launching an application, which is portable? This seems like the start of a remote command execution vulnerability. Obviously, calling any binary on your system is always a risk. This needs to be handled by the operating system. Building some stuff around command execution and stuff, I don't know. The other thing is that if you don't want hardware wallet support, then you don't need boost-process. But when we ship the binary, it's going to be in there. We always have to support the maximal feature set for each platform or at least attempt to. It depends on boost 164. boost-process is in boost 164 but some distros don't have it. They have also broken the API since then.. I know that for a fact. It did change before, and then maybe they fixed it. We just updated to boost 170 so I will see if that's still compatible. If people could look at that pull request, and look at it, or maybe we should investigate other methods.

We could run a server in the HWI script, and then we communicate over RPC between HWI and Bitcoin Core. It's far from perfect but it's an option. If people respond to that pull request, they could suggest other approaches for communicating with it. It's one pull request I'd like to get merged in some shape.