Lightning Specification Meeting 2020/11/23

[t-bast]

The meeting will take place on Monday 2020/11/23 at 7pm UTC on IRC #lightning-dev. It is open to the public.

Pull Request Review

• Correct HTLC size calculation Correct htlc witness size calculations #815
• Public-key based routing Public key-based routing (Feature **NEEDS NUMBER***) #814

Issues

• reply_channel_range is too strict reply_channel_range is now too strict. #804 Encodings in gossip_queries #811

Long Term Updates

• Offers Offers #798
• Blinded paths Route Blinding (Feature 24/25) #765 (@t-bast)
• Evaluate historical min_relay_fee to improve update_fee in anchors (@rustyrussell and @TheBlueMatt)
• Upfront payments / DoS protection
• Trampoline routing Trampoline Routing #654 (@t-bast)

Backlog

The following are topics that we should discuss at some point, so if we have time to discuss them great, otherwise they slip to the next meeting.

• Hornet (@cfromknecht)
• lnprototest https://github.com/rustyrussell/lnprototest (@rustyrussell)

---

Post-Meeting notes:

Action items

• Require to claim revoked local output in its own penalty tx post-anchor #803 small comment pending
• @ariard to investigate github security feature/page
• @ariard to write up concrete proposal [for update_fee changes] for next meeting
• Double-check the calculations in Correct htlc witness size calculations #815 then merge

[cdecker]

I'd really like to make some progress on Blinded paths, and offers (which @rustyrussell recently demoed to the c-lightning team), so maybe we can move them up a bit in the LTU section?

[t-bast]

It would be my pleasure! In that case I'll do a pass on all the pending comments on blinded paths this week.

[cdecker]

That'd be great, I think the blinded routes proposal has had quite a few eyes on it, and if no issues where found so far, it's time to move forward ^^

[niftynei]

We've still got two pending merges from last meeting: #810 needs a second ACK and #803 has an outstanding minor fix request (MUST->SHOULD)

[t-bast]

Indeed, thanks for the update!

[t-bast]

Here are the logs from the meeting:

<cdecker> Don't have any objections ^^
<t-bast> #startmeeting
<rusty> Ack!
<t-bast> hey rusty!
<t-bast> The bot is still down, but I'll copy-paste my logs in the github issue
<t-bast> #topic Previous meeting leftovers
<t-bast> ariard told me he can't attend tonight, so we'll skip the action items that were assigned to him
* jonatack has quit (Quit: jonatack)
<t-bast> #link https://github.com/lightningnetwork/lightning-rfc/pull/810
* luke-jr (~luke-jr@unaffiliated/luke-jr) has joined
<t-bast> This PR was pending a second ACK
<t-bast> IIUC roasbeef seemed positive about it, but haven't heard back. roasbeef are you there? :)
<BlueMatt> ACK'd
<t-bast> great, that does a second ACK then, shall we merge?
<cdecker> I count 3 acks (assuming you support your own proposal) :-)
<t-bast> Perfect let's go
<t-bast> #action Merge 810
<t-bast> Let
<t-bast> Let's start today's agenda now
<t-bast> #link https://github.com/lightningnetwork/lightning-rfc/issues/816
* BlueMatt notes that this *could* be a little looser while not causing force-closes, but I also am not sure its possible to implement practically without just sending in the same order, so whatever.
<t-bast> #topic https://github.com/lightningnetwork/lightning-rfc/pull/815
<t-bast> Look likes the spec 1.0 was wrong about the tx size calculations...woops :)
<roasbeef> yeh 810 lgtm
<BlueMatt> I didnt double-check the numbers - is this due to the low-s-sigs-can-only-be-72-bytes thing?
<t-bast> thanks roasbeef
<t-bast> BlueMatt: I don't think so, it's just that we counted an expiry to use 3 bytes whereas it actually uses 4
<BlueMatt> oh, no, its also cltv_expiry, ok, nvm
<BlueMatt> wellllll...do we *want* to consider that signatures can never be 73 bytes in these numbers?
<rusty> Meh, it'
* lnd-bot (~lnd-bot@165.227.7.29) has joined
<lnd-bot> [lightning-rfc] t-bast merged pull request #810: Clarify relative order of some messages after reestablish (master...issue-794) https://github.com/lightningnetwork/lightning-rfc/pull/810
* lnd-bot (~lnd-bot@165.227.7.29) has left
<rusty> s a made-up number anyway.  But yeah, embarrassing.
* lnd-bot (~lnd-bot@165.227.7.29) has joined
<lnd-bot> [lightning-rfc] t-bast pushed 1 commit to master: https://github.com/lightningnetwork/lightning-rfc/compare/dcf6b0f359c2...920e51986d65
<lnd-bot> lightning-rfc/master 920e519 Bastien Teinturier: Clarify relative order of some messages after reestablish (#810)
* lnd-bot (~lnd-bot@165.227.7.29) has left
* lnd-bot (~lnd-bot@165.227.7.29) has joined
<lnd-bot> [lightning-rfc] t-bast deleted 0 commit to issue-794: https://github.com/lightningnetwork/lightning-rfc/compare/fc07aabd93ff...000000000000
<cdecker> Just a sanity question: it's not like this will end up causing an incompatibility, is it?
* lnd-bot (~lnd-bot@165.227.7.29) has left
<rusty> (I did those original numbers, too :(
<BlueMatt> cdecker: no, we all use the hard-coded numbers there, changing them should not change any implementation, afaiu
<lndbot> <johanth> yeah, it was just a few inconsitensies I noticed when I wrote some test assertions for the final txs we create
<lndbot> <johanth> probably not a big deal, but it leads to some head scratching
<lndbot> <johanth> as long as we all use the same constants
<t-bast> We need to keep using the previous constants, but it's worth fixing the spec like johanth did IMO
* jonatack (~jon@88.124.242.136) has joined
<lndbot> <johanth> especially this one I have no idea how was calculated before: https://github.com/lightningnetwork/lightning-rfc/pull/815/commits/a9e064d04f313032d38407f75c3f1c53e32ad562#diff-6bed824879b760d329ec379b16a1d0e78ffba034fdfa13b95cf0480e09fa7c4bL919
<rusty> t-bast: agreed.  Ack, and I like the wording too.
<t-bast> ACK, I think it helps to clarify the spec if someone re-does the calculation and doesn't understand the results :D
<cdecker> ack ^^
<t-bast> johanth: *magic*
<lndbot> <johanth> t-bast: yeah, that is mostly it. I wasted a few hours scratching my head here
<t-bast> the CI doesn't like your `reeeeally` though, can you fix this johanth?
<t-bast> and then I believe we can merge
<lndbot> <johanth> sure.
<t-bast> #action fix CI then merge
<lndbot> <johanth> someone should double check m calculations tho heh
<lndbot> <johanth> my*
<t-bast> johanth: I did check them, not with code but by hand and I agreed with you, will do another pass to be sure
<t-bast> #topic Public-key based routing
<lndbot> <johanth> awesome
<t-bast> #link https://github.com/lightningnetwork/lightning-rfc/pull/814
<t-bast> joostjgr floor is yours ;)
<joostjgr> Thank you. What do people generally think about the proposal?
<rusty> "the math said 702 and 705, but after intense negotiation with world leaders we settled on 703 and 706"
<roasbeef> main Q for me is what wallet authors and other services think of it 
<roasbeef> it's an old idea, made  bit slimmer now by the tlv onion payloads 
<joostjgr> Yes, def wouldn't claim it is a new idea
<roasbeef> it adds a bit of guess n check forwarding wise re the proper policy, but I think a few other impls already do something like that since you can pick which channel to use anyway 
* jonatack has quit (Ping timeout: 256 seconds)
<roasbeef> IIRC pheonix uses a form of this as well t-bast ? 
<roasbeef> to like make new identifiers for clients that don't have chans on the fly? 
<t-bast> I think it's simple enough that implementing relay for it is trivial, and then once routing nodes have upgrade wallet can leverage it. The main Q for me is whether wallet will want that or blinded paths (given that they'll need to wait for at least some routing nodes to update before they can release it in wallets)
<t-bast> Yes Phoenix uses a variant with a fake short_channel_id, we could experiment with it easily becuase our node is always the next-to-last hop
<joostjgr> I think you don't need to wait for routing nodes to upgrade. Receivers can see the feature bits of their peers and decide based on that whether to omit the channel id
<roasbeef> upgrade paths seems very diff tho right? this is an e2e update, not super up to date on blnded paths but iiuc it'll take _all nodes_ in the path to upgrade so its an internal update 
* jonatack (~jon@134.19.179.163) has joined
<rusty> Yeah, I went for the same form in onion_message: you can use scid or key.  But that's because in theory you don't need a channel, and it can certainly change.
<t-bast> yes that's true, upgrade path is definitely lighter than blinded paths
<roasbeef> also this doesn't affeect what gets back to the sender, and I think blinded routes ends up giving them less info on failures? 
<joostjgr> (ofc if no routing nodes ever upgrade, receivers never see the right feature bit and the invoice will stay as is)
<roasbeef> the code impact for this is also pretty small impl wise I wager 
<roasbeef> and can be phased in with an optional bit in the invoice, etc 
<roasbeef> i think the invoice bit might be missing from the PR last I checked 
<cdecker> Technically any unique alias for a peer can be used to indicate the direction to forward, scid just happened to be shortest one we already had. But I don't think we should be adding aliasing schemes willy nilly
<t-bast> I agree, this is very light so I'd consider adding the relay code to eclair
<rusty> I'm missing how this helps private routes?  Or are you assuming a single private channel?
<roasbeef> cdecker: can just treat the 33 bytes as w/e identifier 
<joostjgr> I tried my hand at a rough PoC impl in lnd: https://github.com/lightningnetwork/lnd/pull/4758
<roasbeef> rusty: I think it's more about not giving away on chain data in invoices (that's how I see it at least) 
<t-bast> rusty: it's just that it doesn't point to an on-chain tx, so you're not revealing the amount you put in
<cdecker> Sure, we could use 33 bytes as a very verbose alias that is guaranteed to be globally unique, whereas blinded paths gives us a naming scheme that doesn't require global uniqueness, just local to the routing node
<rusty> If you put an all-zero scid in the bolt11 routehint, how do you handle old readers who will put it in the onion?
<roasbeef> they seem distinct tho, and also have very diff deployment timelines 
<roasbeef> rusty: a feature bit in the invoice? 
<rusty> roasbeef: you can only break old readers, though.
<roasbeef> may be something that needs to be required from the get go 
<rusty> So this will take along time to deploy.
<niftynei> this seems like a dumb question, but this change is on the hop-level, right? so instead of specifying which channel to forward the payment to, you'd specify the node id
<roasbeef> it's still an e2e update tho 
<t-bast> you should set the feature bit as required in that case I think
<roasbeef> niftynei: yeh
<t-bast> old readers would then fail when reading the invoice
<roasbeef> so up to the receiver if they really want this protection or not 
<t-bast> I'm a bit afraid receivers would not activate it because they can't know whether senders will support it...
<t-bast> Unless it puts a bit of pressure on wallet maintainers to add the feature
<rusty> Ick, it's kind of bad that it's useless until everyone upgrades :(  I'd really rather move to blinded routehints everywhere, which offers better privacy.
<roasbeef> blinded route hints would have the same issues, and reqwuires even more nodes to update to have it be utilized 
<niftynei> this feels vaguely close to rendezvous routing, in that you're giving the onion unwrapper some leeway in *how* to get to the destination, no?
<joostjgr> An alternative is to just use the pubkey truncated to 8 bytes as the channel id in the hop payload
<t-bast> definitely, but at least we're a pain to wallet developers only once, we don't ask them to implement pubkey_routing and then route blinding :)
<roasbeef> this is one of those things also that ppl can just go ahead and deploy, if they want to use another value in the tlv name space 
<roasbeef> niftynei: kinda, but there's no like route-within-a-route, nodes cna already pick amongst channels to use if they have multiple channels to the destination, this kinda just makes that explicit in a sense 
<t-bast> niftynei: it's only a one-hop thing, so it doesn't have the privacy that rendezvous offers (which hides multiple hops)
<niftynei> i guess the scid was always a hint anyway, also lol
<joostjgr> Routing nodes would try to re-interpret the scid as a truncated pubkey if it doesn't match a real channel. That doesn't have the problem with old readers
<rusty> joostjgr: oh, that's kinda neat.  Skip the first byte though, because x-only.
<cdecker> Honestly I'm not seeing the point, there is very little upside, and has a rather annoying backward compatibility story
<rusty> cdecker: I think it's a good idea, I just think route blinding is better.
<cdecker> Route blinding may have the same issues, but at least it brings new features to the table
<cdecker> Agreed
<joostjgr> Perhaps that can be an upgrade path too. Don't omit or zero out the scid from the invoice, but just put the truncated key there. But for implementations, also implement true pubkey routing to get rid of the scid eventually in the invoice
<roasbeef> they have verrry diff scopes fwiw, just look at the relative descriptions of the additions 
<BlueMatt> ugh, once we start to add hacks to shove the pubkey in the scid the "its easy and clean to just add and move forward" argument starts to fall apart
<roasbeef> size of the*
<niftynei> (rusty's already living in an x-only future ;))
<rusty> "I have seen the future and it is 32 bytes long!"
<roasbeef> but as mentioned above, ppl can start to do this already since it's an e2e upgrade, even intermediate nodes can start to use a global bit to say they like it for w/e reason aprt from end destinations 
<t-bast> rusty: xD
<niftynei> i'm not understanding exactly what this pubkey move gets you
<BlueMatt> roasbeef: "people can deploy this already if they know the feature set of all the relevant nodes" doesnt seem like a strong argument for *why* to do something.
<joostjgr> Would be nice if that global feature bit signals support for pubkey routing via a new tlv field AND via truncated pubkey in the scid
<niftynei> like, you still need to know the connection graph, as channels have to exist between the nodes
<roasbeef> BlueMatt: that isn't a why, it's a how, contrasting the deploymen to of alternatives 
<joostjgr> niftynei: you don't need to reveal your channel points in invoices anymore
<joostjgr> for private chans
<cdecker> Sorry but the truncated node_id in the scid is a clear nack from me
<roasbeef> it's like how ppl just started using stuff like keysend, since only the endpoints needed to update 
<t-bast> On the one hand, it's simple enough to implement, but upgrade will still be slow. And even though it's simple, it's time that could be spent implementing route blinding: the sooner route blinding is included in releases, the sooner the clock starts ticking before enough nodes support it to use it E2E
<BlueMatt> what cdecker said. please dear god no.
<roasbeef> yeh just full on pubkey imo 
<t-bast> joostjgr: have you had many users asking for this?
<joostjgr> I've had likes lol
<BlueMatt> roasbeef: yes, I get that, and people can do whatever experiements they want, but "people could" is not an answer to "this has a lot of the issues of just doing route blinding, and only half the features, so lets just spend our precious little time on the first, not the second"
<roasbeef> t-bast: that's a false dilemma imo, they both have widely diff scopes
<niftynei> but how does a node route to that pubkey if the private channels arent in the route graph?
<t-bast> joostjgr: what we've mostly had is users asking for more privacy by hiding both scid *and* node_id, which is why we worked on route blinding
<roasbeef> niftynei: routing hints? 
<niftynei> i feel like i'm missing something here.
<niftynei> ... but isnt' that exposing your channel points in invoices??
<roasbeef> routing hint w/ pubkey instead of scid? 
<t-bast> roasbeef: it's still time spent somewhere...I don't know how much users want this, it feels like it's not enough for users who asked us for more privacy and too much for those who didn't ask
<niftynei> so you know who they're connected to but not the value of any of those connections/channels etc
<rusty> t-bast: I *definitely* want this for onion_messages, since the whole idea of doxing yourself for a refund is anathema obv.
<roasbeef> niftynei: yeh I think that's the idea, since rn I can grab an invoice from a service and learn info their users for wallets like phoenix/breez 
<t-bast> rusty: by "this" you mean route blinding or pubkey routing?
<joostjgr> t-bast: ofc hiding also the node_id is nicer, but given the choice between hidden channel points or no change, I'd expect them to still see benefit in just hiding channel points? Implementation effort is a factor as well
<rusty> t-bast: route blinding
<t-bast> roasbeef: you don't learn anything for Phoenix users, invoices don't expose short_channel_ids...
<roasbeef> cuz y'all use something custom? 
* vtnerd has quit (Ping timeout: 256 seconds)
<roasbeef> what do you put in the scid field? nothing? 
<t-bast> rusty: ok, agreed, I meant that we didn't get user requests for something that achieves what pubkey routing does. But we did get user requests for the properties route blinding brings.
<cdecker> Any alias for the destination in the scid will do, as long as the forwarding node can interpret it correctly
<t-bast> roasbeef: a fake scid, that only allows our node to map
<roasbeef> as I commented on the PR itself, this is one of those things that really just needs either side to opt into, it can just be a document describing how to do it, and ppl can pick it up or not, since it doesn't require internal updates, just as how keysend was progressilvey rolled out, ppl started to use it, more impls started to add it, etc 
* vtnerd (~vtnerd@50-81-138-206.client.mchsi.com) has joined
<BlueMatt> roasbeef: it sounds like probably only lnd has any interest, so maybe it should just be in a docs folder in lnd somewhere?
<cdecker> roasbeef: just like keysend is still not documented, and we had to reverse engineer it? And now we are getting some reports that cross-implementation keysends aren't working anymore...
<roasbeef> more so trying to make a broader point that only things that modify internal routing needs a high level of coordinating across impls 
<t-bast> roasbeef: it could reasonably be in a `proposals` or `lnip` folder, but there are still a few things that need to be merged in the spec, namely the TLV field used in the onion and the feature bits, right?
<roasbeef> just like how pheonix uses scid place holders and none of us really needed to care 
<roasbeef> cdecker: curious what the issues are since it's so impl, the spec is a sentence more or less (put the preimage in this place)
<t-bast> otherwise we risk incompatibilities on the TLV types and feature bits (clashes with other proposals)
<roasbeef> so simpl* 
<rusty> roasbeef: MPP + keysend should Just Work, right?
<cdecker> Yeah, iirc it was something else, but still made me re-read the keysend impl to make sure
<roasbeef> rusty: as in AMP? or something else 
<niftynei> the scid placeholder isn't an exact analog, as it's just repurposing an existing spec field...
<roasbeef> but backing up I don't feel super strongly about this, just trying to make a meta point about the level of coordination required for certain features/extensions 
<cdecker> As in "include the preimage in the TLV payload" 
<cdecker> Whatever the name for that one was
<rusty> roasbeef: yeah, split payments.  Did you do some combination of the preimages in that case, or do they all contain the preimage?
<cdecker> Ah wait, no MPP as in "just reuse the same hash"
<roasbeef> oh solution for that is AMP rusty, we have a path to phase out keysend in lnd as is since multi-path keysend is AMP 
<roasbeef> but ok should we move to other things...?
<rusty> roasbeef: I think that's where the incompatibility is, we try to MPP using the same preimage...
<roasbeef> rusty: ah yeh it's only a single path thing purposefully, to have ppl use AMP by necessity 
<cdecker> Ok, so that might be our issue, need to disable mpp for keysend
<cdecker> (or we keep it on, and just let lnd fail to receive...)
<t-bast> I think there's something coming from this discussion that would be interesting to settle: do we want to move forward with a proposals/lnip format for some features in the spec? As I pointed out above, if we do that, we still need to merge in the main spec body TLV types and feature bits
<rusty> cdecker: or implement AMP?  Shouldn't be that hard, assuming we can derive spec from code?
<roasbeef> yeh they're mutually exclusive for us (keysend and mpp), and we're actually moving to make mpp rewuired for _all_ payments in our next release (0.12 drops "soon") 
<t-bast> TBH if keysend is for donations, it's surprising that it can't work with a single-part...
<roasbeef> t-bast: imo yeh, and I think it would give ppl some clarity to working on higher level wwallet type things, many of which just ended up making their own spec repos 
<t-bast> (small donations)
<roasbeef> it _only_ works w/ single part 
<cdecker> Mhz, I hate having multiple things that achieve the same (see route blinding and pubkey based routing), so also implementing AMP (with XORed preimage) is ugh
<roasbeef> cdecker: what else does what AMP does? or referring to something else? 
<rusty> cdecker: if iut's just xor, that's easy to impl.
* niftynei needs a glossary of lightning routing shorthands
* __gotcha (~Thunderbi@plone/gotcha) has joined
<cdecker> Yes, the current MPP implementation reusing the same hash works perfectly well, and with schnorr we get decorrelation
<roasbeef> yeh ppl hear "additive secret sharing" but it's just xor, you can do a lot w.r.t how you derive the shares on the sender side, but receiver just xors till they can start to pull and eveyrthing is there 
<roasbeef> works well, but re-uses the same payment hash so the route cna be correlated
<BlueMatt> roasbeef: what exactly do you mean "require mpp for all payments" - do you mean require the ability to send outbound mpp for outbound payments, or require the mpp fields in single-path payments?
<roasbeef> "until schnorr" is a vague time frame 
<cdecker> Keysend + MPP is more composable than AMP (which requires the sender to generate the preimage if I'm not mistaken)
<roasbeef> BlueMatt: we require the payment addr field for single-path payments 
<roasbeef> cdecker: keysend still requires the sender to generate tho? 
<roasbeef> idk how you can combine something that's spontaenous w/ something that isn't 
<rusty> Yeah, IMHO if you're doing keysend you might as well xor the preimages.
<cdecker> Exactly, that's what I mean by composability. Keysend generates secret on sender (invoice generation) while MPP takes care of transport
<roasbeef> so new invoices lnd generates will rquire the payment addr, but if there'sr an old unexpired invoice with the optional bit set, then we'll still accept that (is the way we've impl'd it) 
<roasbeef> but should we get to another topic in the remaining time? 
<cdecker> Afaik there is no AMP variant that works without the sender generating the preimag
<BlueMatt> roasbeef: ah, ok. interesting. I presume mostly to get payment_secrets?
<roasbeef> cdecker: yes, that's how it works, the sender genrates the preimage, if you can do dlog stuff the receiver can include a point and the sender uses that to force them to also reveal a secret 
<roasbeef> BlueMatt: yeh the payment addr, which would've partially thwarted that CVE we had, since the party couldn't probe any of our payments w/ that value 
<roasbeef> or w/e it's called lol
<BlueMatt> :+1:
<roasbeef> it also makes ppl doing "zero value invoice stuff" more secure 
<cdecker> Ok, seems nobody cares about having multiple transport mechanisms, so I'll shut up
<roasbeef> we're also gonna require static remote key as well for funding 
<roasbeef> LTU in the last bit? 
<roasbeef> lnd has a PR for #804
<roasbeef> but seems now CL will start to end an actual error if the reply isn't well formed rusty ? 
<t-bast> Let's talk a bit about long-term udpates indeed? And we'll talk about channel_range_queries next time?
<roasbeef> that caused us to mark a channel as borked iirc since it wasn't clear to lnd why the error was sent (since we don't have proper error codes) 
<roasbeef> t-bast: sure
<roasbeef> just wanted to drop that msg lol
<t-bast> #topic Offers
<t-bast> #link https://github.com/lightningnetwork/lightning-rfc/pull/798
<t-bast> rusty, maybe you want to tell us what reviewers should focus on for a first pass on the PR?
<rusty> roasbeef: we fixed latest release to allow that overlap.  before that, yeah, we would have complained.
<t-bast> since it's 1600 lines long, we definitely won't be able to grasp all of it during this meeting :)
<rusty> t-bast: OK, yeah, I have an implementation (as you can tell by all the spec churn!)
<rusty> Well, there are really two modes.  Normal is offer -> invoice_request -> invoice.  
<rusty> A "send_invoice" offer is offer -> invoice.  This is for ATMs and refunds, where you really are offering to send money, not recv it.
<rusty> Feedback welcome esp on stuff like the format (it's bech32 without the csum, of the same formatting we use on the wire)
* bitconner slips in the room, sits down quitely in closest seat
<rusty> Skipping the csum is actually kinda nasty, since std bech32 APIs don't give you that.  Not sure if I should simply shrug and put it back?
<rusty> (I removed it because there's speculation on changing csum calc anyway)
<roasbeef> why not just drop bech32 all together? 
<roasbeef> since it basically does nothing w/ such a large input string
<roasbeef> just use base32 encoding or something if you want the same look and feel, and can add a manual prefix if you want 
<rusty> roasbeef: yeah, I just use the encoding, since we already support it.
<roasbeef> ah ok so just the char set? tho the whole 5 bit word thing is annoying, and iirc a few impls had a bug related to it? 
<rusty> roasbeef: zbase32 just adds YA standard.
<bitconner> roasbeef: are you suggesting prefix||base32(tlv(data), sig)?
<roasbeef> ehh I guess, but it's just extra complexity 
<roasbeef> bitconner: yeh
<rusty> roasbeef: yeah, but that's true of any encoding unless we do hex :)
<rusty> So, yeah, I took the low road.
<roasbeef> idk base32 vanilla is waaay smaller code wise than bech32 itself (like even just skipping the "re-wording" step adds a lot), but hey I guess it's already there 
<rusty> It's TLV, which is good and bad.  bad because there are some compulsory fields which now are in the spec rather than the decoder logic.
<BlueMatt> if we're gonna do bech32, just include the checksum. its worthelss but extra few chars is basically free.
<roasbeef> just fees like over kill 
<rusty> BlueMatt: sure, once they decide what csum to use!
<roasbeef> and we get no benefits from it other than "we're using it too!" 
<lndbot> <johanth> must note (as for invoices) that is not “real” bech32 tho
<rusty> Anyway, the other tech choice is 32-byte pubkeys.  I implemented with 33 byte and ecdsa, and the flip to schnorr and x-only took about a day.  Pretty trivial.
<lndbot> <johanth> which can be confusing, since a bech32 lib won’t parse it
<BlueMatt> rusty: right lol. tbh this seems like a "person who writes the spec choses, everyone else shuts up cause it doesnt matter and is otherwise shed-painting" kinda issue :p
<t-bast> BlueMatt: true dat :)
<rusty> Yeah, so the rest is pretty simple.  There's blinded paths in there, ofc.
<rusty> The main complexity is the recurrence stuff, but it's really a killer feature IMHO.
<roasbeef> so in the past, ppl didnt like HORNET since they felt it allowed arbitrary data transfer or w/e if not limited or metered, but doesn't this allow the same thing with the onion messages? 
<roasbeef> IRC over LN? ;) 
<rusty> roasbeef: yeah, but this is at least super cheap (1 way, no reply).
<t-bast> roasbeef: I think the difference is even with recurrence, it will likely be one message per day / week / month, so hornet woulnd't bring anything since session_keys would need to be reestablished every time (it would be slower than Sphinx)
<BlueMatt> roasbeef: having a spec to do useful things doesnt imply that people *should* run their vpn over it, and it provides useful things on its own, so, :shrug:
<t-bast> People can mis-use it of course though
<roasbeef> BlueMatt: what i'm saying is HORNET gives us all that blinded paths+offers does plus more, but before ppl didn't like the messaging aspect, but offers rolls something new for that purpose 
<roasbeef> t-bast: yeh that's what I mean 
<roasbeef> also fwiw re recurrence, with AMP that can be implemtned in a way that doesn't require a manual invoice fetch (over the network itself?) each time they end user wants to pay, it's more of a cron job at the end user after it has the initial static inovice 
<t-bast> But offers optimizes for the normal path (the way it's intended to be used). Using HORNET there would be less efficient (and more code and complexity to implement)
<roasbeef> unclear if more code from my PoV 
<BlueMatt> right, sure, I mean I dont care about HORNET vs rusty's work or whatever, just noting that the reason this is more of a "yep, sounds good" is that it actually provides all the useful features that people wanted to see, instead of just saying "cool, look messaging protocol, we can add features to use it later"
<roasbeef> since it builds on sphinx 
<rusty> Implementing onion_message is pretty simple.  Was about 200 lines.
<t-bast> roasbeef: blinded path re-uses the existing sphinx scheme without changing anything, it was < 100 loc in eclair
<rusty> HORNET creates (one-way) circuits, which are not useful.
<roasbeef> nah they're bi-directional 
* __gotcha has quit (Ping timeout: 264 seconds)
<lndbot> <johanth> t-bast: isnt anything < 100 loc in scala? :P
<roasbeef> and require no stroage at the intermediaries to process 
<rusty> roasbeef: no, you gotta create them in a loop?
<roasbeef> unlike how we all need to maintain these ciruits 
<rusty> roasbeef: sure, onion_message doesn't have circuits either.
<roasbeef> a single round trip sets up the forward and backwards path, the backwards path can even be a diff route entirely 
<t-bast> The way I understood it is I thought you were pusing for HORNET for something like LSAT and high-frequency tiny payments, but apart from that I'm not sure why
<t-bast> johanth: damn, I'm exposed :)
<roasbeef> mhmm but also we could use this to remove the circuit requiremetns for all reg HTLC forwarding as well 
<roasbeef> t-bast: I mean if you want to use it to fetch an invoice from someone, you can do that as well 
<rusty> roasbeef: yeah, except mpp means you set up circuits and throw them away again.  Why?
<rusty> We just don't seem to have the setup-once-use-many pattern which fits HORNET.
<roasbeef> "use many" is as many as the expiry of the forwarding packets allow, you're right in that there's the extra RTT across the entire route to set it up the first time 
<bitconner> well if you end up requesting invoices frequently, that pattern would fit
<BlueMatt> but only if its frequently enough that nodes on the path would come and go?
<t-bast> depends how frequently, how long are circuits supposed to live in HORNET? clearly less than a day, right?
<BlueMatt> (ie, per-hour, not per-month?)
<niftynei> iiuc HORNET would be useful for lots of payments between two parties in a short timeframe
<roasbeef> but also the burden of the time isn't felt on the intermediate nodes since they don't need to write circuit state for it, the packets carry all the info 
<bitconner> they're routing nodes, we can expect pretty high uptime
<niftynei> so for a HFT application etc
<roasbeef> t-bast: depends really, it can be 1 minute if you want 
<roasbeef> if we're doing it now, would maybe use blockheight as expiry in the packet or something 
<niftynei> (HFT-> high freq trading)
<t-bast> roasbeef: yeah but it's the higher threshold that's interesting there: what's the maximum duration before you need to reset them for security>
<rusty> bitconner: I expect you may try several paths to get the invoice successfully, and maybe several more to make the payment.  Reuse of a single path would be < 1.
<roasbeef> also note you can make a HORNET session to just request invoice or w/e and end up using some other path entirely for the actual payment trasnfer 
<t-bast> niftynei: that was my understanding as well
<roasbeef> can think of it as control+payment sessions 
<bitconner> rusty: isn't it the case that you may need to request several path prefixes for rendezvous style payments?
<rusty> roasbeef: yeah, but you have liveness proof if you get a successful msg though, which makes it a good candidate for payment.
<bitconner> s/prefixes/sufficex
<roasbeef> mhmm just pointing out the flexibility in usage 
<lndbot> <johanth> paving the way for “stuckless payment” types also
<roasbeef> in that you can use nodes that've updated to requiest invoices or w/e, while still using the greater network for normal payment transfer 
<rusty> bitconner: if they want that, they'd have multiple paths in the invoice they send.
<bitconner> and that's all going to fit in one onion?
<roasbeef> old email on the topic which maybe will provide ppl w/ more context: https://lists.linuxfoundation.org/pipermail/lightning-dev/2018-November/001524.html 
<rusty> bitconner: ... 64k?  Sure.
<bitconner> ah okay, was thinking onion payload as the limit
<roasbeef> other stuff I was reminded about from that email: can be used for a pyemtn level ACK (somethign we takced abou tin the past), and also to let nodes rotate onion keys which adds forward secrecy 
<niftynei> it really seems like it's hard to determine what's the most efficient here without a solid usecase in mind
<t-bast> What's unclear to me is how costly is the circuit setup in HORNET. I'd like to see the latency it adds for a wallet user that launches his wallet to make a payment, compared to the offers way. My assumptions was that it would add to much latency for the UX to be reasonable (by extrapolating from the HORNET paper figures), but I may be wrong
<rusty> Since chains of htlcs need state, implementing HORNET for it seems like a waste.
<roasbeef> t-bast: good Q, I think processing is negliible (and actually faster since it' sjust symmetric crypto) so you could just estimate based on RTT guess and rroute length 
<niftynei> wallet user != HFT exchange rails
<rusty> Implementing HORNET for messages makes sense, but then they're usually single-use (and we don't really want to optimize streaming) so why bother?
<roasbeef> rusty: lots of other benefits like faster forwarding in general, forawrd secrecy, payment level ack, etc, etc 
<rusty> roasbeef: none of those are hornet, dude!
<roasbeef> what's the issue with the sessions having a low lifetime? 
<roasbeef> ofc they are? it enables all those things 
<rusty> roasbeef: no.  Payment acks can be done today.
<roasbeef> see that email above, timely now that it's 2 years old 
<roasbeef> lots of things can be done today, this is one thing that gives us several things 
<roasbeef> what would you use for an ack today? a reverse HTLC or something? 
<rusty> "A notable change is that each node no longer needs to maintain per-circuit state as we do now with Sphinx"  Err, yeah, HTLCs need state.
<t-bast> TBH I think the only way we'll settle this discussion about hornet vs no hornet is by having a PoC. If LL wants to spend time on it, I volonteer to do an interoperability Eclair PoC as well and we'll see what it brings.
<rusty> roasbeef: every node already has to remember the incoming<->outgoing HTLC circuit.  It's trivial to route msgs back based on that.,
<roasbeef> yeh HTLCs need state, but that's a diff layer in the stack, that's purely talking about the state needed to forward an htlc and it's errro/settle 
<roasbeef> I think yuo're getting caught up on the state thing rusty
<rusty> roasbeef: yeah, I am.  We have state, we can't get rid of it.  Let's not add something which has that as its entire purpose.
<roasbeef> there're like 5 other things it adds beyond that, the state things is correct in that it's about session state, but correc tyou need htlc state to process them on-chain 
<roasbeef> that's not the entire purpose, that's one aspect it adds amongst several 
<roasbeef> consider it a bonus lol 
<roasbeef> the point there is that if ppl want to use it for fetching invoices or w/e, it does't add any additional state burden to the node processing the session 
<rusty> roasbeef: sure, but since the messaging tends to be one-shot anyway, we don't even need hornet.
<bitconner> which meaens it's probably more DOS resistant
<bitconner> since the sender maintains all circuit state
<rusty> So we have two things: an HTLC which is ungodly heavy and requires state.  And messages which are oneshot.
<roasbeef> it can be used one shot, or few shots, it's just more flexible 
<roasbeef> g2g tho
* BlueMatt still doesnt see a lot of reason to bother optimizing for >1 shot - too few cases that are multi-messages-within-reasonable-timeframe
<BlueMatt> roasbeef, before you go quick q: a) can I presume that folks would be open to a post-covid (lets say fall-ish. 2021) physical spec meeting and b) that no one has yet started organizing one and c) that rusty would only threaten to not, but would still come if its in ny?
<rusty> BlueMatt: oh, I like NY.
<t-bast> BlueMatt: Yay!!
<roasbeef> dunno what the future holds, but sounds like a good idea, has been too long since we've all met up 
<bitconner> DOS thing imo is pretty important as there's no natural ratelimiting for messages like there is for sats: commitment slots, fees, capital, etc
<BlueMatt> rusty: ah, I see you *do* love the inside of small metal tubes :p!
<bitconner> BlueMatt: absolutely!
<rusty> bitconner: yeah, I tried to make onion_messages as light as possible, but we still may end up with some anti-spam pay2play or simple ratelimiting.
<bitconner> t-bast: agree some PoC's would probably be useful here. also depends on if others would be willing to block offers on those numbers or if we'd just proceed with onion msgs anyway..
<bitconner> (assuming they can be completed in a reasonable timeframe)
<rusty> bitconner: TBF, if we get HORNET for ~free, I'd not block it for use in onion_message.  I'll have to look at it.
<rusty> (I don't have a good use for HORNET, but maybe someone will...)
<bitconner> not sure how we get HORNET by only using sphinx tho..
<BlueMatt> yea, dos is maybe the only super-compelling reason, aside from a bunch of pie-in-the-sky kinda thinking.
<rusty> bitconner: yeah, it's a bit more complex.  We need to re-add somef ields we removed.
<t-bast> I'm mostly afraid of complexity + latency when setting up circuits (especially on mobile wallets), but that's what PoCs are for, we'll see when we have code running
<rusty> BlueMatt: doesnt' win on DoS, sicne onion_message is also stateless.
<BlueMatt> ah, ok, I mis-parsed your earlier comment, rusty.
<rusty> (Def beats (ab)using HTLCs for msg transport though)
<BlueMatt> dear god yes.
<bitconner> t-bast: i think the setup latency isn't as bad as one might expect, definitely faster than payments bc you don't need roundtrips to sign states, but a concrete impl will confirm/deny
<t-bast> Great thanks everyone, let's end it for today! Great discussions.
<t-bast> #endmeeting
<t-bast> bitconner: I volunteer to experiment with it, will take a bit of time of course, but we'll see what we get
* BlueMatt will look into thinking about meeting. obviously nothing to propose until its looked into more and we all have clarify of travel safety/ability. no promises.
<t-bast> Thanks BlueMatt, it would be great to have such an event!
<joostjgr> Preventing abuse of htlcs. It is something that I hope will get some attention before moving on to extending the protocol with new functionality
<BlueMatt> since no one screamed, I'll think about new york, if it doesnt work out I'll try to hand it off to someone else. It was ~20 people last time, right?