Lots of Subprotocol errors

[kshinn]

I'm seeing a lot of peers getting removed because of a subprotocol error.

infura:main 213.136.76.42: eth.63,eth.62 +44ms
infura:main 213.136.76.42 SUBPROTOCOL_ERROR +7ms
infura:main 46.101.160.39: eth.63,eth.62 +10ms
infura:main 46.101.160.39 SUBPROTOCOL_ERROR +16ms
  infura:main 199.247.3.3 SUBPROTOCOL_ERROR +7ms
  infura:main 89.34.99.3: eth.62,eth.63,par.1,par.2,pip.1 +65ms
  infura:main 35.234.97.198: eth.62,eth.63,par.1,par.2,par.3,pip.1 +27ms
  infura:main 51.68.172.123 SUBPROTOCOL_ERROR +14ms
  infura:main 51.140.82.22 SUBPROTOCOL_ERROR +3ms
  infura:main 159.65.200.185: eth.63,eth.62 +39ms
  infura:main 137.117.158.210 SUBPROTOCOL_ERROR +22ms
  infura:main 35.234.97.198 undefined +82ms
  infura:main 37.139.23.36: eth.63,eth.62 +11ms
  infura:main 176.223.131.39: eth.63 +85ms
  infura:main 37.139.23.36 SUBPROTOCOL_ERROR +168ms
  infura:main 41.185.97.129: eth.63,eth.62 +2ms
  infura:main 176.223.131.39 SUBPROTOCOL_ERROR +1ms
  infura:main 41.185.97.129 SUBPROTOCOL_ERROR +688ms
  infura:main 206.189.68.191 DISCONNECT_REQUESTED +3s
  infura:main 89.34.99.3 DISCONNECT_REQUESTED +1s
  infura:main 159.65.200.185 SUBPROTOCOL_ERROR +95ms

I've checked the handshake and it looks like the library is doing that properly on the peer connect. I suspect we are seeing a number of missing nodes because of these protocol mismatches.

[kshinn]

The ethereumjs code seems to have a problem parsing the body and assumes the protocol is out of spec or an unknown variant.

[egalano]

It'd be interesting to see as much information as possible from the client connection that is reporting the subprotocol error. Strange custom clients, older versions maybe? The DISCONNECT_REQUESTED could also be coming from a peer that does not have available peering slots

[kshinn]

From what I can tell the disconnect request are part of the normal protocol. I've been seeing them after a successful status call. There is an explicit peering table full error I see as well. There may be value in tracking these in the db as well.

…

On Fri, Dec 28, 2018, 11:21 PM E.G. Galano ***@***.*** wrote: It'd be interesting to see as much information as possible from the client connection that is reporting the subprotocol error. Strange custom clients, older versions maybe? The DISCONNECT_REQUESTED could also be coming from a peer that does not have available peering slots — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACEUXoU_QiNvsAbGX29xhicb9xKSgIEks5u9xgMgaJpZM4ZkcVu> .

[kshinn]

Got a little more information. The reason we are getting a protocol mismatch looks like a network ID mismatch between the Peer status message and the devp2p status message. It ends up throwing an assertion message.

Devp2p client message:

  infura:main 52.3.158.184: eth.62,eth.63 +216ms
  infura:p2p [ <Buffer 3f>,
  infura:p2p   <Buffer 01>, <-- Mainnet
  infura:p2p   <Buffer 04 00 00 00 00>,
  infura:p2p   <Buffer d4 e5 67 40 f8 76 ae f8 c0 10 b8 6a 40 d5 f5 67 45 a1 18 d0 90 6a 34 e6 9a ec 8c 0d b1 cb 8f a3>,
  infura:p2p   <Buffer d4 e5 67 40 f8 76 ae f8 c0 10 b8 6a 40 d5 f5 67 45 a1 18 d0 90 6a 34 e6 9a ec 8c 0d b1 cb 8f a3> ] +195ms

Looking at the wire protocol the first byte is the protocol version (63)
The second is the NetworkID. The 01 above is Mainnet, what the devp2p client is expecting.

Peer status

infura:p2p [ <Buffer 3f>,
  infura:p2p   <Buffer 04>, <-- Rinkby
  infura:p2p   <Buffer 65 51 35>,
  infura:p2p   <Buffer de ed 9b 12 f5 b3 bd bb 51 a9 e6 65 04 00 37 6d 11 0a 90 76 09 96 5e cc d6 b8 61 f0 b7 ea 53 03>,
  infura:p2p   <Buffer 63 41 fd 3d af 94 b7 48 c7 2c ed 5a 5b 26 02 8f 24 74 f5 f0 0d 82 45 04 e4 fa 37 a7 57 67 e1 77> ] +2ms
  infura:main 52.3.158.184:30303 (Peer Error) AssertionError [ERR_ASSERTION]: NetworkId mismatch: 01 / 04 +1ms
  infura:main 52.3.158.184 SUBPROTOCOL_ERROR +2ms

[kshinn]

Posted for reference

No.	hex	Network	Notes
0	0x00	Olympic	Ethereum public pre-release testnet
1	0x01	Frontier	Homestead, Metropolis, the Ethereum public main network
1	0x01	Classic	the (un)forked public Ethereum Classic main network, chain ID 61
1	0x01	Expanse	an alternative Ethereum implementation, chain ID 2
2	0x02	Morden	the public Ethereum testnet, now Ethereum Classic testnet
3	0x03	Ropsten	the public cross-client Ethereum testnet
4	0x04	Rinkeby	the public Geth PoA testnet
8	0x08	Ubiq	the public Gubiq main network with flux difficulty chain ID 8
42	0x2A	Kovan	the public Parity PoA testnet
77	0x4D	Sokol	the public POA Network testnet
99	0x63	Core	the public POA Network main network
100	0x64	xDai	the public MakerDAO/POA Network main network
401697	0x62121	Tobalaba	the public Energy Web Foundation testnet
7762959	0x7F740F	Musicoin	the music blockchain
61717561	0x3ADBC39	Aquachain	ASIC resistant chain