dlb connection flapping when ICMP is blocked.

[alfs]

While preparing a virtual node for 5GENESIS/Berlin, I experienced frequent ssh-tunnel restarts (every few seconds the tunnel was terminated.).

I think this is due to dlb doing connection resets, and from what I can tell from the log (below) it's due to non-ICMP responses. Correct?

In the particular environment (qemu) the network NAT function works well for TCP and UDP, however ICMP is not supported.

I would ask that

• Connection tests be less aggressive (does it need to be so frequent (every second?) If ICMP is blocked, firewalled, etc, the curl tests will also consume data traffic while doing the throughput estimation. At the least a cool-down period would be reasonable.
• TCP connection test would trump ICMP tests - or is there some specific reason that both must function?

(06.12.2018 07:45:28:743) [INFO]: TcpChecker.cpp handleSuccess 263 Connection establishment time 0.009058
(06.12.2018 07:45:28:743) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 1
(06.12.2018 07:45:28:804) [INFO]: ThroughputChecker.cpp handleCurlMsg 240 Throughput 9874.9kB/s 10.0.2.15 eth0 11499 736 0.001239
(06.12.2018 07:45:28:804) [INFO]: Address.cpp updateQuality 1178 Changed quality from 0 to 2 for 10.0.2.15/24
(06.12.2018 07:45:34:726) [ERROR]: IcmpChecker.cpp onTimeoutExpired 120 Did not receive any ICMP reply before timeout on 10.0.2.15 eth0
(06.12.2018 07:45:34:726) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 0
(06.12.2018 07:45:34:751) [INFO]: TcpChecker.cpp handleSuccess 262 Connection established on interface eth0 (10.0.2.15)
(06.12.2018 07:45:34:751) [INFO]: TcpChecker.cpp handleSuccess 263 Connection establishment time 0.007557
(06.12.2018 07:45:34:751) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 1
(06.12.2018 07:45:34:812) [INFO]: ThroughputChecker.cpp handleCurlMsg 240 Throughput 15538.9kB/s 10.0.2.15 eth0 11462 736 0.000785
(06.12.2018 07:45:34:812) [INFO]: Address.cpp updateQuality 1178 Changed quality from 0 to 2 for 10.0.2.15/24
(06.12.2018 07:45:40:569) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:40:569) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:40:749) [ERROR]: IcmpChecker.cpp onTimeoutExpired 120 Did not receive any ICMP reply before timeout on 10.0.2.15 eth0
(06.12.2018 07:45:40:750) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 0
(06.12.2018 07:45:40:797) [INFO]: TcpChecker.cpp handleSuccess 262 Connection established on interface eth0 (10.0.2.15)
(06.12.2018 07:45:40:797) [INFO]: TcpChecker.cpp handleSuccess 263 Connection establishment time 0.007786
(06.12.2018 07:45:40:797) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 1
(06.12.2018 07:45:40:863) [INFO]: ThroughputChecker.cpp handleCurlMsg 240 Throughput 10827.4kB/s 10.0.2.15 eth0 11499 736 0.00113
(06.12.2018 07:45:40:863) [INFO]: Address.cpp updateQuality 1178 Changed quality from 0 to 2 for 10.0.2.15/24
(06.12.2018 07:45:46:388) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:46:396) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:46:396) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:46:401) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:46:401) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:46:429) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:46:429) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:46:675) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:46:676) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:46:789) [ERROR]: IcmpChecker.cpp onTimeoutExpired 120 Did not receive any ICMP reply before timeout on 10.0.2.15 eth0
(06.12.2018 07:45:46:789) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 0
(06.12.2018 07:45:46:833) [INFO]: TcpChecker.cpp handleSuccess 262 Connection established on interface eth0 (10.0.2.15)
(06.12.2018 07:45:46:833) [INFO]: TcpChecker.cpp handleSuccess 263 Connection establishment time 0.02526
(06.12.2018 07:45:46:841) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 1
(06.12.2018 07:45:46:921) [INFO]: ThroughputChecker.cpp handleCurlMsg 240 Throughput 6233.44kB/s 10.0.2.15 eth0 11494 736 0.001962
(06.12.2018 07:45:46:921) [INFO]: Address.cpp updateQuality 1178 Changed quality from 0 to 2 for 10.0.2.15/24
(06.12.2018 07:45:52:831) [ERROR]: IcmpChecker.cpp onTimeoutExpired 120 Did not receive any ICMP reply before timeout on 10.0.2.15 eth0
(06.12.2018 07:45:52:832) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 0
(06.12.2018 07:45:52:859) [INFO]: TcpChecker.cpp handleSuccess 262 Connection established on interface eth0 (10.0.2.15)
(06.12.2018 07:45:52:859) [INFO]: TcpChecker.cpp handleSuccess 263 Connection establishment time 0.012327
(06.12.2018 07:45:52:860) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 1
(06.12.2018 07:45:52:944) [INFO]: ThroughputChecker.cpp handleCurlMsg 240 Throughput 10974kB/s 10.0.2.15 eth0 11489 736 0.001114
(06.12.2018 07:45:52:944) [INFO]: Address.cpp updateQuality 1178 Changed quality from 0 to 2 for 10.0.2.15/24
(06.12.2018 07:45:56:968) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:56:968) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:58:160) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:58:161) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:58:193) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:58:193) [ERROR]: NetlinkHandlers.cpp handleNewConntrack 556 Missing required information
(06.12.2018 07:45:58:849) [ERROR]: IcmpChecker.cpp onTimeoutExpired 120 Did not receive any ICMP reply before timeout on 10.0.2.15 eth0
(06.12.2018 07:45:58:849) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 0
(06.12.2018 07:45:58:879) [INFO]: TcpChecker.cpp handleSuccess 262 Connection established on interface eth0 (10.0.2.15)
(06.12.2018 07:45:58:879) [INFO]: TcpChecker.cpp handleSuccess 263 Connection establishment time 0.008684
(06.12.2018 07:45:58:880) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 1
(06.12.2018 07:45:58:954) [INFO]: ThroughputChecker.cpp handleCurlMsg 240 Throughput 11364.7kB/s 10.0.2.15 eth0 11481 736 0.001075
(06.12.2018 07:45:58:954) [INFO]: Address.cpp updateQuality 1178 Changed quality from 0 to 2 for 10.0.2.15/24
(06.12.2018 07:46:04:868) [ERROR]: IcmpChecker.cpp onTimeoutExpired 120 Did not receive any ICMP reply before timeout on 10.0.2.15 eth0
(06.12.2018 07:46:04:868) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 0
(06.12.2018 07:46:04:886) [INFO]: TcpChecker.cpp handleSuccess 262 Connection established on interface eth0 (10.0.2.15)
(06.12.2018 07:46:04:886) [INFO]: TcpChecker.cpp handleSuccess 263 Connection establishment time 0.006926
(06.12.2018 07:46:04:886) [INFO]: GroupsManager.cpp updateNumConnected 802 Cur connected links 1
(06.12.2018 07:46:04:939) [INFO]: ThroughputChecker.cpp handleCurlMsg 240 Throughput 25279.5kB/s 10.0.2.15 eth0 11474 736 0.000483
(06.12.2018 07:46:04:939) [INFO]: Address.cpp updateQuality 1178 Changed quality from 0 to 2 for 10.0.2.15/24

[kristrev]

• The reason the test is run so often, is that background traffic is not detected on Monroe nodes. Background traffic is detected by using nfacct + iptables-rules, so it is just a matter of installing the nfacct-package + adding the correct firewall rules. If background traffic is detected and is above a certain threshold, connectivity checks/throughput are not run.
• ICMP can be disabled by adding "do_icmp":false to the dlb-config (main object).