Use dynamic publish requests limits based on subscriptions and latency

[svanharmelen]

This PR tries to incorporate both publishing strategies as described in the spec:

Client strategies for issuing Publish requests may vary depending on the networking delays between the Client and the Server. In many cases, the Client may wish to issue a Publish request immediately after creating a Subscription, and thereafter, immediately after receiving a Publish response.

In other cases, especially in high latency networks, the Client may wish to pipeline Publish requests to ensure cyclic reporting from the Server. Pipelining involves sending more than one Publish request for each Subscription before receiving a response. For example, if the network introduces a delay between the Client and the Server of 5 seconds and the publishing interval for a Subscription is one second, then the Client shall issue Publish requests every second instead of waiting for a response to be received before sending the next request.

It follows the first strategy by sending a publish request right after a subscription is created and immediately responding to a publish response with a new request if the number of inflight publish requests is less then the currently active subscriptions times two (so that there is always one available at the server while the client is processing the other one). So this (active subscriptions times two) defines the min_publish_requests.

In most low-latency environments this is enough to make sure the server always has a publish request available. Even if the server had a temporary performance issue (e.g. due to a backup or other external process) and wasn't able to send the publish response within the specified window, it is able to "catch up" due to the fact that the client will immediately send a new publish request instead of waiting for a certain interval to have passed.

But in case the network between your client and the sever has a high latency, this approach alone is not enough to make sure there are enough publish requests available to the server to be able to send continuous publish responses. So in addition to the above we also keep track of the current network latency by measuring how long the round-trip of the keep-alive messages take. We divide the measured round-trip time by the lowest publish interval of the active subscriptions and multiple that by the min_publish_requests to define the max_publish_requests.

Changes in the max_publish_requests will be handled by checking at every publish interval. When the timer ticks it will check if the total number of in flight requests in below the max_publish_requests and if so, send an additional publish request.

We checked this approach against different types of servers and with different types of networks (with fluctuating latency) and the combined strategy was able to adjust itself in all tested situations so that there always was a steady flow of publish requests and responses.

[einarmo]

This solution seems good to me, as we discussed. The "big number" in the sender queue is maybe a little questionable, but the reasoning is sound. I'll think about a better solution in the future, maybe.