HttpContactPointBootstrap always dead after probing timeout

[Roiocam]

Explain

In the cluster bootstrapping, we will create a child actor for handling HTTP probing, this actor will use the config probingFailureTimeout as the deadline time:

pekko-management/management-cluster-bootstrap/src/main/scala/org/apache/pekko/management/cluster/bootstrap/internal/HttpContactPointBootstrap.scala

Lines 99 to 103 in 7ed2b5b

	/**
	* If probing keeps failing until the deadline triggers, we notify the parent,
	* such that it rediscover again.
	*/
	private var probingKeepFailingDeadline: Deadline = settings.contactPoint.probingFailureTimeout.fromNow

At the same time, we are using the same configuration probingFailureTimeout as probing future timeout too.

pekko-management/management-cluster-bootstrap/src/main/scala/org/apache/pekko/management/cluster/bootstrap/internal/HttpContactPointBootstrap.scala

Lines 113 to 116 in 7ed2b5b

	log.debug("Probing [{}] for seed nodes...", probeRequest.uri)
	val reply = http.singleRequest(probeRequest, settings = connectionPoolWithoutRetries).flatMap(handleResponse)
	val afterTimeout = after(settings.contactPoint.probingFailureTimeout, context.system.scheduler)(replyTimeout)
	Future.firstCompletedOf(List(reply, afterTimeout)).pipeTo(self)

There is only one way to handle these timeouts and deadlines, As you can see, because of the existence of a deadline, the else logic will never be executed.

pekko-management/management-cluster-bootstrap/src/main/scala/org/apache/pekko/management/cluster/bootstrap/internal/HttpContactPointBootstrap.scala

Lines 118 to 127 in 7ed2b5b

	case Status.Failure(cause) =>
	log.warning("Probing [{}] failed due to: {}", probeRequest.uri, cause.getMessage)
	if (probingKeepFailingDeadline.isOverdue()) {
	log.error("Overdue of probing-failure-timeout, stop probing, signaling that it's failed")
	context.parent ! BootstrapCoordinator.Protocol.ProbingFailed(contactPoint, cause)
	context.stop(self)
	} else {
	// keep probing, hoping the request will eventually succeed
	scheduleNextContactPointProbing()
	}

Discuss

I think we may need two configurations for deadline and timeout. In such cases, when there is network latency for the contact point node, theHttpContactPointBootstrap actor does not need to be frequently destroyed and created. At least we have some buffer time.

wdyt @pjfanning @He-Pin @mdedetrich @samueleresca

[He-Pin]

Maybe we can add something timefactor for it, eg what we are using in the testkit？

[He-Pin]

Or we can introduce a connect timeout and start the tick after connected

[Roiocam]

eg what we are using in the testkit？

Which testkit did you mention?

HttpContactPointBootstrap is very lack of test cases, so that is why this problem hasn't been discovered for so long?

pekko-management/management-cluster-bootstrap/src/test/scala/org/apache/pekko/management/cluster/bootstrap/internal/HttpContactPointBootstrapSpec.scala

Lines 22 to 29 in 7ed2b5b

	class HttpContactPointBootstrapSpec extends AnyWordSpec with Matchers {
	"HttpContactPointBootstrap" should {
	"use a safe name when connecting over IPv6" in {
	val name = HttpContactPointBootstrap.name(Host("[fe80::1013:2070:258a:c662]"), 443)
	ActorPath.isValidPathElement(name) should be(true)
	}
	}
	}

[He-Pin]

I mean some kind of factor, any way , send a pr when you have time, I will review it.

[pjfanning]

@Roiocam @mdedetrich I haven't reread this in detail but saw it while searching for other issues - is this an issue that we could look at?