HttpRemoteTaskRunner enhancements

[jtuglu1]

Description

Clone of #18729 but merged into current runner per @kfaraz request.

I've seen on the giant lock in HttpRemoteTaskRunner cause severe performance degradation under heavy load(200-500ms per acquisition with 1000s of activeTasks can slow down the startPendingTasks loop in TaskQueue). This leads to scheduling delays, which leads to more lag, which auto-scales more tasks, ..., etc. The runner also has a few (un)documented races abundant in the code. This overhead also slows down query tasks under load (e.g. MSQE and others) which utilize the scheduler for execution.

I'm attempting a rewrite of this class to optimize for throughput and safety.

Apart from the performance improvements/bug fixes, this will also include some new features:

• Simpler code. The old task runner had old, legacy ZK references dangling around as well as a pretty complicated scheduling loop.

I would ultimately like to make this the default HttpRemoteTaskRunner and have it run in all tests/production clusters, etc. as I think that would help catch more bugs/issues.

Performance Testing

Test results thus far have shown ~100-300ms speed up per task runner operation (add(), etc.). Over 1000s of tasks, this amounts to minutes of delay saved.

Release note

Speed up throughput and improve thread safety of HttpRemoteTaskRunner

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[kfaraz]

Thanks for creating this PR, @jtuglu1 ! The patch seems much simpler now.
I should be able to complete an initial review today.

[kfaraz]

Leaving a partial review, will try to finish going through the rest of the changes today.

[kfaraz]

Finished going through the bulk of the changes.

On the whole, the patch looks good. I have these major suggestions:

• For the time being, it would be cleaner to use workerStateLock consistently whenever accessing the workers map. We can try to improve this later.
• Avoid use of .forEach() and use .compute() instead, preferably encasing it in an addOrUpdate method similar to TaskQueue.
• Do not perform any heavy operation like metadata store access, metric emission, listener notification, etc. inside the .compute() lambda.
• Avoid throwing exceptions inside the lambda, if they are just to be caught back in the same method/loop. Instead, log an error and continue with the loop.
• Remove the priority scheduling changes for now.
• Reduce debug logging.

[kgyrtkirk]

I will move this out from 36.0.0 for now - it doesn't seem like something which should block the release.
If it gets merged in the upcoming days it could still be ported over to the release branch - and thus be part of it!

[jtuglu1]

@gianm any thoughts here?

[gianm]

@gianm any thoughts here?

I will try to take a look. It may take some time to get to it, since the changes look quite extensive.

Have you run this on a real production at-scale cluster yet (something with hundreds or thousands of tasks running simultaneously, ideally)? If so, that's always helpful to know.

[jtuglu1]

@gianm any thoughts here?

I will try to take a look. It may take some time to get to it, since the changes look quite extensive.

Have you run this on a real production at-scale cluster yet (something with hundreds or thousands of tasks running simultaneously, ideally)? If so, that's always helpful to know.

Yes, no observed issues. We run with close to 10k tasks at peak per cluster.

[jtuglu1]

@kfaraz @gianm thoughts here?

[jtuglu1]

@kfaraz @gianm any thoughts here?