[SDTEST-3832] Add target time parallelism setting

[anmarchenko]

What

Adds a --target-time / DD_TEST_OPTIMIZATION_RUNNER_TARGET_TIME duration setting for the parallelism calculator. When configured, DDTest first chooses among candidate splits whose expected wall time is at or below the target. If none can satisfy the target within the min/max parallelism range, it warns and selects the split with the lowest expected wall time, ignoring CI job overhead, to get as close as possible to the target.

The plan report now explains split selection: selected split, target-time status, modeled CI overhead, selection score, what DDTest would choose without target time, and the best five candidate splits by score.

Example report:

Planning
  Discovery
    Method: full
    Test files: 642
    Cache: used
    Duration: 3s
    Suites discovered: 1,284
    Tests discovered: 18,921
  Duration estimates
    Backend durations used: 431 suites
    Default durations used: 90 suites
    Backend-only suites added: 12
  Skipping
    TIA skippables applied: 312 suites
    Disabled tests applied: 3 tests
    Suites marked unskippable: 5
    Files fully skipped: 118
  Run set
    Test files to run: 524
    Estimated time saved: 38.40%
  Split selection
    Full test suite time without TIA: 37m12s
    Estimated runtime with TIA: 23m46s
    Selected: 6 runners
    Reason: lowest selection score among splits that meet target time
    Target time: 5m0s, satisfied
    Expected wall time: 4m12s
    Modeled CI overhead: 3m0s (6 runners x configured CI job overhead 30s)
    Selection score: 7m12s (wall time + modeled CI overhead)
    Imbalance: 11s

    Without target time: 4 runners (wall 5m5s, overhead 2m0s, score 7m5s)
      Selected vs without target: 53s faster wall time, 1m0s more CI overhead

    Candidates
      4 runners: wall 5m5s, overhead 2m0s, score 7m5s, missed target by 5s; would choose without target time
      6 runners: wall 4m12s, overhead 3m0s, score 7m12s, met target; selected
      5 runners: wall 4m50s, overhead 2m30s, score 7m20s, met target
      1 runner: wall 23m46s, overhead 30s, score 24m16s, missed target by 18m46s

Why

Customers need a way to bias DDTest toward a wall-time goal while still understanding the cost of that choice. Showing the modeled CI overhead (runners x configured CI job overhead) and the no-target alternative makes the consequences of selecting a target time visible in the plan output.

When the target cannot be reached, the best outcome is to minimize actual wall time rather than optimize the score that includes CI overhead, because the configured target means the user is asking DDTest to get as close as possible to that wall-time goal.

E2E testing

Manual scenario: run ddtest plan --min-parallelism 1 --max-parallelism 8 --target-time 5m --ci-job-overhead 30s on a repository with known test suite durations. Confirm .testoptimization/runner/parallel-runners.txt contains the selected runner count, and confirm the plan report includes the Split selection section with the selected split, no-target alternative, modeled CI overhead, and up to five candidates sorted by score.

Also run with an unreachable target, such as --target-time 30s for a suite whose fastest split is above 30s. Confirm DDTest logs a target-time warning and selects the split with the lowest expected wall time.

[datadog-datadog-us1-prod]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 83.02%
• Overall Coverage: 87.90% (-0.26%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 20519dc | Docs | Datadog PR Page | Give us feedback!}

[datadog-datadog-us1-prod]

More details

The selection algorithm correctly tracks best (globally optimal by score) and targetBest (best meeting target) as independent variables, so bestWithoutTarget is always the true global optimum even when a different split is chosen to satisfy the target. Boundary conditions (exact equality at target, fallback with warning when unreachable, targetTime=0 disabled path, single-candidate edge case) all behave as designed across 11 adversarial scenarios.

Was this helpful? React 👍 or 👎

_{📊 Validated against 11 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 4b3b3f6 · What is Autotest? · Any feedback? Reach out in #autotest}

[anmarchenko]

E2E Test Report: SUCCESS

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

• Method: Local crook testing with the RuboCop playground, ddtest-plan, and mockdog
• Revision tested: 20519dcc36302dfbb4cf3c0c975f777754599ad3
• Dependency under test: ddtest=anmarchenko/add-target-time-parallelism
• Playground: rubocop
• Command shape: crook run rubocop -c ddtest-plan --scenario default --debug

Results

Check	Status	Evidence
Unreachable target-time fallback selects lowest expected wall time	PASS	With DD_TEST_OPTIMIZATION_RUNNER_TARGET_TIME=500ms, no split could meet the target. DDTest warned No parallel runner split meets target time; selecting split with lowest expected wall time, selected 7 runners, and reported the no-target alternative as 1 runner. The stashed parallel-runners.txt also contained 7.
Reachable target-time selection still uses lowest selection score among matching splits	PASS	With DD_TEST_OPTIMIZATION_RUNNER_TARGET_TIME=5s, DDTest selected 5 runners, reported Target time: 5s, satisfied, and used reason lowest selection score among splits that meet target time. The stashed parallel-runners.txt also contained 5.
Plan report explains target-time tradeoff	PASS	Both reports included Split selection, selected runner count, target status, expected wall time, modeled CI overhead, selection score, no-target alternative, and candidate splits.
Cleanup/restoration	PASS	Crook restored RuboCop Gemfile / Gemfile.lock, removed injected ddtest, removed .shepherd, and stopped mockdog.

Issues Found

No product issues found.

During the retest I first tried to pin crook to the raw commit SHA, but crook builds test runners using git clone --branch, so raw SHAs are not valid runner refs. I reran with the PR branch name after confirming the PR head was still 20519dcc36302dfbb4cf3c0c975f777754599ad3.

Verification

Datadog UI verification was not applicable for this change because the validation used ddtest plan only against mockdog. No test workers were executed and no Test Visibility session telemetry was expected. The relevant behavior is local planner output plus .testoptimization/runner/parallel-runners.txt, both of which were verified.

Test Methodology

1. Fetched the latest PR metadata and confirmed the new commit Handle unreachable target time by wall time.
2. Reviewed the diff to identify the changed behavior: unreachable target-time now selects lowest expected wall time, ignoring CI job overhead.
3. Ran RuboCop ddtest-plan with spec/rubocop/*_spec.rb, min=1, max=8, and ci-job-overhead=30s.
4. Used target-time=500ms to verify the unreachable-target fallback path.
5. Used target-time=5s to verify the reachable-target path still selects the lowest score among target-satisfying splits.
6. Inspected debug logs and stashed .testoptimization runner files for both runs.

---

This E2E test was performed by Shepherd - autonomous QA for Datadog Test Optimization.