[SDTEST-3870] Improve planner report narrative by anmarchenko · Pull Request #100 · DataDog/ddtest

anmarchenko · 2026-06-30T09:33:08Z

What

Reworks the DDTest plan report and simplifies the report data flow:

Keeps Planning focused on planner decisions: discovery, duration application, applied skips, run set size, and split quality.
Prints DDTest settings only when they differ from defaults.
Reports discovery/cache/duration details only when they apply to the selected discovery path.
Reports TIA skippables as tests or suites, matching the configured skipping level.
Builds report data at print time from planner state and the test optimization client instead of storing report-only backend wrappers in the planner.
Uses backend request timings recorded by the API transport instead of duplicating timing state in the test optimization client.
Removes ddtest's library capability tag injection; only the skippables correlation ID tag remains.

Why

The previous planner report mixed backend inventory, planner actions, and split details in a way that was hard for customers to interpret. It also made the planner and test optimization client carry report-only state, which made the implementation harder to review and maintain.

This keeps the customer report concise while reducing the amount of plumbing outside report.go and report_data.go.

E2E testing

Manual scenario: run ddtest plan with report output enabled on a suite-skipping project that has backend known tests, skippables, managed flaky tests, suite durations, and a configured discovery cache.

To test different report shapes, use mock backend.
Test also with real backend (EU target) to confirm that numbers are real.

Confirm the report separates backend-returned data from planning decisions and prints concise labels like this:

+++ DDTest: plan report

Run
  Service: checkout-api
  Repository: https://github.com/acme/checkout.git
  Commit: 9f3a1c7d2b4e
  Branch: feature/split-report
  Platform: ruby / rspec
  OS tags: os.platform=linux, os.architecture=amd64, os.version=6.8.0
  Runtime tags: runtime.name=ruby, runtime.version=3.3.4

DDTest settings
  Platform: python
  Framework: pytest
  Min parallelism: 9
  Max parallelism: 10
  CI job overhead: 30s
  Worker env: DATABASE_PASSWORD, RAILS_ENV
  CI node: 0
  CI node workers: 2
  Command: pytest -q
  Tests location: spec/**/*_spec.rb
  Tests exclude pattern: spec/system/**/*_spec.rb
  Test discovery cache: .ddtest-cache/tests.json
  Test skipping mode: suite
  Force full test discovery: true
  Strict discovery: true
  Runtime tags: {"runtime.version":"3.3.4"}
  Report enabled: false

Datadog settings
  Fetch duration: 240ms
  Test Impact Analysis: enabled
    Test skipping: enabled
    Test impact collection: disabled
  Known tests: enabled
  Early flake detection: enabled
  Auto test retries: enabled
  Flaky test management: enabled

Backend data
  Known tests: 4 modules, 1,284 suites, 18,921 tests (fetched in 80ms)
  TIA skippables returned: 312 suites (fetched in 110ms)
  Managed flaky tests: 26 total, 8 quarantined, 3 disabled, 5 attempt-to-fix (fetched in 90ms)
  Test suite durations: 3 modules, 1,491 suites (fetched in 140ms)

Planning
  Discovery
    Method: full
    Test files: 642
    Cache: used
    Duration: 3s
    Suites discovered: 1,284
    Tests discovered: 18,921
  Duration estimates
    Backend durations used: 431 suites
    Default durations used: 90 suites
    Backend-only suites added: 12
  Skipping
    TIA skippables applied: 312 suites
    Disabled tests applied: 3 tests
    Suites marked unskippable: 5
    Files fully skipped: 118
  Run set
    Test files to run: 524
    Estimated time saved: 38.40%
  Runner split
    Full runtime: 37m12s
    Estimated runtime: 23m46s
    Runners: 6
    Expected wall time: 4m12s
    Imbalance: 11s

Slow suites on dedicated runners
  ATTENTION: 1 dedicated runner
  1. runner 0, rspec / Checkout::Slow (spec/slow_spec.rb): historical duration 2m0s, estimated runtime 1m40s

10 slowest test suites overall
  1. rspec / Checkout::VerySlow (spec/very_slow_spec.rb): historical duration 3m0s, estimated runtime 3m0s
  2. rspec / Checkout::Slow (spec/slow_spec.rb): historical duration 2m0s, estimated runtime 1m40s

datadog-datadog-prod-us1-2

More details

Report generation refactoring is correct and handles all input variations properly. The PR separates backend inventory from planner decisions, tracks applied skippables accurately across multiple match types (TIA test, TIA suite, disabled test), and correctly formats output based on discovery mode. Reflection-based settings comparison produces only non-default values. All 30+ test scenarios pass including edge cases (zero values, unavailable data, malformed input, singular/plural formatting). No regressions detected.

Was this helpful? React 👍 or 👎

_{📊 Validated against 32 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit ab3a902 · What is Autotest? · Any feedback? Reach out in #autotest}

datadog-datadog-prod-us1-2 · 2026-06-30T09:45:15Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 97.41%
• Overall Coverage: 88.21% (+0.76%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a95edf6 | Docs | Datadog PR Page | Give us feedback!}

anmarchenko · 2026-06-30T14:55:50Z

E2E Test Report: SUCCESS

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

Method: Local Shepherd testing with crook
PR: [SDTEST-3870] Improve planner report narrative #100
Branch tested: anmarchenko/report-default-settings
Revision tested: a95edf6599cc018d07552da472d41140f679e8f4
Primary real backend target: eu (datadoghq.eu)
Mock backend: mockdog

Results

Check	Status	Evidence
PR branch builds and injects through crook	PASS	`ddtest` built from `anmarchenko/report-default-settings` in all runs
Planner report with real EU backend data	PASS	Settings, known tests, skippables, managed flaky tests, and suite durations fetched from EU
Real EU backend response shape	PASS	Settings type `ci_app_tracers_test_service_settings`; known tests `1 module / 834 suites / 9,497 tests`; skippables `9,377` with `meta.correlation_id`; durations `1 module / 832 suites`
Mock report with TIA + suite durations	PASS	Forem scenario applied 4 TIA skippables, 23 duration suites, fully skipped 1 file
Mock report with Test Management disabled tests	PASS	Sidekiq scenario applied 3 TIA skippables + 3 disabled tests, fully skipped 3 files
Discovery cache report shape	PASS	First run reported cache miss; restored-cache run reported `Cache: used`
Full plan + run smoke	PASS	Quotes Rails `ddtest` run: 51 runs, 87 assertions, 0 failures, 0 errors; mockdog received 66 test-cycle events

Issues Found

No product issues found.

One environment note: the real EU backend validation required explicit approval because it sends repository/test metadata to datadoghq.eu. After approval, the EU run completed successfully.

Verification

The real backend validation was performed through Datadog EU API responses used by ddtest plan. The plan-only EU run did not execute tests or create Test Visibility session events, so there was no Datadog UI session to manually verify. The full run against mockdog did submit test-cycle events and telemetry successfully.

Test Methodology

Fetched PR metadata and tested branch anmarchenko/report-default-settings.
Ran ddtest planner scenarios through Shepherd crook with --debug.
Used mockdog scenarios to validate report shapes for TIA skippables, suite durations, Test Management disabled tests, and discovery cache behavior.
Ran one full ddtest plan + worker execution smoke test on Quotes Rails.
Ran Forem ddtest-plan against the real EU backend to validate live response numbers and report output.
Inspected stashed .testoptimization output to confirm planner files matched report summaries.

Real EU Backend DDTest Plan Report

+++ DDTest: plan report

Run
  Service: forem
  Repository: git@github.com:anmarchenko/forem.git
  Commit: daf7fc0b3f2833d10b9d5f2d5d27fd59974f32e7
  Branch: anmarchenko/dogfood-suite-level-skipping
  Platform: ruby / rspec
  OS tags: os.platform=darwin23, os.architecture=arm64, os.version=25.5.0
  Runtime tags: runtime.name=ruby, runtime.version=3.3.0

DDTest settings
  Min parallelism: 1
  Max parallelism: 8
  Tests location: spec/policies/*_spec.rb

Datadog settings
  Fetch duration: 333ms
  Test Impact Analysis: enabled
    Test skipping: enabled
    Test impact collection: enabled
  Known tests: enabled
  Early flake detection: enabled
  Auto test retries: disabled
  Flaky test management: enabled

Backend data
  Known tests: 1 modules, 834 suites, 9,497 tests (fetched in 297ms)
  TIA skippables returned: 9,377 tests (fetched in 2.42s)
  Managed flaky tests: 0 total, 0 quarantined, 0 disabled, 0 attempt-to-fix (fetched in 47ms)
  Test suite durations: 1 modules, 832 suites (fetched in 116ms)

Planning
  Discovery
    Method: full
    Test files: 23
    Cache: not configured
    Duration: 6.056s
    Suites discovered: 23
    Tests discovered: 363
  Duration estimates
    Backend durations used: 1 suite
    Default durations used: 22 suites
    Backend-only suites added: 0
  Skipping
    TIA skippables applied: 362 tests
    Disabled tests applied: 0 tests
    Suites marked unskippable: 0
    Files fully skipped: 22
  Run set
    Test files to run: 1
    Estimated time saved: 99.98%
  Runner split
    Full runtime: 42.805s
    Estimated runtime: 9ms
    Runners: 1
    Expected wall time: 9ms
    Imbalance: 0s

Slow suites on dedicated runners
  None

10 slowest test suites overall
  1. rspec / ArticlePolicy at ./spec/policies/article_policy_spec.rb (spec/policies/article_policy_spec.rb): historical duration 30.799s, estimated runtime 0s
  2. rspec / CommentPolicy at ./spec/policies/comment_policy_spec.rb (spec/policies/comment_policy_spec.rb): historical duration 4.426s, estimated runtime 0s
  3. rspec / ReactionPolicy at ./spec/policies/reaction_policy_spec.rb (spec/policies/reaction_policy_spec.rb): historical duration 3.481s, estimated runtime 0s
  4. rspec / ResponseTemplatePolicy at ./spec/policies/response_template_policy_spec.rb (spec/policies/response_template_policy_spec.rb): historical duration 1.206s, estimated runtime 0s
  5. rspec / UserPolicy at ./spec/policies/user_policy_spec.rb (spec/policies/user_policy_spec.rb): historical duration 471ms, estimated runtime 0s
  6. rspec / Authorizer at ./spec/policies/authorizer_spec.rb (spec/policies/authorizer_spec.rb): historical duration 426ms, estimated runtime 0s
  7. rspec / DiscussionLockPolicy at ./spec/policies/discussion_lock_policy_spec.rb (spec/policies/discussion_lock_policy_spec.rb): historical duration 329ms, estimated runtime 0s
  8. rspec / OrganizationPolicy at ./spec/policies/organization_policy_spec.rb (spec/policies/organization_policy_spec.rb): historical duration 297ms, estimated runtime 0s
  9. rspec / SpacePolicy at ./spec/policies/space_policy_spec.rb (spec/policies/space_policy_spec.rb): historical duration 220ms, estimated runtime 0s
  10. rspec / ApiSecretPolicy at ./spec/policies/api_secret_policy_spec.rb (spec/policies/api_secret_policy_spec.rb): historical duration 203ms, estimated runtime 0s

This E2E test was performed by Shepherd - autonomous QA agent for Datadog Test Optimization.

anmarchenko added 4 commits June 30, 2026 09:52

Report changed DDTest settings

03b7004

Report backend duration data

b4cb55b

Clarify skippables report counts

f55a2e3

Improve planner report narrative

ab3a902

anmarchenko changed the title ~~Improve planner report narrative~~ [SDTEST-3870] Improve planner report narrative Jun 30, 2026

anmarchenko marked this pull request as ready for review June 30, 2026 09:34

anmarchenko requested a review from a team as a code owner June 30, 2026 09:34

datadog-datadog-prod-us1-2 Bot reviewed Jun 30, 2026

View reviewed changes

Report discovery cache usage

0bebb65

anmarchenko added 2 commits June 30, 2026 11:51

Report planning operation durations

1e9c00c

Simplify plan report data flow

a95edf6

juan-fernandez approved these changes Jun 30, 2026

View reviewed changes

anmarchenko merged commit 5890608 into main Jun 30, 2026
4 checks passed

anmarchenko deleted the anmarchenko/report-default-settings branch June 30, 2026 14:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SDTEST-3870] Improve planner report narrative#100

[SDTEST-3870] Improve planner report narrative#100
anmarchenko merged 7 commits into
mainfrom
anmarchenko/report-default-settings

anmarchenko commented Jun 30, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jun 30, 2026 •

edited by datadog-prod-us1-5 Bot

Loading

Uh oh!

anmarchenko commented Jun 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

anmarchenko commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

E2E testing

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jun 30, 2026 • edited by datadog-prod-us1-5 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anmarchenko commented Jun 30, 2026

E2E Test Report: SUCCESS

Test Environment

Results

Issues Found

Verification

Test Methodology

Real EU Backend DDTest Plan Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

anmarchenko commented Jun 30, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Jun 30, 2026 •

edited by datadog-prod-us1-5 Bot

Loading