Skip to content

[Bug] Concurrent parse_junit --run-id uploads to one run drop results and still exit 0 #438

Description

@marksaxer-cbs

Description

trcli ... parse_junit --run-id <RUN> does not appear to be safe to run concurrently against the same run. When several CI shards each upload their own JUnit report to one shared run (created with include_all=false), results are silently dropped and the CLI still exits 0.

Two observable failure modes:

  • Loud: a shard fails with
    Field :results contains one or more invalid results (case C<id> unknown or not part of the test run),
    for cases that were in its own report.
  • Silent: in larger fan-outs, a whole module's results disappear from the run even though that shard's upload step logged success and exited 0.

Which shards survive is non-deterministic, depending on upload timing.

Steps to reproduce

  1. Create a run with include_all=false and an empty/partial case set.
  2. Prepare 4 JUnit reports covering different cases (e.g. 116 + 43 + 14 + 17 = 190 cases).
  3. Fire 4 uploads concurrently, all targeting the same run:
    trcli -y -h https://<instance>.testrail.io/ --project <P> -u <user> -k <key> \
      parse_junit --suite-id <S> --run-id <RUN> \
      --case-matcher auto --update-existing-cases yes -f report-<n>.xml
    
  4. Compare the run's final case count and per-module presence against the inputs.

Expected result

All 190 results land in the run regardless of upload timing/concurrency. Concurrent uploads to a shared run should be safe, or the tool should fail loudly rather than lose data and still exit 0.

Actual result

Only some cases land (observed: 159 of 190; 31 dropped). Losers either error with case C<id> not part of the test run, or — in larger fan-outs — disappear silently while the upload step reports success and exits 0.

Severity / Reproducibility

  • Severity: S2 Major — silent data loss in reporting; CI result aggregation blocked, no in-tool workaround.
  • Reproducibility: Intermittent (timing-dependent) — reliably reproducible on demand with as few as 4 concurrent uploads.

Versions / environment

  • trcli: 1.14.3
  • Python: 3.11
  • pytest: 7.x; JUnit XML via pytest
  • CI provider: GitHub Actions
  • OS: Ubuntu 22.04 x86_64

Metadata

Metadata

Assignees

No one assigned

    Labels

    TriageThis issue currently being reviewed by the TestRail team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions