Skip to content

Preserve Firebase polling diagnostics and collect terminal iOS results #518

Description

@leehack

Problem

The PR #515 remote controller loses the actionable polling failure and collects results before cancellation has reached a terminal state. A bounded iOS attempt therefore ended with no test execution, collection=COMPLETE, and only a generic StateError diagnostic. Qualification correctly stayed false and cleanup succeeded.

Reproduction evidence

  • Runner commit: 21f89f94d is the current reviewed head; the submitted run began on b92a0e1df. Both retain the same controller code.
  • Clean signed bundle: 31867c10c, native v0.4.1, Qwen3.5 GGUF Metal.
  • Target: Firebase iPhone 16 Pro / iOS 18.3, Xcode 26.2, eight-minute test timeout, no flaky retries.
  • Matrix: matrix-3k4vi693lyhyc; run: qa-1789730285358167.
  • Journal: FAILED, cleanup VERIFIED, collection COMPLETE. Error: Remote operation failed (StateError); inspect bounded local diagnostics.
  • Collected matrix.json is still PENDING; only the uploaded tests.zip was available at collection time.
  • Fresh provider read: matrix FINISHED / outcome INCONCLUSIVE; execution CANCELLED. Tool Results: complete, abortedByUser=true, no testProcessDuration, no XCTest output. The controller's cleanup issued cancellation; this is not evidence of an iOS runtime/model failure.
  • Android's earlier run completed collection normally but failed its model download deadline: 2,285,638,612 / 2,588,147,712 bytes in ten minutes, before inference. That is a separate model-delivery gap.

Expected follow-up

  • Persist safe structured polling diagnostics (phase, operation, HTTP status or typed failure code) without tokens, URLs with credentials, or arbitrary secret-bearing error text.
  • Exercise transient polling failures explicitly; any bounded retry must retain the existing absolute deadline and must never resubmit the matrix.
  • Refresh the matrix/result inventory after terminal cleanup and distinguish a transferred pre-terminal snapshot from complete final test artifacts.
  • Keep cancelled/preparation-only runs unqualified. Preserve verified cleanup and fail-closed allowance handling when test-process usage is absent.
  • Add production controller-path positive and negative tests before another bounded iOS attempt.

No automatic rerun is requested. Both matrices are terminal and no GCE VM was created.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:P1Act next: current correctness, contract, or repository-integrity risk

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions