You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
check-instance.yml in contrail-gh runs contrail sync --dry-run on every pull request, which returns from _dry_run_report before it ever constructs a provider. So the whole emissions path is untouched by CI, and nothing anywhere exercises it against the live API:
The test suite can't. "No test may make a real network call" — requests is mocked in both directions, by design. Every TIM test asserts against a fixture of what the API returned once.
The pull request check doesn't, because --dry-run never prices.
The scheduled sync does, at 15:37 UTC, once a day, in a repo where a red X is easy to miss for a week.
Two different failures hide in that gap, and they want the same thing:
An instance is misconfigured.TIM_API_KEY absent, expired, restricted to the wrong API, or the Google Cloud project's TIM API quietly disabled. Today the first sign is a failed scheduled run.
The provider drifts from the API. TIM changes a field name, a status code, or its validation, and contrail's mocks keep passing because they encode the old shape.
What isn't obvious
Sampling N of the user's own flights can silently sample zero. The repriceable set is whatever this run found — on a quiet fortnight there are no upcoming flights, so sync --sample 1 prices nothing, exits 0, and proves exactly as much as --dry-run does. That is the same blind spot wearing a new hat, and it fails in the least visible way: green.
"Light" isn't really about call volume.BATCH_SIZE = 200, so N=1 and N=200 are the same number of HTTP requests, and TIM is free. What a cap actually buys is a bounded blast radius and a short, readable log — not saved quota. Worth knowing before designing around cost.
There are three endpoints, not one.compute() posts computeDetailedFlightEmissions, falls back to plain computeFlightEmissions per-flight on a 400, then sends whatever came back empty to computeTypicalFlightEmissions. A sample of past flights only reaches the third. Only a sample containing an upcoming flight reaches the first — and the 400 fallback (tim.py:203) is a binary-search split-and-retry that exists solely because TIM rejects a whole batch over one bad entry. It is the most intricate code in the provider and it is entirely mock-tested.
Two shapes
A. contrail check — a fixed synthetic probe. Ask TIM about one known route a fixed number of days ahead, assert an exact figure comes back; ask about one known past route, assert a route average. Deterministic, exercises the detailed and typical endpoints every time, touches none of the user's data, writes nothing, and gives CI a clean pass/fail. It won't catch "this instance's data trips an edge case", and the probe's forward date needs to stay far enough out not to depart.
B. sync --sample N — price at most N real flights and discard. Catches data-specific failures A can't. Non-deterministic per the above, so it can't be the only check, and it needs care that nothing reaches the CSV or the raw log.
Leaning A, with B as a later addition rather than an alternative. A is the one CI can depend on.
Notes
Whatever this is, it is a CLI capability that CI invokes, not a pytest. The no-network rule on the suite stays.
The consumer is contrail-gh's check-instance.yml, which gains a step and the TIM_API_KEY secret it is currently and deliberately denied. That secret is only worth handing over once there is something for it to prove — which is the whole point of this issue.
check-instance.ymlin contrail-gh runscontrail sync --dry-runon every pull request, which returns from_dry_run_reportbefore it ever constructs a provider. So the whole emissions path is untouched by CI, and nothing anywhere exercises it against the live API:requestsis mocked in both directions, by design. Every TIM test asserts against a fixture of what the API returned once.--dry-runnever prices.Two different failures hide in that gap, and they want the same thing:
TIM_API_KEYabsent, expired, restricted to the wrong API, or the Google Cloud project's TIM API quietly disabled. Today the first sign is a failed scheduled run.What isn't obvious
Sampling N of the user's own flights can silently sample zero. The repriceable set is whatever this run found — on a quiet fortnight there are no upcoming flights, so
sync --sample 1prices nothing, exits 0, and proves exactly as much as--dry-rundoes. That is the same blind spot wearing a new hat, and it fails in the least visible way: green."Light" isn't really about call volume.
BATCH_SIZE = 200, so N=1 and N=200 are the same number of HTTP requests, and TIM is free. What a cap actually buys is a bounded blast radius and a short, readable log — not saved quota. Worth knowing before designing around cost.There are three endpoints, not one.
compute()postscomputeDetailedFlightEmissions, falls back to plaincomputeFlightEmissionsper-flight on a 400, then sends whatever came back empty tocomputeTypicalFlightEmissions. A sample of past flights only reaches the third. Only a sample containing an upcoming flight reaches the first — and the 400 fallback (tim.py:203) is a binary-search split-and-retry that exists solely because TIM rejects a whole batch over one bad entry. It is the most intricate code in the provider and it is entirely mock-tested.Two shapes
A.
contrail check— a fixed synthetic probe. Ask TIM about one known route a fixed number of days ahead, assert an exact figure comes back; ask about one known past route, assert a route average. Deterministic, exercises the detailed and typical endpoints every time, touches none of the user's data, writes nothing, and gives CI a clean pass/fail. It won't catch "this instance's data trips an edge case", and the probe's forward date needs to stay far enough out not to depart.B.
sync --sample N— price at most N real flights and discard. Catches data-specific failures A can't. Non-deterministic per the above, so it can't be the only check, and it needs care that nothing reaches the CSV or the raw log.Leaning A, with B as a later addition rather than an alternative. A is the one CI can depend on.
Notes
check-instance.yml, which gains a step and theTIM_API_KEYsecret it is currently and deliberately denied. That secret is only worth handing over once there is something for it to prove — which is the whole point of this issue.Filed here rather than in contrail-gh because the flag or subcommand has to exist here first; the workflow change there is a follow-up of a few lines.