Description
For reasons explained in #16800 there can be changes pushed while a test is running, making its results completely unreliable, both false positives and false negatives. While the linked ticket aims at surfacing that visibly, there's also the option for a newer run enqueued to cancel out the previous one. I originally wanted to avoid this to always have the results of the previous run, even if not completely reliable — under the condition it's annotated so, but the ticket below shows the disparity in the (fast subsequent) deployment, and running older specs (or is it actually newer? gotta verify how SHAs are used from the webhooks) for the integration test CI as it was back when the former deployment went to roll out is probably not worth it.
Success Criteria
Description
For reasons explained in #16800 there can be changes pushed while a test is running, making its results completely unreliable, both false positives and false negatives. While the linked ticket aims at surfacing that visibly, there's also the option for a newer run enqueued to cancel out the previous one. I originally wanted to avoid this to always have the results of the previous run, even if not completely reliable — under the condition it's annotated so, but the ticket below shows the disparity in the (fast subsequent) deployment, and running older specs (or is it actually newer? gotta verify how SHAs are used from the webhooks) for the integration test CI as it was back when the former deployment went to roll out is probably not worth it.
Success Criteria
#www-notifyis understandable