[COST-7490] Add midnight hold mechanism for PR checks

[bacciotti]

Jira Ticket

COST-7490

Summary

• Add new Tekton task midnight-hold that acts as a gate before reserve-namespace
• Task fetches PR labels via GitHub API and estimates IQE job duration using empirical data from historical check runs
• If the estimated duration would overlap the UTC midnight boundary, the pipeline sleeps until 00:05 UTC before proceeding
• While sleeping, the task pod remains alive — holding its pipeline slot for up to ~60 min in the worst case
• Pipelines triggered well outside the midnight window pass through immediately with no delay
• Scheduled jobs (IS_SCHEDULED_TEST_JOB=true) and non-IQE labels (ok-to-skip-smokes, run-jenkins-tests) are skipped unconditionally

Label → Estimated Duration Mapping

PR Label	Estimated Duration
hot-fix-smoke-tests	~30 min
aws-smoke-tests / azure-smoke-tests / gcp-smoke-tests	~80 min
ocp-on-prem-smoke-tests	~130 min
smoke-tests	~170 min
ocp-smoke-tests	~240 min
full-run-smoke-tests	~290 min
(unrecognised)	parse IQE_CJI_TIMEOUT, capped at 240 min

Pipeline Change

midnight-hold is inserted between init-pipeline-context and reserve-namespace, ensuring the hold occurs before any ephemeral namespace is allocated.

Notes

This is an interim defensive measure (see COST-7402 for the long-term fix). If the test suite is refactored to be date-resilient, this task can be decommissioned.

[gemini-code-assist]

Code Review

This pull request expands the CI management infrastructure to support multiple scheduled jobs, including a new 'onprem' job, and updates the associated scripts, Makefile, and documentation. It also introduces a midnight-hold task to the pipeline, which estimates test duration based on PR labels and delays execution if it would overlap with the UTC midnight boundary to prevent test failures. Feedback highlights a race condition in the time calculation within the midnight-hold task and recommends a more reliable approach for parsing JSON from the GitHub API.

[bacciotti]

Race condition fix (high priority): Fixed in 5db0905 — replaced the three separate date calls with a single epoch-based calculation ($(date -u +%s) % 86400) as suggested.

JSON parsing (medium priority): Acknowledged. The current grep + cut approach is intentionally consistent with how init-pipeline-context.yaml already parses PR labels in this same repository. PR labels in project-koku/koku are controlled strings without special characters, so the fragility risk is low. Switching to jq would require changing the base image (curlimages/curl → alpine + apk add jq), which adds complexity not justified for an interim defensive mechanism. Can revisit if the label parsing ever proves fragile in practice.