test: add long-running runtime reliability tests

[RentnerKev]

Summary

Add long-running reliability coverage beyond short production smoke
tests.

The existing production smokes remain the fast merge gate.

This issue adds longer runtime/soak testing.

Scenarios

Repeatedly exercise:

• create/update/delete Proxy Hosts
• create/update Redirect Hosts
• certificate changes
• Access Policy changes
• repeated reconciliation
• Caddy restart
• Controller restart
• Web restart
• transient runtime failures
• recovery
• backup/restore where appropriate

Test Duration

Do not run a multi-hour test on every PR.

Prefer:

nightly/scheduled
or explicit release-gate workflow

while keeping representative shorter reliability checks in normal CI.

Detect

• memory/resource leaks where reasonably observable
• stuck reconciliation
• duplicate apply
• stale revision activation
• runaway retries
• process crashes
• state divergence

Acceptance Criteria

• deterministic long-running test harness
• bounded runtime
• useful failure diagnostics
• no production secrets
• clean resource cleanup
• scheduled/release execution

Priority

P0

Dependencies and sequencing

Extends completed reconciliation #12 and fast production-smoke coverage #13, rather than replacing either. Include representative #64/#65 policy behavior when those features land. Long-running release evidence is required by #75.

[github-actions]

🔍 Similar issues found

This issue may overlap with existing reports. Please take a look at these related items.

Match	Issue	Status
Medium	#6 – [Bug]: test	CLOSED · COMPLETED

This is an automated suggestion. This issue has not been closed automatically.