Separate PostgresQL runtime and schema ownership

[Jeremy Alvis (iplay88keys)]

Fixes #997

Summary

Separate ateapi's normal database traffic from schema ownership and maintenance by supporting distinct runtime and DDL connection strings.

The existing PostgreSQL connection remains the runtime/DML connection. A new optional DDL connection defaults to the runtime connection, preserving simple single-role deployments.

What changed

• Added a dedicated owner/maintenance pool for:
  • schema creation
  • Goose migrations
  • runtime privilege grants
  • outbox partition creation, cleanup, and retention
• Kept request handling and watch polling on runtime connections.
• Limited the owner pool to two connections, sufficient for Goose's session lock and migration work.
• Granted the runtime role DML access to tables and sequences after migrations while excluding the migration ledger.
• Preserved single-DSN operation by making the DDL DSN default to the runtime DSN.

The bundled PostgreSQL deployment now:

• provisions separate ateapi_runtime and ateapi_ddl roles
• generates and persists independent passwords
• restricts TCP access to those roles and the atepg database
• requires both SCRAM authentication and a trusted client certificate
• assigns schema and object ownership to the DDL role
• disables SCRAM channel binding in generated bundled DSNs because pgx cannot derive channel-binding data from the Ed25519-signed service certificate; verified TLS, client certificates, and SCRAM remain enabled

Installer changes include:

• storing credential-bearing DSNs in a Secret
• keeping the schema name in the ConfigMap
• rejecting a DDL DSN without an explicit runtime DSN
• hashing the effective database configuration into the pod template so credential or schema changes roll ateapi
• redacting DSNs from logs
• applying and waiting for bundled PostgreSQL when deploying ateapi independently

Documentation now recommends a dedicated Substrate schema for split-role deployments because runtime grants cover every table and sequence in that schema.

Security considerations

This prevents normal request paths from performing migrations or outbox partition DDL and provides database-level separation and auditing.

Both credentials still exist inside the serving ateapi process, so this is not a security boundary against complete process compromise. Moving migrations and partition maintenance to a separate process remains a possible future hardening step.

Testing

• go test ./cmd/ateapi/internal/store/atepg ./cmd/ateapi ./cmd/ate-setup/internal/config ./cmd/ate-setup/internal/steps
• bash hack/verify/shellcheck.sh
• bash -n hack/install-ate.sh
• git diff --check

Tests cover split-role permissions, single-DSN operation, owner-pool routing, installer credential reuse, DSN validation, and configuration-hash rollouts.