k8s Secret manifest ships plaintext CHANGE_ME placeholders with no documented safe-apply workflow

[Wilfred007]

What

Document (and ideally enforce via tooling) a safe workflow for populating k8s/base/backend-secret.yaml so real secrets never get hand-typed into a manifest that could be accidentally committed, and evaluate External Secrets/Sealed Secrets for a longer-term fix.

Why

k8s/base/backend-secret.yaml currently ships plaintext CHANGE_ME placeholders for DATABASE_URL, JWT_SECRET, STELLAR_SECRET_KEY, ANCHOR_API_KEY, and SDS_API_KEY, intended to be hand-edited and then kubectl apply -f'd. There's no External Secrets Operator or Sealed Secrets integration, and no documented safeguard preventing a real value from ending up committed to git by mistake — notably including a real Stellar secret key.

Scope

In scope:

• Add a clear k8s/README (or extend infrastructure/README.md) section documenting the safe workflow: never commit real values, use kubectl create secret from local env vars or a secrets manager, or template via CI secrets
• Evaluate and, if adopted, wire up External Secrets Operator or Sealed Secrets so the committed manifest never contains real values even as placeholders in a way that invites hand-editing in place
• Add a pre-commit or CI check that flags if backend-secret.yaml contains anything other than the placeholder values

Out of scope:

• Migrating actual production secrets (that's an operational task for whoever runs the cluster)
• Other k8s manifests unrelated to secrets

Acceptance Criteria

• Documented, discoverable safe-apply workflow exists
• A CI/pre-commit check exists that fails if backend-secret.yaml is committed with non-placeholder values
• (If adopted) External Secrets/Sealed Secrets integration is wired and documented

Technical Context

• k8s/base/backend-secret.yaml
• infrastructure/README.md, charts/payd/values.yaml / values-production.yaml

[Mainnet-ops]

Application to Work on Kubernetes Secret Management and Security Workflow

I am interested in contributing to this issue because securely managing secrets is critical for any application deployed to Kubernetes. The current approach of keeping CHANGE_ME placeholders in backend-secret.yaml and expecting developers to manually replace them creates a significant risk of accidentally committing real credentials to the repository.

I understand the goal of this issue is to establish a safe and repeatable workflow where real secrets are never stored directly in Git, while also adding safeguards to catch accidental changes.

My proposed approach would include:

• Reviewing the current k8s/base/backend-secret.yaml and identifying all sensitive values that must never be committed.
• Documenting a clear workflow for developers to create or apply Kubernetes secrets locally using kubectl create secret, local environment variables, or an approved secrets manager.
• Ensuring the committed Kubernetes manifest contains only safe placeholders or references rather than real credentials.
• Adding a CI or pre-commit validation that specifically checks backend-secret.yaml and fails if any value differs from the approved placeholder values.
• Evaluating External Secrets Operator and Sealed Secrets based on security, maintainability, deployment complexity, and compatibility with the existing infrastructure.
• If one of these solutions is adopted, integrating it into the existing Kubernetes configuration and documenting how developers and deployment pipelines should use it.
• Reviewing the existing infrastructure documentation and Helm values to ensure the new workflow is discoverable and consistent.

I would also pay particular attention to preventing sensitive credentials such as STELLAR_SECRET_KEY, JWT_SECRET, database credentials, and API keys from appearing in Git history or Kubernetes manifests.

For testing, I would verify that the security check passes with the expected placeholder values and fails when a placeholder is replaced with a realistic secret-like value. I would also test the documented secret creation and deployment workflow to ensure it works without requiring developers to modify the committed manifest.

My goal would be to make the secure workflow simple enough that developers naturally follow it, while using automated checks as a second layer of protection against accidental secret exposure.

I would be glad to contribute to the implementation, documentation, testing, and evaluation of the long-term Kubernetes secrets solution.

[grantfox-oss]

🦊 GrantFox — @Mainnet-ops has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #455)
2. Your PR will be reviewed by the Protocol-Guild maintainers

Good luck! Track your progress on GrantFox.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox as part of the Third Campaign campaign!

@Mainnet-ops's PR #583 was approved and merged by @Wilfred007.

🏆 @Mainnet-ops: You earned 35 FoxPoints for this contribution! Your current tier: Explorer (306 total points). Track your full progress on GrantFox.

👏 Great work, @Mainnet-ops! Keep contributing to Protocol-Guild.