diff --git a/.github/workflows/secrets-check.yml b/.github/workflows/secrets-check.yml new file mode 100644 index 00000000..5e989082 --- /dev/null +++ b/.github/workflows/secrets-check.yml @@ -0,0 +1,21 @@ +name: Kubernetes Secrets Check + +on: + push: + branches: ["main"] + paths: + - "k8s/**" + pull_request: + branches: ["main"] + paths: + - "k8s/**" + +jobs: + check-secrets-placeholders: + name: Verify no real secrets in k8s manifests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Check backend-secret.yaml for non-placeholder values + run: ./scripts/check-k8s-secrets.sh diff --git a/.husky/pre-commit b/.husky/pre-commit index 2312dc58..2eee4916 100644 --- a/.husky/pre-commit +++ b/.husky/pre-commit @@ -1 +1,2 @@ npx lint-staged +./scripts/check-k8s-secrets.sh diff --git a/docs/deployment.md b/docs/deployment.md index 553c0d80..3488d039 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -40,21 +40,28 @@ This guide covers deploying PayD to a Kubernetes cluster using either raw manife ## Option 1: Using Raw Manifests -### 1. Update Secrets +### 1. Create Secrets -Edit `k8s/base/backend-secret.yaml` with your actual secrets: +**Do not edit `k8s/base/backend-secret.yaml` with real values.** That file is +tracked by git and must only contain placeholders. Instead, create the secret +imperatively: -```yaml -stringData: - DATABASE_URL: "postgresql://user:password@your-db-host:5432/payd_db" - DB_USER: "your_db_user" - DB_PASSWORD: "your_db_password" - JWT_SECRET: "your_secure_jwt_secret" - STELLAR_SECRET_KEY: "your_stellar_secret_key" - ANCHOR_API_KEY: "your_anchor_api_key" - SDS_API_KEY: "your_sds_api_key" +```bash +kubectl create secret generic payd-backend-secrets \ + --namespace payd \ + --from-literal=DATABASE_URL="postgresql://user:password@your-db-host:5432/payd_db" \ + --from-literal=DB_USER="your_db_user" \ + --from-literal=DB_PASSWORD="your_db_password" \ + --from-literal=JWT_SECRET="$(openssl rand -hex 32)" \ + --from-literal=STELLAR_SECRET_KEY="your_stellar_secret_key" \ + --from-literal=ANCHOR_API_KEY="your_anchor_api_key" \ + --from-literal=SDS_API_KEY="your_sds_api_key" \ + --dry-run=client -o yaml | kubectl apply -f - ``` +For production, use the External Secrets Operator to sync from AWS Secrets +Manager. See [k8s/README.md](../k8s/README.md) for details. + ### 2. Update ConfigMap Edit `k8s/base/backend-configmap.yaml` if needed: @@ -346,7 +353,7 @@ kubectl delete -k k8s/base/ ## Security Considerations -1. **Secrets Management**: Use external secret managers (AWS Secrets Manager, HashiCorp Vault) for production +1. **Secrets Management**: Never commit real secrets to `backend-secret.yaml`. Use `kubectl create secret` from env vars, CI secret injection, or the External Secrets Operator (see [k8s/README.md](../k8s/README.md)). A pre-commit hook and CI check enforce this automatically. 2. **Network Policies**: Add NetworkPolicy resources to restrict pod-to-pod communication 3. **Pod Security**: Enable Pod Security Standards/Policies 4. **RBAC**: Create ServiceAccounts with minimal permissions diff --git a/k8s/README.md b/k8s/README.md new file mode 100644 index 00000000..0dfcfd84 --- /dev/null +++ b/k8s/README.md @@ -0,0 +1,223 @@ +# PayD Kubernetes Manifests + +Kustomize-based manifests for deploying PayD to a Kubernetes cluster. + +## Directory Structure + +``` +k8s/base/ +├── kustomization.yaml # Resource list and common labels +├── backend-deployment.yaml # Backend API deployment (2 replicas) +├── backend-service.yaml # Backend ClusterIP service +├── backend-configmap.yaml # Non-sensitive backend config +├── backend-secret.yaml # Secret placeholders (never commit real values) +├── backend-hpa.yaml # Horizontal Pod Autoscaler +├── frontend-deployment.yaml # Frontend deployment (2 replicas) +├── frontend-service.yaml # Frontend ClusterIP service +├── frontend-configmap.yaml # Frontend runtime config +└── ingress.yaml # nginx ingress with TLS +``` + +## Secrets Management + +**`backend-secret.yaml` must never contain real secret values.** The committed file +contains `CHANGE_ME` placeholders only. Real values are injected at deploy time +using one of the methods below. + +### Why this matters + +The `STELLAR_SECRET_KEY` field holds a Stellar private key that controls funds on +the network. A leaked key means immediate, irreversible loss of assets. The other +fields (database credentials, JWT signing key, API keys) are equally sensitive. +Committing any of them to git — even briefly — means they live in the repository +history forever unless every clone is force-purged. + +### Method 1: `kubectl create secret` (simplest, no extra tooling) + +Create the secret imperatively from environment variables or a local file so that +no real value ever touches a tracked file: + +```bash +# From environment variables (recommended for CI) +kubectl create secret generic payd-backend-secrets \ + --namespace payd \ + --from-literal=DATABASE_URL="$DATABASE_URL" \ + --from-literal=DB_USER="$DB_USER" \ + --from-literal=DB_PASSWORD="$DB_PASSWORD" \ + --from-literal=JWT_SECRET="$JWT_SECRET" \ + --from-literal=STELLAR_SECRET_KEY="$STELLAR_SECRET_KEY" \ + --from-literal=ANCHOR_API_KEY="$ANCHOR_API_KEY" \ + --from-literal=SDS_API_KEY="$SDS_API_KEY" \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +Then deploy the remaining manifests without the secret file: + +```bash +kubectl apply -k k8s/base/ --prune -l app=payd +``` + +Or exclude the secret from kustomize by removing it from `kustomization.yaml` +locally (do not commit that change). + +### Method 2: Helm `--set` (if using the Helm chart) + +```bash +helm install payd charts/payd \ + --namespace payd --create-namespace \ + --set backend.secrets.DATABASE_URL="$DATABASE_URL" \ + --set backend.secrets.DB_PASSWORD="$DB_PASSWORD" \ + --set backend.secrets.JWT_SECRET="$JWT_SECRET" \ + --set backend.secrets.STELLAR_SECRET_KEY="$STELLAR_SECRET_KEY" \ + --set backend.secrets.ANCHOR_API_KEY="$ANCHOR_API_KEY" \ + --set backend.secrets.SDS_API_KEY="$SDS_API_KEY" +``` + +### Method 3: External Secrets Operator (recommended for production) + +See [External Secrets Operator](#external-secrets-operator) below. + +### Method 4: CI-injected secrets + +In GitHub Actions or similar CI, store secrets in the CI provider's secret store +and inject them at deploy time: + +```yaml +# In your deploy step +- name: Deploy to Kubernetes + env: + DATABASE_URL: ${{ secrets.DATABASE_URL }} + JWT_SECRET: ${{ secrets.JWT_SECRET }} + STELLAR_SECRET_KEY: ${{ secrets.STELLAR_SECRET_KEY }} + ANCHOR_API_KEY: ${{ secrets.ANCHOR_API_KEY }} + SDS_API_KEY: ${{ secrets.SDS_API_KEY }} + run: | + kubectl create secret generic payd-backend-secrets \ + --namespace payd \ + --from-literal=DATABASE_URL="$DATABASE_URL" \ + --from-literal=DB_USER="payd_user" \ + --from-literal=DB_PASSWORD="$DB_PASSWORD" \ + --from-literal=JWT_SECRET="$JWT_SECRET" \ + --from-literal=STELLAR_SECRET_KEY="$STELLAR_SECRET_KEY" \ + --from-literal=ANCHOR_API_KEY="$ANCHOR_API_KEY" \ + --from-literal=SDS_API_KEY="$SDS_API_KEY" \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +## External Secrets Operator + +For production clusters, the [External Secrets Operator (ESO)](https://external-secrets.io/) +syncs Kubernetes Secrets from an external provider (AWS Secrets Manager, HashiCorp +Vault, GCP Secret Manager, Azure Key Vault, etc.) so that no secret value is ever +stored in the git repository — not even as a placeholder that invites hand-editing. + +### Why ESO over Sealed Secrets + +| Criterion | External Secrets Operator | Sealed Secrets | +|---|---|---| +| Secret source | External provider (AWS SM, Vault, etc.) | Encrypted blob committed to git | +| Key rotation | Automatic via provider | Requires re-encryption with `kubeseal` | +| Access control | IAM/policies at the provider layer | Cluster-scoped sealing key | +| Audit trail | Provider-native (CloudTrail, Vault audit) | Only K8s audit logs | +| Fits existing infra | Yes — PayD already uses AWS Secrets Manager (see `infrastructure/terraform/modules/secrets/`) | Requires new Sealed Secrets controller | +| Secret updates | Automatic sync on provider change | Manual re-seal + commit | + +ESO is the better fit because PayD's Terraform stack already provisions secrets in +AWS Secrets Manager. ESO bridges that into Kubernetes without introducing a second +secret store. + +### Quick start (AWS Secrets Manager) + +1. Install ESO in the cluster: + +```bash +helm repo add external-secrets https://charts.external-secrets.io +helm install external-secrets external-secrets/external-secrets \ + --namespace external-secrets --create-namespace +``` + +2. Create a `SecretStore` pointing at AWS Secrets Manager: + +```yaml +# k8s/base/secret-store.yaml +apiVersion: external-secrets.io/v1beta1 +kind: SecretStore +metadata: + name: aws-secrets-manager + labels: + app: payd +spec: + provider: + aws: + service: SecretsManager + region: us-east-1 + auth: + jwt: + serviceAccountRef: + name: external-secrets-sa +``` + +3. Create an `ExternalSecret` that maps provider keys to K8s secret keys: + +```yaml +# k8s/base/external-secret.yaml +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: payd-backend-secrets + labels: + app: payd + component: backend +spec: + refreshInterval: 1h + secretStoreRef: + name: aws-secrets-manager + kind: SecretStore + target: + name: payd-backend-secrets + creationPolicy: Owner + data: + - secretKey: DATABASE_URL + remoteRef: + key: payd/prod/database-url + - secretKey: DB_USER + remoteRef: + key: payd/prod/db-user + - secretKey: DB_PASSWORD + remoteRef: + key: payd/prod/db-password + - secretKey: JWT_SECRET + remoteRef: + key: payd/prod/jwt-secret + - secretKey: STELLAR_SECRET_KEY + remoteRef: + key: payd/prod/stellar-secret-key + - secretKey: ANCHOR_API_KEY + remoteRef: + key: payd/prod/anchor-api-key + - secretKey: SDS_API_KEY + remoteRef: + key: payd/prod/sds-api-key +``` + +4. Remove `backend-secret.yaml` from `kustomization.yaml` (the ExternalSecret + creates the K8s Secret automatically). + +5. Configure IRSA or static credentials for the ESO service account to read from + AWS Secrets Manager. + +## Pre-commit Safety Check + +A pre-commit hook (`scripts/check-k8s-secrets.sh`) verifies that +`backend-secret.yaml` contains only the expected `CHANGE_ME` placeholders. This +runs automatically via Husky and is also enforced in CI. + +To run it manually: + +```bash +./scripts/check-k8s-secrets.sh +``` + +## Deploying + +See [docs/deployment.md](../docs/deployment.md) for full deployment instructions. diff --git a/scripts/check-k8s-secrets.sh b/scripts/check-k8s-secrets.sh new file mode 100755 index 00000000..6736a67d --- /dev/null +++ b/scripts/check-k8s-secrets.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# +# check-k8s-secrets.sh — verify that k8s/base/backend-secret.yaml contains +# only the expected CHANGE_ME placeholders and no real secret values. +# +# Exit codes: +# 0 — file is clean (only placeholders) +# 1 — file contains values that are NOT known placeholders (possible leak) +# 2 — file not found or parse error +# +# Usage: +# ./scripts/check-k8s-secrets.sh # standalone +# # Also wired into .husky/pre-commit and .github/workflows/secrets-check.yml + +set -euo pipefail + +SECRET_FILE="k8s/base/backend-secret.yaml" + +if [[ ! -f "$SECRET_FILE" ]]; then + echo "ERROR: $SECRET_FILE not found." + exit 2 +fi + +# Allowed placeholder values. Any stringData value not in this set is flagged. +ALLOWED_VALUES=( + "CHANGE_ME" + "CHANGE_ME_TO_A_SECURE_RANDOM_STRING" + "postgresql://payd_user:CHANGE_ME@postgres:5432/payd_db" + "payd_user" +) + +# Extract values from the stringData block. Uses Python for reliable YAML +# parsing — avoids fragile awk/grep that breaks on edge cases. +values=$(python3 -c " +import sys, re + +with open('$SECRET_FILE') as f: + content = f.read() + +# Find the stringData block +match = re.search(r'^stringData:\s*\n((?:\s+\w+:.*\n?)*)', content, re.MULTILINE) +if not match: + sys.exit(2) + +block = match.group(1) +for line in block.strip().splitlines(): + # Extract value after the key: separator + val = line.split(':', 1)[1].strip().strip('\"') + print(val) +" 2>/dev/null) + +if [[ -z "$values" ]]; then + echo "ERROR: Could not parse stringData values from $SECRET_FILE" + exit 2 +fi + +failed=0 +while IFS= read -r value; do + matched=0 + for allowed in "${ALLOWED_VALUES[@]}"; do + if [[ "$value" == "$allowed" ]]; then + matched=1 + break + fi + done + if [[ $matched -eq 0 ]]; then + echo "FAIL: $SECRET_FILE contains a non-placeholder value." + echo " Real secrets must not be committed. See k8s/README.md for safe alternatives." + failed=1 + fi +done <<< "$values" + +if [[ $failed -eq 1 ]]; then + exit 1 +fi + +echo "OK: $SECRET_FILE contains only placeholder values." +exit 0