Skip to content

python_ssl_summary.md

Latest commit

 

History

History
117 lines (91 loc) · 3.97 KB

python_ssl_summary.md

File metadata and controls

117 lines (91 loc) · 3.97 KB
Raw

Python 3.13, Kubernetes, and Vault – SSL Troubles & Fixes

With Python 3.13 enforcing VERIFY_X509_STRICT by default, both the Python Kubernetes client and Vault encountered SSL verification failures with locally generated certifiactes due to issues with Certificate Authority configuration and certificate without IP-based SANs. Here’s how I resolved them.

If you're using proper RFC 5280 and RFC 2818 ca-certs and server-certs for your cluster and vault, you'll likely not see these issues.

Issue Summary

🔹 Kubernetes (MicroK8s 1.32.3)

  • The default CA certificate generated by MicroK8s lacks the required attribute:
    keyUsage=critical,digitalSignature,keyCertSign
  • This causes Python-Kubernetes API calls to fail spectacularly 💥
  • Solution: Regenerate the CA cert with proper key usage extensions

🔹 Vault (1.19.4)

  • Python 3.13 enforces stricter SSL validation, requiring an RFC 5280-compliant CA cert
  • Vault requires at least one IP-based SAN in its TLS certificate for proper authentication
  • Solution: Ensure Vault certs include both the keyUsage extension and an IP-based SAN

Step 1: Fixing MicroK8s CA Certificate

If you’re setting up a fresh MicroK8s cluster, run this right after installing the snap: Original Post

cd /var/snap/microk8s/current/certs
mkdir cadir
openssl genrsa -out cadir/ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days 360 -out cadir/ca.crt -addext "keyUsage=critical,digitalSignature,keyCertSign"
microk8s refresh-certs cadir

If modifying an existing multi-node cluster, additional steps are required:

1️⃣ Prep for Certificate Update

  • Designate one node as the ‘prime’
  • On each remaining node:
kubectl taint nodes <node1> <node2> key=value:NoSchedule
kubectl drain lboost cboost --ignore-daemonsets --delete-emptydir-data
microk8s leave
  • Remove nodes from the cluster:
microk8s remove-node <node1>
microk8s remove-node <node2>

2️⃣ Apply New CA Certificate Run the CA certificate update from step 1 on the ‘prime’ node.

3️⃣ Restart & Rejoin Nodes

  • Restart MicroK8s with the new certificate:
microk8s stop && microk8s start
  • Re-add the nodes back to the cluster:
microk8s add-node

Step 2: Fixing Vault SSL Certificates

Vault’s authentication fails unless its TLS certificate includes an IP-based SAN. Here’s how I generated a valid cert:

1️⃣ Create OpenSSL Config File (openssl.cnf):

[ v3_req ]
keyUsage = critical, digitalSignature, keyEncipherment
subjectAltName = IP:192.168.86.60, IP:127.0.0.1

2️⃣ Generate & Sign Certificates

openssl genpkey -algorithm RSA -out vault.key
openssl req -new -key vault.key -out vault.csr -subj "/CN=vault-server" -addext "subjectAltName=IP:192.168.86.60,IP:127.0.0.1,DNS:vault.local"
openssl x509 -req -in vault.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out vault.crt -days 365 -sha256 -extfile openssl.cnf -extensions v3_req

3️⃣ Verify Certificate Includes Correct SANs

openssl x509 -in vault.crt -text -noout

4️⃣ Update Vault Config to Use New Cert Modify vault.hcl:

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault/certs/vault.crt"
  tls_key_file  = "/etc/vault/certs/vault.key"
}

Restart Vault:

systemctl restart vault

Step 3: Streamlining the Process

I streamlined everything by using the MicroK8s CA to generate Vault’s certs instead of managing two separate CAs. This ensures consistent compliance across services. So far, it’s working well—but time will tell!

Bonus: Common Confusion

🔹 IP-based SAN ≠ IP-based Storage Area Network 😆 Turns out, they are completely unrelated—lesson learned!

Final Thoughts

This setup has eliminated Python 3.13 SSL errors, restored Kubernetes API functionality, and made Vault authentication work again. Hope this helps anyone running into similar issues!