From a71ff37fc659c2cf0f4421709b49c4612172f36d Mon Sep 17 00:00:00 2001 From: ikrispin Date: Mon, 16 Feb 2026 18:04:15 +0200 Subject: [PATCH 1/6] feat: add comprehensive skill coverage to support the troubleshooting use-case --- rh-developer/docs/debugging-patterns.md | 377 ++++++++++++++ rh-developer/docs/prerequisites.md | 5 + rh-developer/docs/selinux-troubleshooting.md | 387 +++++++++++++++ .../skills/containerize-deploy/SKILL.md | 38 ++ rh-developer/skills/debug-build/SKILL.md | 387 +++++++++++++++ rh-developer/skills/debug-container/SKILL.md | 437 ++++++++++++++++ rh-developer/skills/debug-network/SKILL.md | 397 +++++++++++++++ rh-developer/skills/debug-pod/SKILL.md | 321 ++++++++++++ rh-developer/skills/debug-rhel/SKILL.md | 465 ++++++++++++++++++ rh-developer/skills/deploy/SKILL.md | 57 +++ rh-developer/skills/rhel-deploy/SKILL.md | 56 +++ rh-developer/skills/s2i-build/SKILL.md | 22 +- 12 files changed, 2945 insertions(+), 4 deletions(-) create mode 100644 rh-developer/docs/debugging-patterns.md create mode 100644 rh-developer/docs/selinux-troubleshooting.md create mode 100644 rh-developer/skills/debug-build/SKILL.md create mode 100644 rh-developer/skills/debug-container/SKILL.md create mode 100644 rh-developer/skills/debug-network/SKILL.md create mode 100644 rh-developer/skills/debug-pod/SKILL.md create mode 100644 rh-developer/skills/debug-rhel/SKILL.md diff --git a/rh-developer/docs/debugging-patterns.md b/rh-developer/docs/debugging-patterns.md new file mode 100644 index 00000000..420878b9 --- /dev/null +++ b/rh-developer/docs/debugging-patterns.md @@ -0,0 +1,377 @@ +--- +title: Debugging Patterns +category: references +sources: + - title: Kubernetes Debugging Pods + url: https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/ + sections: Debugging Pods, Common Errors + date_accessed: 2026-02-16 + - title: OpenShift Troubleshooting Guide + url: https://docs.openshift.com/container-platform/latest/support/troubleshooting/troubleshooting-operator-issues.html + sections: Pod issues, Build issues + date_accessed: 2026-02-16 + - title: Podman Troubleshooting + url: https://github.com/containers/podman/blob/main/troubleshooting.md + sections: Common Issues + date_accessed: 2026-02-16 +--- + +# Debugging Patterns + +This document provides common error patterns, exit codes, and troubleshooting decision trees for the debugging skills. + +## Exit Code Reference + +### Container/Process Exit Codes + +| Exit Code | Signal | Meaning | Common Cause | +|-----------|--------|---------|--------------| +| 0 | - | Success | Normal termination | +| 1 | - | General error | Application error, unhandled exception | +| 2 | - | Misuse of shell | Invalid arguments, syntax error | +| 126 | - | Permission denied | Cannot execute command | +| 127 | - | Command not found | Binary/script missing in PATH | +| 128 | - | Invalid exit argument | Exit called with non-integer | +| 128+N | Signal N | Killed by signal | See signal table below | +| 137 | SIGKILL (9) | Force killed | OOM kill, manual kill, timeout | +| 139 | SIGSEGV (11) | Segmentation fault | Memory corruption, null pointer | +| 143 | SIGTERM (15) | Terminated | Graceful shutdown request | + +### Signal Reference (128+N) + +| Signal | Number | Exit Code | Typical Cause | +|--------|--------|-----------|---------------| +| SIGHUP | 1 | 129 | Terminal closed | +| SIGINT | 2 | 130 | Ctrl+C | +| SIGQUIT | 3 | 131 | Ctrl+\ | +| SIGKILL | 9 | 137 | OOM, forced termination | +| SIGSEGV | 11 | 139 | Segmentation fault | +| SIGTERM | 15 | 143 | Graceful stop request | + +## Pod Failure Patterns + +### CrashLoopBackOff + +**Symptom:** Pod repeatedly crashes and restarts + +**Diagnosis Flow:** +``` +CrashLoopBackOff +├─ Check exit code +│ ├─ 0 → Application exits normally (missing loop/server?) +│ ├─ 1 → Application error (check logs) +│ ├─ 127 → Command not found (check entrypoint) +│ └─ 137 → OOM killed (check memory limits) +├─ Check logs (current + previous) +│ ├─ Import errors → Missing dependencies +│ ├─ Connection errors → External service down +│ └─ Config errors → Missing env vars/secrets +└─ Check events + └─ FailedMount → Missing secrets/configmaps +``` + +**Common Causes:** +1. Application crashes on startup (dependency errors) +2. Memory limit too low (OOMKilled) +3. Missing environment variables or secrets +4. Database/service connection failures +5. Health probe failing immediately + +### ImagePullBackOff + +**Symptom:** Cannot pull container image + +**Diagnosis Flow:** +``` +ImagePullBackOff +├─ Check event message +│ ├─ "unauthorized" → Registry authentication +│ │ └─ Check imagePullSecrets +│ ├─ "not found" → Wrong image name/tag +│ │ └─ Verify image exists in registry +│ ├─ "timeout" → Network/registry issue +│ │ └─ Check cluster network egress +│ └─ "manifest unknown" → Tag doesn't exist +│ └─ Verify tag in registry +└─ Check image reference + ├─ Missing registry prefix? + ├─ Typo in image name? + └─ Tag exists? +``` + +**Common Causes:** +1. Private registry without imagePullSecret +2. Image tag doesn't exist +3. Registry URL typo +4. Network policy blocking egress +5. Registry rate limiting + +### Pending Pod + +**Symptom:** Pod stuck in Pending state + +**Diagnosis Flow:** +``` +Pending +├─ Check events +│ ├─ "FailedScheduling" +│ │ ├─ "Insufficient cpu/memory" → Scale cluster or reduce requests +│ │ ├─ "node selector" → No matching nodes +│ │ ├─ "taints" → Need tolerations +│ │ └─ "PVC not bound" → Storage issue +│ └─ No events → Check resourceQuota +└─ Check node status + └─ All nodes NotReady? → Node issue +``` + +**Common Causes:** +1. Insufficient cluster resources +2. Node selector doesn't match any nodes +3. PersistentVolumeClaim not bound +4. Resource quota exceeded +5. Affinity/anti-affinity rules too strict + +### OOMKilled + +**Symptom:** Container terminated with exit code 137 + +**Diagnosis Flow:** +``` +OOMKilled (exit 137) +├─ Check container state +│ └─ OOMKilled: true → Memory exhaustion confirmed +├─ Compare memory usage vs limit +│ ├─ Limit too low → Increase memory limit +│ └─ Memory leak → Profile application +└─ Check for: + ├─ Java → Heap size (-Xmx) exceeds limit + ├─ Node.js → --max-old-space-size too high + └─ Python → Large data structures in memory +``` + +**Common Causes:** +1. Memory limit set too low for application +2. Memory leak in application +3. Java heap size exceeds container limit +4. Processing large files/datasets in memory + +## Build Failure Patterns + +### S2I Build Phases + +| Phase | What Happens | Common Failures | +|-------|--------------|-----------------| +| **fetch-source** | Clone git repository | Auth failure, repo not found | +| **pull-builder** | Pull S2I builder image | Image not found, auth | +| **assemble** | Run S2I assemble script | Dependency install, build errors | +| **commit** | Create image layer | Disk space | +| **push** | Push to internal registry | Auth, quota | + +### Assemble Phase Failures + +**Node.js:** +``` +npm ERR! 404 Not Found +└─ Package doesn't exist in registry + → Check package.json for typos + +npm ERR! code ERESOLVE +└─ Dependency conflict + → Run npm install --legacy-peer-deps + +npm ERR! code ENOENT +└─ File not found + → Check paths in package.json +``` + +**Python:** +``` +ERROR: Could not find a version that satisfies the requirement +└─ Package not found + → Check requirements.txt spelling + +ModuleNotFoundError: No module named 'X' +└─ APP_MODULE misconfigured + → See docs/python-s2i-entrypoints.md + +gunicorn: command not found +└─ gunicorn not in requirements + → Add gunicorn to requirements.txt +``` + +**Java:** +``` +[ERROR] Failed to execute goal +└─ Maven/Gradle build failure + → Check pom.xml or build.gradle + +java.lang.OutOfMemoryError: Java heap space +└─ Build needs more memory + → Add MAVEN_OPTS=-Xmx512m +``` + +## Network Troubleshooting + +### Service Has No Endpoints + +**Diagnosis Flow:** +``` +No endpoints +├─ Check service selector +│ └─ Compare with pod labels +│ ├─ Labels don't match → Fix selector or pod labels +│ └─ Labels match → Check pod readiness +├─ Check pod status +│ ├─ Pods not running → Debug pods first +│ └─ Pods running but not ready → Check readiness probe +└─ Check readiness probe + ├─ HTTP probe failing → Application not listening + └─ TCP probe failing → Wrong port +``` + +### Route Returning 503 + +**Diagnosis Flow:** +``` +503 Service Unavailable +├─ Check endpoints +│ └─ No endpoints → Pods not ready +├─ Check backend pods +│ ├─ All pods failing readiness → Application issue +│ └─ Some pods ready → Load balancer issue +└─ Check route configuration + └─ Wrong service or port → Fix route spec +``` + +### Connection Refused + +**Diagnosis Flow:** +``` +Connection refused +├─ Is service created? → oc get svc +├─ Does service have endpoints? → oc get endpoints +├─ Is pod running? → oc get pods +├─ Is application listening? → Check container port +└─ Is port correct? → Compare service port vs container port +``` + +## RHEL System Patterns + +### systemd Service Failures + +| Exit Code | Meaning | Common Fix | +|-----------|---------|------------| +| 1 | General error | Check application logs | +| 126 | Permission | Check ExecStart permissions | +| 127 | Not found | Check binary path in ExecStart | +| 203 | EXEC | Wrong architecture or format | +| 217 | USER | Service user doesn't exist | + +### SELinux Denial Patterns + +| Denial Type | Example | Typical Fix | +|-------------|---------|-------------| +| Port binding | `httpd_t` bind `port_t` | `semanage port -a -t http_port_t -p tcp [port]` | +| File read | `httpd_t` read `user_home_t` | `semanage fcontext` + `restorecon` | +| Network connect | `httpd_t` connect | `setsebool -P httpd_can_network_connect on` | +| Container | `container_t` manage | `setsebool -P container_manage_cgroup on` | + +See [selinux-troubleshooting.md](selinux-troubleshooting.md) for detailed SELinux guidance. + +## Troubleshooting Decision Tree + +### Application Not Accessible + +``` +Cannot access application +├─ Internal (from cluster)? +│ ├─ Yes, works internally → Route/Ingress issue +│ │ ├─ Check route admitted +│ │ ├─ Check route host/path +│ │ └─ Check TLS configuration +│ └─ No, fails internally too → Service/Pod issue +│ ├─ Check service endpoints +│ ├─ Check pod status +│ └─ Check pod readiness +└─ Neither works? + └─ Debug pod first (/debug-pod) +``` + +### Build Keeps Failing + +``` +Build failures +├─ Which phase? +│ ├─ fetch-source → Git access issue +│ │ ├─ Check source secret +│ │ └─ Verify git URL +│ ├─ pull-builder → Builder image issue +│ │ ├─ Check image reference +│ │ └─ Import ImageStream +│ ├─ assemble → Build script issue +│ │ ├─ Check dependencies +│ │ └─ Check language-specific config +│ └─ push → Registry issue +│ └─ Check push secret +└─ Same failure pattern? + └─ Compare with last successful build +``` + +## Quick Reference Commands + +### OpenShift Debugging + +```bash +# Pod status and events +oc describe pod [pod-name] + +# Pod logs (current) +oc logs [pod-name] + +# Pod logs (previous container) +oc logs [pod-name] --previous + +# All events in namespace +oc get events --sort-by='.lastTimestamp' + +# Check endpoints +oc get endpoints [service-name] + +# Build logs +oc logs build/[build-name] +``` + +### RHEL Debugging + +```bash +# Service status +systemctl status [service] + +# Journal logs +journalctl -u [service] -n 100 + +# SELinux denials +ausearch -m AVC -ts recent + +# Firewall rules +firewall-cmd --list-all + +# SELinux context +ls -lZ [path] +``` + +### Container Debugging + +```bash +# List all containers +podman ps -a + +# Container inspect +podman inspect [container] + +# Container logs +podman logs [container] + +# Run interactively for debugging +podman run -it --entrypoint /bin/sh [image] +``` diff --git a/rh-developer/docs/prerequisites.md b/rh-developer/docs/prerequisites.md index c1d1b70c..d81a9b5c 100644 --- a/rh-developer/docs/prerequisites.md +++ b/rh-developer/docs/prerequisites.md @@ -35,6 +35,11 @@ This document lists all tools required by the rh-developer agentic collection. | `/containerize-deploy` | `oc` | `git`, `helm` | | `/rhel-deploy` | `ssh`, `podman` or `docker` | `git`, `dnf` | | `/recommend-image` | - | `skopeo`, `curl`, `jq` | +| `/debug-pod` | `oc` | - | +| `/debug-build` | `oc` | - | +| `/debug-network` | `oc` | - | +| `/debug-rhel` | `ssh` | `ausearch`, `journalctl` | +| `/debug-container` | `podman` or `docker` | - | ## Tool Reference diff --git a/rh-developer/docs/selinux-troubleshooting.md b/rh-developer/docs/selinux-troubleshooting.md new file mode 100644 index 00000000..9942375c --- /dev/null +++ b/rh-developer/docs/selinux-troubleshooting.md @@ -0,0 +1,387 @@ +--- +title: SELinux Troubleshooting +category: references +sources: + - title: Red Hat SELinux User's and Administrator's Guide + url: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/using_selinux/index + sections: Troubleshooting, Managing confined services + date_accessed: 2026-02-16 + - title: SELinux Project Wiki + url: https://selinuxproject.org/page/Main_Page + sections: Troubleshooting + date_accessed: 2026-02-16 + - title: Fedora SELinux Guide + url: https://docs.fedoraproject.org/en-US/quick-docs/selinux-getting-started/ + sections: Troubleshooting + date_accessed: 2026-02-16 +--- + +# SELinux Troubleshooting + +This document provides guidance for diagnosing and resolving SELinux access denials on RHEL/Fedora/CentOS systems. + +## Understanding SELinux + +### SELinux Modes + +| Mode | Description | Use Case | +|------|-------------|----------| +| **Enforcing** | SELinux policy is enforced, denials are blocked and logged | Production | +| **Permissive** | SELinux policy is not enforced, denials are logged only | Debugging | +| **Disabled** | SELinux is completely disabled | Not recommended | + +```bash +# Check current mode +getenforce + +# Temporarily switch to permissive (until reboot) +sudo setenforce 0 + +# Switch back to enforcing +sudo setenforce 1 +``` + +### SELinux Contexts + +Every file, process, and port has an SELinux context: + +``` +user:role:type:level +``` + +Example: `system_u:object_r:httpd_sys_content_t:s0` + +- **user**: SELinux user (system_u, user_u, etc.) +- **role**: Role (object_r for files) +- **type**: Type label (most important for troubleshooting) +- **level**: MLS/MCS level (usually s0) + +```bash +# View file context +ls -lZ /path/to/file + +# View process context +ps auxZ | grep [process] + +# View port context +semanage port -l | grep [port] +``` + +## Finding SELinux Denials + +### Using ausearch + +```bash +# Recent denials (last 10 minutes) +sudo ausearch -m AVC -ts recent + +# Denials from today +sudo ausearch -m AVC -ts today + +# Denials for specific process +sudo ausearch -m AVC -c [command-name] + +# Denials involving specific file +sudo ausearch -m AVC -f /path/to/file +``` + +### Using journalctl + +```bash +# SELinux messages in journal +sudo journalctl -t setroubleshoot + +# AVC messages +sudo journalctl | grep "avc: denied" +``` + +### Using sealert + +```bash +# Install setroubleshoot (if not installed) +sudo dnf install setroubleshoot-server + +# Analyze all denials +sudo sealert -a /var/log/audit/audit.log + +# Interactive analysis +sudo sealert -b +``` + +## Reading AVC Denials + +Example AVC denial: + +``` +type=AVC msg=audit(1234567890.123:456): avc: denied { bind } for pid=1234 comm="httpd" src=8080 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=0 +``` + +**Breakdown:** +| Field | Value | Meaning | +|-------|-------|---------| +| `denied { bind }` | bind | Denied action (bind to socket) | +| `pid=1234` | 1234 | Process ID | +| `comm="httpd"` | httpd | Command name | +| `src=8080` | 8080 | Port number | +| `scontext=...httpd_t...` | httpd_t | Source type (process) | +| `tcontext=...unreserved_port_t...` | unreserved_port_t | Target type (port) | +| `tclass=tcp_socket` | tcp_socket | Object class | + +**Translation:** Process `httpd` (type `httpd_t`) was denied permission to `bind` to port `8080` (type `unreserved_port_t`). + +## Common Denial Types and Fixes + +### Port Binding Denials + +**Symptom:** Application cannot bind to non-standard port + +**Example denial:** +``` +avc: denied { name_bind } for comm="nginx" src=8080 scontext=httpd_t tcontext=unreserved_port_t +``` + +**Fix:** +```bash +# Add port to allowed type +sudo semanage port -a -t http_port_t -p tcp 8080 + +# Verify +sudo semanage port -l | grep 8080 +``` + +**Common port types:** +| Port Type | Typical Ports | Used By | +|-----------|---------------|---------| +| `http_port_t` | 80, 443, 8080 | Web servers | +| `postgresql_port_t` | 5432 | PostgreSQL | +| `mysqld_port_t` | 3306 | MySQL/MariaDB | +| `redis_port_t` | 6379 | Redis | +| `mongod_port_t` | 27017 | MongoDB | + +### File Access Denials + +**Symptom:** Application cannot read/write files + +**Example denial:** +``` +avc: denied { read } for comm="httpd" name="config.yaml" scontext=httpd_t tcontext=user_home_t +``` + +**Fix - Change file context:** +```bash +# Set file context pattern +sudo semanage fcontext -a -t httpd_sys_content_t "/srv/myapp(/.*)?" + +# Apply the context +sudo restorecon -Rv /srv/myapp + +# Verify +ls -lZ /srv/myapp +``` + +**Common file types:** +| File Type | Access | Use Case | +|-----------|--------|----------| +| `httpd_sys_content_t` | Read | Web content | +| `httpd_sys_rw_content_t` | Read/Write | Web app data | +| `container_file_t` | Container access | Podman volumes | +| `var_log_t` | Log files | Application logs | + +### Network Connection Denials + +**Symptom:** Application cannot connect to external services + +**Example denial:** +``` +avc: denied { name_connect } for comm="httpd" dest=5432 scontext=httpd_t tcontext=postgresql_port_t +``` + +**Fix - Enable boolean:** +```bash +# Allow httpd to connect to network +sudo setsebool -P httpd_can_network_connect on + +# Or specifically to databases +sudo setsebool -P httpd_can_network_connect_db on + +# List all httpd booleans +sudo getsebool -a | grep httpd +``` + +**Common booleans:** +| Boolean | Purpose | +|---------|---------| +| `httpd_can_network_connect` | Allow outbound network connections | +| `httpd_can_network_connect_db` | Allow database connections | +| `httpd_can_sendmail` | Allow sending email | +| `httpd_use_nfs` | Allow NFS access | +| `container_manage_cgroup` | Allow container cgroup management | + +## Container-Specific Issues + +### Podman Volume Mounts + +When mounting host directories into containers, SELinux may block access. + +**Solutions:** + +1. **Shared label (:z)** - Multiple containers can access + ```bash + podman run -v /host/path:/container/path:z [image] + ``` + +2. **Private label (:Z)** - Only this container can access + ```bash + podman run -v /host/path:/container/path:Z [image] + ``` + +3. **Manual relabeling:** + ```bash + sudo semanage fcontext -a -t container_file_t "/data(/.*)?" + sudo restorecon -Rv /data + ``` + +### Container Booleans + +```bash +# Enable container to manage cgroups (for systemd in container) +sudo setsebool -P container_manage_cgroup on + +# Allow containers to connect to any port +sudo setsebool -P container_connect_any on + +# List all container booleans +sudo getsebool -a | grep container +``` + +## Troubleshooting Workflow + +### Step 1: Confirm SELinux is the Issue + +```bash +# Temporarily disable SELinux +sudo setenforce 0 + +# Test if application works +[test application] + +# Re-enable SELinux +sudo setenforce 1 +``` + +If application works with SELinux permissive, SELinux is blocking. + +### Step 2: Find the Denial + +```bash +# Get recent denials +sudo ausearch -m AVC -ts recent + +# Or use sealert for analysis +sudo sealert -a /var/log/audit/audit.log +``` + +### Step 3: Determine Fix Type + +| Denial Type | Fix Approach | +|-------------|--------------| +| Port binding | `semanage port` | +| File access | `semanage fcontext` + `restorecon` | +| Network connection | `setsebool` | +| Process capability | Custom policy or boolean | + +### Step 4: Apply Fix + +```bash +# For port: +sudo semanage port -a -t [type] -p [tcp/udp] [port] + +# For file: +sudo semanage fcontext -a -t [type] "[path](/.*)?" +sudo restorecon -Rv [path] + +# For boolean: +sudo setsebool -P [boolean] on +``` + +### Step 5: Verify + +```bash +# Test application +[restart and test] + +# Check for new denials +sudo ausearch -m AVC -ts recent +``` + +## Generating Custom Policies + +If no existing type or boolean works, generate a custom policy: + +```bash +# Generate policy from recent denials +sudo ausearch -m AVC -ts recent | audit2allow -M mypolicy + +# Review the policy +cat mypolicy.te + +# Install the policy +sudo semodule -i mypolicy.pp +``` + +**Warning:** Custom policies should be reviewed carefully. They grant permanent permissions. + +## Quick Reference + +### Common Commands + +```bash +# SELinux status +getenforce +sestatus + +# File context +ls -lZ [path] +restorecon -Rv [path] + +# Process context +ps auxZ | grep [process] + +# Port context +semanage port -l | grep [port] +semanage port -a -t [type] -p tcp [port] + +# Booleans +getsebool -a | grep [keyword] +setsebool -P [boolean] on + +# File context rules +semanage fcontext -l | grep [path] +semanage fcontext -a -t [type] "[path](/.*)?" + +# Audit logs +ausearch -m AVC -ts recent +sealert -a /var/log/audit/audit.log +``` + +### Common Types for Web Applications + +| Resource | Type | +|----------|------| +| Web content (read-only) | `httpd_sys_content_t` | +| Web content (read-write) | `httpd_sys_rw_content_t` | +| Web scripts | `httpd_sys_script_exec_t` | +| Application logs | `httpd_log_t` | +| HTTP ports | `http_port_t` | +| Container files | `container_file_t` | + +### Common Booleans for Applications + +| Application | Boolean | Purpose | +|-------------|---------|---------| +| Web server | `httpd_can_network_connect` | Outbound connections | +| Web server | `httpd_can_network_connect_db` | Database connections | +| Web server | `httpd_unified` | Unified handling | +| Container | `container_manage_cgroup` | cgroup management | +| Container | `container_connect_any` | Connect to any port | +| NFS | `use_nfs_home_dirs` | NFS home directories | diff --git a/rh-developer/skills/containerize-deploy/SKILL.md b/rh-developer/skills/containerize-deploy/SKILL.md index 4335c47b..a64bd459 100644 --- a/rh-developer/skills/containerize-deploy/SKILL.md +++ b/rh-developer/skills/containerize-deploy/SKILL.md @@ -475,6 +475,34 @@ Continue to deployment? (yes/no) Rollout complete! ``` +**If rollout fails** (pods not ready, CrashLoopBackOff, ImagePullBackOff, etc.): + +```markdown +## Deployment Failed + +The deployment did not complete successfully. + +**Pod Status:** +| Pod | Status | Ready | Restarts | +|-----|--------|-------|----------| +| [app-name]-xxx-yyy | [status] | 0/1 | [count] | + +--- + +**Would you like me to diagnose the issue?** + +1. **Debug Pod** (`/debug-pod`) - Investigate pod failures +2. **Debug Network** (`/debug-network`) - Check service/route connectivity +3. **Debug Build** (`/debug-build`) - Re-check build if image issues +4. **View logs manually** +5. **Rollback and stop** + +Select an option: +``` + +- If user selects a debug option → Invoke the corresponding skill +- After debugging → Offer to retry deployment + --- ## HELM PATH (If DEPLOYMENT_STRATEGY is "Helm") @@ -596,6 +624,15 @@ All tools from child skills: | Helm | `helm_install`, `helm_upgrade`, `helm_status`, `helm_list`, `pods_list` | | Rollback | `resources_delete`, `helm_uninstall`, `helm_rollback` | +## Related Skills + +| Skill | Use When | +|-------|----------| +| `/debug-pod` | Pod failures (CrashLoopBackOff, OOMKilled, ImagePullBackOff) | +| `/debug-build` | S2I or Podman build failures | +| `/debug-network` | Service connectivity issues (no endpoints, 503 errors) | +| `/debug-rhel` | RHEL deployment failures (systemd, SELinux, firewall) | + ## Reference Documentation For detailed guidance, see: @@ -603,4 +640,5 @@ For detailed guidance, see: - [docs/image-selection-criteria.md](../docs/image-selection-criteria.md) - Image variant selection, LTS timelines - [docs/python-s2i-entrypoints.md](../docs/python-s2i-entrypoints.md) - Python S2I configuration - [docs/rhel-deployment.md](../docs/rhel-deployment.md) - RHEL host deployment (when delegating to /rhel-deploy) +- [docs/debugging-patterns.md](../docs/debugging-patterns.md) - Common error patterns and troubleshooting - [docs/prerequisites.md](../docs/prerequisites.md) - All required tools by skill diff --git a/rh-developer/skills/debug-build/SKILL.md b/rh-developer/skills/debug-build/SKILL.md new file mode 100644 index 00000000..16b53d75 --- /dev/null +++ b/rh-developer/skills/debug-build/SKILL.md @@ -0,0 +1,387 @@ +--- +name: debug-build +description: | + Diagnose OpenShift build failures including S2I builds, Docker/Podman builds, and BuildConfig issues. Automates multi-step diagnosis: BuildConfig validation, build pod logs, registry authentication, and source repository access. Use this skill when builds fail, hang, or produce unexpected results. Triggers on /debug-build command or phrases like "build failed", "S2I error", "can't pull builder image", "can't push to registry", "build timeout". +user_invocable: true +--- + +# /debug-build Skill + +Diagnose OpenShift build failures by automatically gathering BuildConfig, Build status, build pod logs, and related resources. + +## Prerequisites + +Before running this skill: +1. User is logged into OpenShift cluster +2. User has access to the target namespace +3. Build or BuildConfig name is known (or can be identified from recent builds) + +## Critical: Human-in-the-Loop Requirements + +See [Human-in-the-Loop Requirements](../../docs/human-in-the-loop.md) for mandatory checkpoint behavior. + +**IMPORTANT:** This skill requires explicit user confirmation at each step. You MUST: +1. **Wait for user confirmation** before executing diagnostic actions +2. **Do NOT proceed** to the next step until the user explicitly approves +3. **Present findings clearly** and ask if user wants deeper analysis +4. **Never auto-execute** remediation actions without user approval + +If the user says "no" or wants to focus on specific areas, address their concerns before proceeding. + +## Trigger + +- User types `/debug-build` +- User says "build failed", "S2I error", "build won't complete" +- User says "can't pull builder image", "can't push to registry" +- User says "build timeout", "build stuck", "assemble failed" +- After `/s2i-build` reports a failure + +## Input Parameters + +| Parameter | Description | Default | +|-----------|-------------|---------| +| `BUILD_NAME` | Name of specific build to debug | Latest failed build | +| `BUILDCONFIG_NAME` | BuildConfig to analyze | Auto-detect | +| `NAMESPACE` | Target namespace | Current namespace | + +## Workflow + +### Step 1: Identify Target Build + +```markdown +## Build Debugging + +**Current OpenShift Context:** +- Cluster: [cluster] +- Namespace: [namespace] + +Which build would you like me to debug? + +1. **Specify build name** - Enter the build name directly (e.g., myapp-1) +2. **List failed builds** - Show recent failed builds in current namespace +3. **From BuildConfig** - Debug latest build from a specific BuildConfig + +Select an option or enter a build name: +``` + +**WAIT for user response.** Do NOT proceed until user identifies the target build. + +If user selects "List failed builds": +Use kubernetes MCP `resources_list` for builds, filter by Failed phase: + +```markdown +## Recent Failed Builds in [namespace] + +| Build | BuildConfig | Status | Started | Duration | +|-------|-------------|--------|---------|----------| +| [app-1] | [app] | Failed | [timestamp] | [duration] | +| [app-2] | [app] | Cancelled | [timestamp] | [duration] | +| [other-1] | [other] | Failed | [timestamp] | [duration] | + +Which build would you like me to debug? +``` + +**WAIT for user to select a build.** + +### Step 2: Get Build Status Overview + +Use kubernetes MCP `resources_get` to get Build details: + +```markdown +## Build Status: [build-name] + +**Build Info:** +| Field | Value | +|-------|-------| +| BuildConfig | [buildconfig-name] | +| Strategy | [Source/Docker/JenkinsPipeline] | +| Phase | [New/Pending/Running/Complete/Failed/Cancelled] | +| Started | [timestamp] | +| Completed | [timestamp or "Still running"] | +| Duration | [duration] | + +**Build Configuration:** +| Setting | Value | +|---------|-------| +| Source Type | [Git/Binary/Dockerfile] | +| Git URL | [url] | +| Git Ref | [branch/tag] | +| Builder Image | [image:tag] | +| Output Image | [imagestream:tag] | + +**Build Status:** +- Phase: [phase] +- Reason: [reason if failed] +- Message: [message if available] + +**Quick Assessment:** +[Based on status, provide initial assessment - e.g., "Build failed during assemble phase - likely dependency installation issue"] + +Continue with detailed analysis? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 3: Analyze BuildConfig + +Use kubernetes MCP `resources_get` to get BuildConfig: + +```markdown +## BuildConfig Analysis: [buildconfig-name] + +**Source Configuration:** +| Setting | Value | Status | +|---------|-------|--------| +| Git URL | [url] | [OK/WARN: check access] | +| Git Ref | [ref] | [OK/WARN: branch not found] | +| Context Dir | [dir or "/"] | [OK] | +| Source Secret | [secret-name or "None"] | [OK/MISSING] | + +**Builder Image:** +| Setting | Value | Status | +|---------|-------|--------| +| Image | [image:tag] | [OK/WARN: check exists] | +| Pull Secret | [secret-name or "None"] | [OK/MISSING] | + +**Output Configuration:** +| Setting | Value | Status | +|---------|-------|--------| +| Output To | [ImageStreamTag] | [OK] | +| Push Secret | [secret-name or "None"] | [OK/MISSING] | + +**Environment Variables:** +| Name | Value | Source | +|------|-------|--------| +| [VAR] | [value or "***"] | [Direct/ConfigMap/Secret] | + +**Issues Found:** +- [Issue 1 - e.g., "Source secret 'github-creds' referenced but not found"] +- [Issue 2 - e.g., "Builder image uses older tag, may have compatibility issues"] + +Continue to view build logs? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 4: Get Build Pod Logs + +Use kubernetes MCP `pod_logs` for the builder pod: + +```markdown +## Build Logs: [build-name] + +**Build Phases:** +| Phase | Status | Duration | +|-------|--------|----------| +| Fetching source | [Complete/Failed] | [duration] | +| Pulling builder image | [Complete/Failed] | [duration] | +| Assemble | [Complete/Failed] | [duration] | +| Commit | [Complete/Failed] | [duration] | +| Push | [Complete/Failed] | [duration] | + +**Failed Phase: [phase-name]** + +``` +[Last 100 lines of build logs, focused on the failing phase] +``` + +**Log Analysis:** + +[Analyze logs and identify errors:] + +**Errors Found:** +- Line [X]: [error description - e.g., "npm ERR! 404 Not Found - package 'nonexistent@1.0.0'"] +- Line [Y]: [error description - e.g., "error: unable to resolve 'github.com/private/repo'"] + +**S2I Phase Explanation:** + +[For S2I builds, explain what the failed phase does:] +- **assemble**: Installs dependencies and builds application +- **commit**: Creates the final container image layer +- **push**: Pushes image to internal registry + +Continue to check related resources? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 5: Check Related Resources + +Check secrets, imagestreams, and source access: + +```markdown +## Related Resources Analysis + +**ImageStreams:** +| ImageStream | Tags | Last Updated | Status | +|-------------|------|--------------|--------| +| [app] | [latest, v1.0] | [timestamp] | [OK] | +| [builder] | [imported] | [timestamp] | [OK/MISSING] | + +**Secrets:** +| Secret | Type | Used By | Status | +|--------|------|---------|--------| +| [source-secret] | kubernetes.io/basic-auth | Source | [OK/MISSING] | +| [push-secret] | kubernetes.io/dockerconfigjson | Output | [OK/MISSING] | + +**Source Repository Access:** +[If GitHub MCP available, check if source URL is accessible] +- URL: [git-url] +- Status: [Accessible/401 Unauthorized/404 Not Found/Timeout] + +**Registry Access:** +[Check if internal registry is accessible] +- Registry: image-registry.openshift-image-registry.svc:5000 +- Status: [OK/Unreachable] + +**Issues Found:** +- [Issue 1 - e.g., "Secret 'github-token' missing - cannot authenticate to private repo"] +- [Issue 2 - e.g., "Builder ImageStreamTag 'nodejs:18' not imported"] + +Continue to full diagnosis summary? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 6: Present Diagnosis Summary + +```markdown +## Diagnosis Summary: [build-name] + +### Root Cause + +**Primary Issue:** [Categorized root cause] + +| Category | Status | Details | +|----------|--------|---------| +| Source Access | [OK/FAIL] | [details] | +| Builder Image | [OK/FAIL] | [details] | +| Dependencies | [OK/FAIL] | [details] | +| Build Script | [OK/FAIL] | [details] | +| Registry Push | [OK/FAIL] | [details] | + +### Detailed Findings + +**[Category 1: e.g., Dependency Installation]** +- Problem: [specific problem - e.g., "npm package 'lodash@99.0.0' does not exist"] +- Evidence: [from build logs] +- Impact: [build fails during assemble phase] + +**[Category 2: e.g., Source Authentication]** +- Problem: [specific problem] +- Evidence: [from events/logs] +- Impact: [cannot clone repository] + +### Recommended Actions + +1. **[Action 1]** - [description] + ```bash + [command to fix - e.g., oc create secret generic github-token --from-literal=...] + ``` + +2. **[Action 2]** - [description] + ```bash + [command to fix - e.g., oc import-image nodejs:18 --from=registry.access.redhat.com/ubi9/nodejs-18] + ``` + +3. **[Action 3]** - [description] + +### Retry Build + +After fixing the issue: +```bash +# Start a new build +oc start-build [buildconfig-name] -n [namespace] + +# Or start build with follow +oc start-build [buildconfig-name] -n [namespace] --follow +``` + +--- + +Would you like me to: +1. Execute one of the recommended fixes +2. Retry the build +3. Compare with last successful build +4. Debug the build pod (/debug-pod) +5. Exit debugging + +Select an option: +``` + +**WAIT for user to select next action.** + +## Build Failure Categories + +### Common S2I Build Failures + +| Phase | Failure Type | Key Indicators | Common Fix | +|-------|--------------|----------------|------------| +| **fetch-source** | Auth failure | 401/403 in logs | Add/fix source secret | +| **fetch-source** | Repo not found | 404 in logs | Check git URL | +| **pull-builder** | Image not found | ImagePullBackOff | Import builder image | +| **pull-builder** | Auth failure | unauthorized | Add pull secret | +| **assemble** | Dependency error | npm ERR, pip error | Fix package.json/requirements | +| **assemble** | Build script fail | Non-zero exit | Check application code | +| **assemble** | Out of memory | OOMKilled | Increase build resources | +| **push** | Registry auth | unauthorized | Check push secret | +| **push** | Registry full | quota exceeded | Clean up old images | + +### Python S2I Specific Issues + +| Issue | Symptom | Solution | +|-------|---------|----------| +| Wrong entry point | "No module named app" | Set APP_MODULE or rename to app.py | +| Missing gunicorn | "gunicorn: command not found" | Add gunicorn to requirements.txt | +| Version mismatch | Import errors | Match Python version to builder | + +See [docs/python-s2i-entrypoints.md](../../docs/python-s2i-entrypoints.md) for detailed Python guidance. + +### Node.js S2I Specific Issues + +| Issue | Symptom | Solution | +|-------|---------|----------| +| Build script missing | "npm run build" fails | Add build script or set NPM_RUN | +| Node version mismatch | Syntax errors | Match engines.node in package.json | +| Private npm registry | 401 Unauthorized | Configure .npmrc or NPM_MIRROR | + +## MCP Tools Used + +| Tool | Purpose | +|------|---------| +| `resources_list` | List builds, find failed builds | +| `resources_get` | Get Build spec, BuildConfig details | +| `pod_logs` | Get build pod logs | +| `events_list` | Get build events | +| `resources_list` | Check secrets, imagestreams | +| `get_file_contents` (github) | Verify source repository access | + +## Output Variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `BUILD_NAME` | Debugged build name | `myapp-1` | +| `BUILDCONFIG_NAME` | Associated BuildConfig | `myapp` | +| `BUILD_NAMESPACE` | Namespace | `my-project` | +| `FAILURE_PHASE` | Phase where build failed | `assemble` | +| `FAILURE_CATEGORY` | Categorized failure type | `DependencyError` | +| `ROOT_CAUSE` | Identified root cause | `npm package not found` | + +## Dependencies + +### Required MCP Servers +- `openshift` (kubernetes MCP server) +- `github` (optional, for source repository verification) + +### Related Skills +- `/s2i-build` - To retry build after fixing issues +- `/debug-pod` - To debug the builder pod directly +- `/detect-project` - To re-analyze project and builder image selection + +## Reference Documentation + +For detailed guidance, see: +- [docs/builder-images.md](../../docs/builder-images.md) - S2I builder image selection, version mapping +- [docs/python-s2i-entrypoints.md](../../docs/python-s2i-entrypoints.md) - Python APP_MODULE configuration +- [docs/debugging-patterns.md](../../docs/debugging-patterns.md) - Common error patterns +- [docs/prerequisites.md](../../docs/prerequisites.md) - Required tools (oc), cluster access verification diff --git a/rh-developer/skills/debug-container/SKILL.md b/rh-developer/skills/debug-container/SKILL.md new file mode 100644 index 00000000..92d5b146 --- /dev/null +++ b/rh-developer/skills/debug-container/SKILL.md @@ -0,0 +1,437 @@ +--- +name: debug-container +description: | + Diagnose local container issues with Podman/Docker including image pull errors, container startup failures, OOM kills, and networking problems. Automates multi-step diagnosis: container inspect, logs retrieval, image analysis, and resource constraint checking. Use this skill when containers fail to run locally before deployment. Triggers on /debug-container command or phrases like "container won't start", "podman run fails", "local container crashing", "container exits immediately". +user_invocable: true +--- + +# /debug-container Skill + +Diagnose local Podman/Docker container issues by automatically gathering container status, logs, and configuration. + +## Overview + +``` +[Identify Container] → [Inspect] → [Logs] → [Image Analysis] → [Resource Check] → [Summary] +``` + +**This skill diagnoses:** +- Container startup failures +- Immediate exit (exit codes) +- OOM kills +- Image pull errors +- Entrypoint/CMD issues +- Volume mount problems + +## Prerequisites + +1. Podman or Docker installed locally +2. Container or image name is known + +## Critical: Human-in-the-Loop Requirements + +See [Human-in-the-Loop Requirements](../../docs/human-in-the-loop.md) for mandatory checkpoint behavior. + +**IMPORTANT:** This skill requires explicit user confirmation at each step. You MUST: +1. **Wait for user confirmation** before executing diagnostic actions +2. **Do NOT proceed** to the next step until the user explicitly approves +3. **Present findings clearly** and ask if user wants deeper analysis +4. **Never auto-execute** remediation actions without user approval + +If the user says "no" or wants to focus on specific areas, address their concerns before proceeding. + +## Trigger + +- User types `/debug-container` +- User says "container won't start", "podman run fails" +- User says "container exits immediately", "container crashing" +- User says "can't pull image", "image not found" +- User says "OOM", "out of memory", "exit code 137" + +## Input Parameters + +| Parameter | Description | Default | +|-----------|-------------|---------| +| `CONTAINER` | Container name or ID | Auto-detect from recent | +| `IMAGE` | Image name to debug | From container | + +## Workflow + +### Step 1: Identify Target Container + +```markdown +## Container Debugging + +What would you like me to debug? + +1. **Running/stopped container** - Debug an existing container +2. **Failed container run** - Debug a recent failed `podman run` +3. **Image issue** - Debug image pull or build problems +4. **List containers** - Show all containers (including stopped) + +Select an option or enter a container name/ID: +``` + +**WAIT for user response.** Do NOT proceed until user identifies the target. + +If user selects "List containers": +Use Podman MCP `container_list`: + +```markdown +## Containers + +| Container ID | Image | Status | Created | Names | +|--------------|-------|--------|---------|-------| +| [abc123] | [myapp:latest] | Exited (1) 5 minutes ago | [time] | [myapp] | +| [def456] | [nginx:latest] | Up 2 hours | [time] | [webserver] | +| [ghi789] | [postgres:15] | Exited (137) 1 hour ago | [time] | [db] | + +Which container would you like me to debug? +``` + +**WAIT for user to select a container.** + +### Step 2: Inspect Container + +Use Podman MCP `container_inspect`: + +```markdown +## Container Inspection: [container-name] + +**Basic Info:** +| Field | Value | +|-------|-------| +| ID | [full-id] | +| Image | [image:tag] | +| Created | [timestamp] | +| Status | [running/exited/created] | + +**State:** +| Field | Value | +|-------|-------| +| Running | [true/false] | +| Paused | [true/false] | +| Restarting | [true/false] | +| OOMKilled | [true/false] | +| Exit Code | [code] | +| Error | [error message or empty] | +| Started At | [timestamp] | +| Finished At | [timestamp] | + +**Configuration:** +| Setting | Value | +|---------|-------| +| Entrypoint | [entrypoint] | +| Cmd | [command] | +| Working Dir | [workdir] | +| User | [user or root] | + +**Port Mappings:** +| Container Port | Host Binding | +|----------------|--------------| +| [8080/tcp] | [0.0.0.0:8080] | + +**Volume Mounts:** +| Source | Destination | Mode | +|--------|-------------|------| +| [/host/path] | [/container/path] | [rw/ro] | + +**Environment Variables:** +| Name | Value | +|------|-------| +| [VAR1] | [value] | +| [VAR2] | [value] | + +**Quick Assessment:** +[Based on state, provide initial assessment - e.g., "Container exited with code 1 - application error. OOMKilled=false, so not a memory issue."] + +Continue with container logs? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 3: Get Container Logs + +Use Podman MCP `container_logs`: + +```markdown +## Container Logs: [container-name] + +**Last 100 lines:** +``` +[container log output] +``` + +**Log Analysis:** + +[Analyze logs and identify errors:] + +**Errors Found:** +- [error 1 - e.g., "Error: Cannot find module 'express'"] +- [error 2 - e.g., "FATAL: password authentication failed for user 'app'"] +- [error 3 - e.g., "bind: address already in use"] + +**Error Categories:** +| Category | Count | First Occurrence | +|----------|-------|------------------| +| Module/Import | [X] | [line] | +| Connection | [Y] | [line] | +| Permission | [Z] | [line] | + +Continue to check image? (yes/no/skip) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 4: Analyze Image + +Use Podman MCP `image_list` to check the image: + +```markdown +## Image Analysis: [image:tag] + +**Image Info:** +| Field | Value | +|-------|-------| +| Repository | [repo] | +| Tag | [tag] | +| ID | [image-id] | +| Created | [timestamp] | +| Size | [size] | + +**Image Layers:** +[If available, show layer info] + +**Image Issues:** +- [Issue 1 - e.g., "Image is 2 years old - may have outdated dependencies"] +- [Issue 2 - e.g., "Using 'latest' tag - version not pinned"] + +**Entrypoint/CMD Check:** + +[Compare image defaults with container override] + +| Setting | Image Default | Container Override | +|---------|---------------|-------------------| +| Entrypoint | [image-entrypoint] | [container-entrypoint or "none"] | +| Cmd | [image-cmd] | [container-cmd or "none"] | + +**Potential Issues:** +- [Issue - e.g., "CMD is empty and no command provided at runtime"] +- [Issue - e.g., "Entrypoint is shell script but container run overrides it"] + +Continue to resource analysis? (yes/no/skip) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 5: Resource Analysis + +```markdown +## Resource Analysis + +**Container Resource Limits:** +| Resource | Limit | Status | +|----------|-------|--------| +| Memory | [512m or unlimited] | [OK/WARNING: OOMKilled] | +| CPU | [1.0 or unlimited] | [OK] | +| PIDs | [unlimited] | [OK] | + +**OOM Analysis:** + +[If OOMKilled=true:] +**Container was killed due to Out of Memory!** + +- Memory limit: [limit] +- Recommendation: Increase memory limit or optimize application + +```bash +# Run with more memory +podman run --memory=1g [image] +``` + +**Port Binding Analysis:** + +[Check if ports conflict:] + +| Port | Requested | Status | +|------|-----------|--------| +| [8080] | 0.0.0.0:8080 | [OK/ERROR: already in use] | + +[If port conflict:] +```bash +# Find process using port +lsof -i :[port] +# Or use different port +podman run -p 8081:8080 [image] +``` + +Continue to diagnosis summary? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 6: Present Diagnosis Summary + +```markdown +## Diagnosis Summary: [container-name] + +### Root Cause + +**Primary Issue:** [Categorized root cause] + +| Category | Status | Details | +|----------|--------|---------| +| Container State | [OK/FAIL] | [exit code, status] | +| Entrypoint/CMD | [OK/FAIL] | [details] | +| Dependencies | [OK/FAIL] | [missing modules] | +| Environment | [OK/FAIL] | [missing vars] | +| Volumes | [OK/FAIL] | [mount issues] | +| Ports | [OK/FAIL] | [binding issues] | +| Memory | [OK/FAIL] | [OOM status] | + +### Detailed Findings + +**[Category 1: e.g., Exit Code 1 - Application Error]** +- Problem: [specific problem - e.g., "Cannot find module 'express'"] +- Evidence: [from logs] +- Impact: [container exits immediately] + +**[Category 2: e.g., Volume Mount Issue]** +- Problem: [specific problem - e.g., "Permission denied on /data"] +- Evidence: [from logs] +- Impact: [application cannot access data] + +### Exit Code Reference + +| Exit Code | Meaning | Your Container | +|-----------|---------|----------------| +| 0 | Success | [match?] | +| 1 | General error | [match?] | +| 126 | Permission problem | [match?] | +| 127 | Command not found | [match?] | +| 137 | SIGKILL (OOM) | [match?] | +| 139 | Segfault | [match?] | +| 143 | SIGTERM | [match?] | + +### Recommended Actions + +1. **[Action 1]** - [description] + ```bash + podman run [fixed-command] + ``` + +2. **[Action 2]** - [description] + ```bash + [command to fix - e.g., podman run --memory=1g ...] + ``` + +3. **[Action 3]** - [description] + +### Test Fix + +```bash +# Remove failed container +podman rm [container-name] + +# Run with fixes applied +podman run [corrected-options] [image] + +# Or run interactively to debug +podman run -it --entrypoint /bin/sh [image] +``` + +--- + +Would you like me to: +1. Execute one of the recommended fixes +2. Run container interactively for debugging +3. Inspect the image layers +4. Remove and recreate the container +5. Exit debugging + +Select an option: +``` + +**WAIT for user to select next action.** + +## Exit Code Reference + +| Exit Code | Signal | Meaning | Common Cause | +|-----------|--------|---------|--------------| +| 0 | - | Success | Normal exit | +| 1 | - | General error | Application error, unhandled exception | +| 2 | - | Misuse of shell | Invalid arguments | +| 126 | - | Permission denied | Cannot execute entrypoint | +| 127 | - | Command not found | Entrypoint binary missing | +| 128+N | Signal N | Killed by signal | See signal table | +| 137 | SIGKILL (9) | Force killed | OOM kill, `podman kill` | +| 139 | SIGSEGV (11) | Segmentation fault | Memory corruption | +| 143 | SIGTERM (15) | Terminated | `podman stop`, graceful shutdown | + +## Common Container Issues + +### Startup Failures + +| Issue | Symptom | Diagnosis | Fix | +|-------|---------|-----------|-----| +| Missing entrypoint | Exit 127 | "executable not found" | Check ENTRYPOINT/CMD | +| Wrong command | Exit 127 | "no such file" | Verify command path | +| Permission denied | Exit 126 | "permission denied" | Check file permissions | +| Missing dependency | Exit 1 | "cannot find module" | Add dependency to image | +| Port conflict | Exit 1 | "address in use" | Use different port | + +### Runtime Issues + +| Issue | Symptom | Diagnosis | Fix | +|-------|---------|-----------|-----| +| OOM killed | Exit 137 | OOMKilled=true | Increase memory limit | +| Volume permission | Exit 1 | "permission denied" | Use :Z/:z labels or fix perms | +| Missing env var | Exit 1 | "undefined" errors | Add -e VAR=value | +| Network issue | Exit 1 | "connection refused" | Check network mode | + +### SELinux Volume Issues + +On RHEL/Fedora with SELinux, volume mounts may need labels: + +```bash +# Shared label (multiple containers can access) +podman run -v /host/path:/container/path:z [image] + +# Private label (only this container) +podman run -v /host/path:/container/path:Z [image] +``` + +## MCP Tools Used + +| Tool | Purpose | +|------|---------| +| `container_list` | List containers | +| `container_inspect` | Get container details | +| `container_logs` | Get container output | +| `image_list` | Check image info | + +## Output Variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `CONTAINER_ID` | Container ID | `abc123def456` | +| `CONTAINER_NAME` | Container name | `myapp` | +| `CONTAINER_IMAGE` | Image used | `myapp:latest` | +| `EXIT_CODE` | Exit code | `137` | +| `OOM_KILLED` | OOM status | `true` / `false` | +| `ROOT_CAUSE` | Identified cause | `Out of memory` | + +## Dependencies + +### Required MCP Servers +- `podman` (Podman MCP server) + +### Related Skills +- `/debug-rhel` - For systemd service issues on RHEL hosts +- `/recommend-image` - To select a better base image + +## Reference Documentation + +For detailed guidance, see: +- [docs/debugging-patterns.md](../../docs/debugging-patterns.md) - Common error patterns, exit codes +- [docs/prerequisites.md](../../docs/prerequisites.md) - Required tools (podman) diff --git a/rh-developer/skills/debug-network/SKILL.md b/rh-developer/skills/debug-network/SKILL.md new file mode 100644 index 00000000..ab35e557 --- /dev/null +++ b/rh-developer/skills/debug-network/SKILL.md @@ -0,0 +1,397 @@ +--- +name: debug-network +description: | + Diagnose OpenShift service connectivity issues including DNS resolution, service endpoints, route ingress, and network policies. Automates multi-step diagnosis: service endpoint verification, pod selector matching, route status, and network policy analysis. Use this skill when services can't communicate, routes return 503/502 errors, or external access fails. Triggers on /debug-network command or phrases like "can't reach service", "route returning 503", "pods can't communicate", "no endpoints". +user_invocable: true +--- + +# /debug-network Skill + +Diagnose OpenShift service connectivity issues by automatically checking endpoints, routes, network policies, and pod readiness. + +## Prerequisites + +Before running this skill: +1. User is logged into OpenShift cluster +2. User has access to the target namespace +3. Service, Route, or application name is known + +## Critical: Human-in-the-Loop Requirements + +See [Human-in-the-Loop Requirements](../../docs/human-in-the-loop.md) for mandatory checkpoint behavior. + +**IMPORTANT:** This skill requires explicit user confirmation at each step. You MUST: +1. **Wait for user confirmation** before executing diagnostic actions +2. **Do NOT proceed** to the next step until the user explicitly approves +3. **Present findings clearly** and ask if user wants deeper analysis +4. **Never auto-execute** remediation actions without user approval + +If the user says "no" or wants to focus on specific areas, address their concerns before proceeding. + +## Trigger + +- User types `/debug-network` +- User says "can't reach service", "service not working" +- User says "route returning 503", "502 Bad Gateway" +- User says "pods can't communicate", "network timeout" +- User says "no endpoints", "service has no backends" + +## Input Parameters + +| Parameter | Description | Default | +|-----------|-------------|---------| +| `SERVICE_NAME` | Service to debug | Auto-detect | +| `ROUTE_NAME` | Route to debug | Same as service | +| `NAMESPACE` | Target namespace | Current namespace | + +## Workflow + +### Step 1: Identify Target Service + +```markdown +## Network Debugging + +**Current OpenShift Context:** +- Cluster: [cluster] +- Namespace: [namespace] + +What connectivity issue would you like me to debug? + +1. **Service connectivity** - Internal service-to-service communication +2. **Route/Ingress** - External access to application +3. **Specify service name** - Debug a specific service +4. **List services** - Show services in current namespace + +Select an option or enter a service name: +``` + +**WAIT for user response.** Do NOT proceed until user identifies the target. + +If user selects "List services": +Use kubernetes MCP `resources_list` for services: + +```markdown +## Services in [namespace] + +| Service | Type | Cluster IP | Ports | Endpoints | +|---------|------|------------|-------|-----------| +| [app-service] | ClusterIP | [ip] | [8080/TCP] | [2 ready] | +| [db-service] | ClusterIP | [ip] | [5432/TCP] | [0 - no endpoints!] | +| [api-service] | ClusterIP | [ip] | [3000/TCP] | [1 ready] | + +Which service would you like me to debug? +``` + +**WAIT for user to select a service.** + +### Step 2: Check Service and Endpoints + +Use kubernetes MCP `resources_get` for Service and Endpoints: + +```markdown +## Service Analysis: [service-name] + +**Service Configuration:** +| Field | Value | +|-------|-------| +| Type | [ClusterIP/NodePort/LoadBalancer] | +| Cluster IP | [ip] | +| Ports | [port-mappings] | +| Selector | [label-selector] | + +**Endpoints:** +| Subset | Addresses | Ports | Status | +|--------|-----------|-------|--------| +| [subset] | [pod-ip-1, pod-ip-2] | [port] | [Ready] | + +[If no endpoints:] +**WARNING: Service has NO endpoints!** + +This means no pods match the service selector, or matching pods are not ready. + +**Service Selector:** `app=[value], tier=[value]` + +**Quick Assessment:** +[Based on endpoints status, provide initial assessment] + +Continue with pod analysis? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 3: Verify Backend Pods + +Use kubernetes MCP `pod_list` with label selector matching service: + +```markdown +## Backend Pods for Service: [service-name] + +**Service Selector:** `[selector-labels]` + +**Matching Pods:** +| Pod | Status | Ready | IP | Node | +|-----|--------|-------|-----|------| +| [pod-1] | Running | 1/1 | [ip] | [node] | +| [pod-2] | Running | 0/1 | [ip] | [node] | +| [pod-3] | CrashLoopBackOff | 0/1 | [ip] | [node] | + +**Readiness Analysis:** +| Pod | Readiness Probe | Last Check | Status | +|-----|-----------------|------------|--------| +| [pod-1] | HTTP GET :8080/ | [time] | Passing | +| [pod-2] | HTTP GET :8080/ | [time] | Failing - Connection refused | +| [pod-3] | HTTP GET :8080/ | [time] | Failing - Container not running | + +[If selector mismatch:] +**WARNING: Label Mismatch Detected!** + +Service selector: `app=myapp` +Pod labels: `app=my-app` (hyphen difference!) + +**Issues Found:** +- [Issue 1 - e.g., "Pod [pod-2] failing readiness probe - application not listening on port 8080"] +- [Issue 2 - e.g., "Pod [pod-3] is in CrashLoopBackOff - run /debug-pod for details"] + +Continue to check Route? (yes/no/skip) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 4: Check Route Status + +Use kubernetes MCP `resources_get` for Route: + +```markdown +## Route Analysis: [route-name] + +**Route Configuration:** +| Field | Value | +|-------|-------| +| Host | [hostname] | +| Path | [path or "/"] | +| TLS Termination | [edge/passthrough/reencrypt/none] | +| Insecure Policy | [Redirect/Allow/None] | +| Target Service | [service-name] | +| Target Port | [port-name or port-number] | +| Weight | [100] | + +**Route Status:** +| Condition | Status | Reason | Message | +|-----------|--------|--------|---------| +| Admitted | [True/False] | [reason] | [message] | + +[If not admitted:] +**WARNING: Route NOT admitted by router!** + +**Ingress Status:** +| Router | Admitted | Host | Conditions | +|--------|----------|------|------------| +| [default] | [True/False] | [host] | [conditions] | + +**TLS Configuration:** +| Setting | Value | +|---------|-------| +| Certificate | [Provided/Default/None] | +| Key | [Provided/None] | +| CA Certificate | [Provided/None] | +| Destination CA | [Provided/None] (for reencrypt) | + +**Issues Found:** +- [Issue 1 - e.g., "Route not admitted - hostname conflicts with existing route"] +- [Issue 2 - e.g., "TLS termination is 'passthrough' but backend is HTTP only"] + +Continue to check Network Policies? (yes/no/skip) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 5: Analyze Network Policies + +Use kubernetes MCP `resources_list` for NetworkPolicy: + +```markdown +## Network Policy Analysis + +**NetworkPolicies in [namespace]:** +| Policy | Pod Selector | Ingress Rules | Egress Rules | +|--------|--------------|---------------|--------------| +| [policy-1] | app=myapp | [2 rules] | [Allow all] | +| [policy-2] | tier=backend | [1 rule] | [1 rule] | +| [default-deny] | {} (all pods) | [Deny all] | [Allow all] | + +**Policies Affecting [service-name] Pods:** + +**Policy: [policy-name]** +```yaml +ingress: +- from: + - podSelector: + matchLabels: + app: frontend + ports: + - port: 8080 + protocol: TCP +``` + +**Analysis:** +- Pods with `app=myapp` only accept traffic from pods with `app=frontend` +- Traffic from other namespaces is BLOCKED +- Traffic on ports other than 8080 is BLOCKED + +**Potential Blocking:** +- [Issue 1 - e.g., "Source pods have label 'app=web' but policy requires 'app=frontend'"] +- [Issue 2 - e.g., "Cross-namespace traffic blocked - no namespaceSelector in policy"] + +Continue to diagnosis summary? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 6: Present Diagnosis Summary + +```markdown +## Network Diagnosis Summary: [service-name] + +### Connectivity Path + +``` +[Source] → [Service] → [Endpoints] → [Pod] + OK → OK → [STATUS] → [STATUS] + +[External] → [Route] → [Router] → [Service] → [Pod] + OK → OK → OK → OK → [STATUS] +``` + +### Root Cause + +**Primary Issue:** [Categorized root cause] + +| Component | Status | Details | +|-----------|--------|---------| +| Service | [OK/FAIL] | [details] | +| Endpoints | [OK/FAIL] | [count] ready | +| Pod Readiness | [OK/FAIL] | [X/Y] pods ready | +| Route | [OK/FAIL] | [details] | +| Network Policy | [OK/WARN] | [details] | +| DNS | [OK/FAIL] | [details] | + +### Detailed Findings + +**[Category 1: e.g., No Endpoints]** +- Problem: [specific problem - e.g., "Service selector doesn't match any pods"] +- Evidence: [selector vs pod labels] +- Impact: [all traffic to service fails] + +**[Category 2: e.g., Readiness Probe Failing]** +- Problem: [specific problem] +- Evidence: [probe configuration and failure reason] +- Impact: [pod removed from endpoints] + +### Recommended Actions + +1. **[Action 1]** - [description] + ```bash + [command to fix - e.g., oc label pod myapp-xxx app=myapp --overwrite] + ``` + +2. **[Action 2]** - [description] + ```bash + [command to fix - e.g., oc patch svc myapp -p '{"spec":{"selector":{"app":"my-app"}}}'] + ``` + +3. **[Action 3]** - [description] + +### Test Connectivity + +After fixing, verify with: +```bash +# Test internal connectivity from another pod +oc run test-curl --rm -i --tty --image=curlimages/curl -- \ + curl -v http://[service-name].[namespace].svc.cluster.local:[port] + +# Test route externally +curl -v https://[route-host] + +# Check endpoints +oc get endpoints [service-name] -n [namespace] +``` + +--- + +Would you like me to: +1. Execute one of the recommended fixes +2. Test connectivity from a debug pod +3. Debug specific pods (/debug-pod) +4. Check DNS resolution +5. Exit debugging + +Select an option: +``` + +**WAIT for user to select next action.** + +## Common Connectivity Issues + +### Service Issues + +| Issue | Symptom | Diagnosis | Fix | +|-------|---------|-----------|-----| +| No endpoints | Connection refused | Empty endpoints list | Fix selector or pod labels | +| Selector mismatch | Some pods missing | Compare selector to labels | Update selector or labels | +| Wrong port | Connection refused | Check targetPort | Update service port mapping | +| Pod not ready | Intermittent failures | Readiness probe failing | Fix application or probe | + +### Route Issues + +| Issue | Symptom | Diagnosis | Fix | +|-------|---------|-----------|-----| +| 503 Service Unavailable | No healthy backends | Check endpoints | Ensure pods are ready | +| 502 Bad Gateway | Backend connection error | Pod crash or wrong port | Debug pod or fix port | +| 404 Not Found | Route not admitted | Check route status | Fix hostname conflict | +| TLS errors | Certificate issues | Check TLS config | Update certificates | +| Host not found | DNS issue | Check route host | Verify wildcard DNS | + +### Network Policy Issues + +| Issue | Symptom | Diagnosis | Fix | +|-------|---------|-----------|-----| +| Ingress blocked | Connection timeout | Check policy rules | Add ingress rule | +| Egress blocked | Can't reach external | Check egress rules | Add egress rule | +| Cross-namespace blocked | Namespace isolation | Check namespaceSelector | Add namespace rule | +| Port blocked | Specific port fails | Check port rules | Add port to policy | + +## MCP Tools Used + +| Tool | Purpose | +|------|---------| +| `resources_get` | Get Service, Route, NetworkPolicy details | +| `resources_list` | List services, endpoints, pods, policies | +| `pod_list` | Check pod status and labels | +| `events_list` | Get route/service events | + +## Output Variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `SERVICE_NAME` | Debugged service | `myapp` | +| `SERVICE_NAMESPACE` | Namespace | `my-project` | +| `HAS_ENDPOINTS` | Endpoints exist | `true` / `false` | +| `ENDPOINTS_READY` | Ready endpoint count | `2` | +| `ROUTE_ADMITTED` | Route status | `true` / `false` | +| `NETWORK_POLICY_BLOCKING` | Policy blocking traffic | `true` / `false` | +| `ROOT_CAUSE` | Identified root cause | `Selector mismatch` | + +## Dependencies + +### Required MCP Servers +- `openshift` (kubernetes MCP server) + +### Related Skills +- `/debug-pod` - To debug specific backend pods +- `/deploy` - To fix and redeploy the service + +## Reference Documentation + +For detailed guidance, see: +- [docs/debugging-patterns.md](../../docs/debugging-patterns.md) - Common error patterns +- [docs/prerequisites.md](../../docs/prerequisites.md) - Required tools (oc), cluster access verification diff --git a/rh-developer/skills/debug-pod/SKILL.md b/rh-developer/skills/debug-pod/SKILL.md new file mode 100644 index 00000000..66107525 --- /dev/null +++ b/rh-developer/skills/debug-pod/SKILL.md @@ -0,0 +1,321 @@ +--- +name: debug-pod +description: | + Diagnose pod failures on OpenShift including CrashLoopBackOff, ImagePullBackOff, OOMKilled, and pending pods. Automates multi-step diagnosis: pod status, events, logs (current + previous), and resource constraint analysis. Use this skill when pods are not running, restarting frequently, or stuck in non-ready states. Triggers on /debug-pod command or phrases like "my pod is crashing", "pod won't start", "CrashLoopBackOff", "ImagePullBackOff", "OOMKilled". +user_invocable: true +--- + +# /debug-pod Skill + +Diagnose pod failures on OpenShift by automatically gathering status, events, logs, and resource information. + +## Prerequisites + +Before running this skill: +1. User is logged into OpenShift cluster +2. User has access to the target namespace +3. Pod or deployment name is known (or can be identified from recent deployments) + +## Critical: Human-in-the-Loop Requirements + +See [Human-in-the-Loop Requirements](../../docs/human-in-the-loop.md) for mandatory checkpoint behavior. + +**IMPORTANT:** This skill requires explicit user confirmation at each step. You MUST: +1. **Wait for user confirmation** before executing diagnostic actions +2. **Do NOT proceed** to the next step until the user explicitly approves +3. **Present findings clearly** and ask if user wants deeper analysis +4. **Never auto-execute** remediation actions without user approval + +If the user says "no" or wants to focus on specific areas, address their concerns before proceeding. + +## Trigger + +- User types `/debug-pod` +- User says "my pod is crashing", "pod won't start", "CrashLoopBackOff" +- User says "ImagePullBackOff", "OOMKilled", "pod stuck pending" +- User says "container terminated", "pod restarting" + +## Input Parameters + +| Parameter | Description | Default | +|-----------|-------------|---------| +| `POD_NAME` | Name of pod to debug | Auto-detect from failed deployment | +| `NAMESPACE` | Target namespace | Current namespace | +| `CONTAINER` | Specific container (for multi-container pods) | All containers | + +## Workflow + +### Step 1: Identify Target Pod + +```markdown +## Pod Debugging + +**Current OpenShift Context:** +- Cluster: [cluster] +- Namespace: [namespace] + +Which pod would you like me to debug? + +1. **Specify pod name** - Enter the pod name directly +2. **List failing pods** - Show pods with issues in current namespace +3. **From deployment** - Debug pods from a specific deployment + +Select an option or enter a pod name: +``` + +**WAIT for user response.** Do NOT proceed until user identifies the target pod. + +If user selects "List failing pods": +Use kubernetes MCP `pod_list` with namespace, then filter to show pods NOT in Running/Succeeded state: + +```markdown +## Pods with Issues in [namespace] + +| Pod | Status | Restarts | Age | Reason | +|-----|--------|----------|-----|--------| +| [pod-name] | CrashLoopBackOff | 5 | 10m | [waiting reason] | +| [pod-name-2] | ImagePullBackOff | 0 | 3m | [waiting reason] | +| [pod-name-3] | Pending | 0 | 15m | [conditions] | + +Which pod would you like me to debug? +``` + +**WAIT for user to select a pod.** + +### Step 2: Get Pod Status Overview + +Use kubernetes MCP `resources_get` to get pod details: + +```markdown +## Pod Status: [pod-name] + +**Basic Info:** +| Field | Value | +|-------|-------| +| Namespace | [namespace] | +| Node | [node-name or "Not scheduled"] | +| Status | [phase: Pending/Running/Failed/Succeeded] | +| IP | [pod-ip or "Not assigned"] | +| Created | [timestamp] | + +**Container Status:** +| Container | State | Ready | Restarts | Exit Code | Reason | +|-----------|-------|-------|----------|-----------|--------| +| [container-name] | [Waiting/Running/Terminated] | [true/false] | [count] | [code or N/A] | [reason] | + +**Quick Assessment:** +[Based on status, provide initial assessment - e.g., "Pod is in CrashLoopBackOff - container keeps crashing after startup"] + +Continue with detailed analysis? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 3: Analyze Events + +Use kubernetes MCP `events_list` filtered by pod: + +```markdown +## Recent Events for [pod-name] + +| Time | Type | Reason | Message | +|------|------|--------|---------| +| [timestamp] | [Normal/Warning] | [reason] | [message] | +| [timestamp] | [Normal/Warning] | [reason] | [message] | +| ... | + +**Event Analysis:** + +[Analyze events and identify key issues:] + +**Issues Found:** +- [Issue 1 - e.g., "FailedScheduling: 0/3 nodes available - insufficient memory"] +- [Issue 2 - e.g., "ImagePullBackOff: unauthorized - check image pull secrets"] + +Continue to view container logs? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 4: Get Container Logs + +Use kubernetes MCP `pod_logs` for current and previous container: + +```markdown +## Container Logs: [container-name] + +**Current Container Logs** (last 50 lines): +``` +[log output] +``` + +[If container has restarted, also show previous logs:] + +**Previous Container Logs** (before last restart): +``` +[log output from --previous] +``` + +**Log Analysis:** + +[Analyze logs and identify errors:] + +**Errors Found:** +- Line [X]: [error description - e.g., "Connection refused to database on port 5432"] +- Line [Y]: [error description - e.g., "Out of memory - heap allocation failed"] + +Continue to analyze resource constraints? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 5: Analyze Resource Constraints + +Check resource requests, limits, and actual usage: + +```markdown +## Resource Analysis: [pod-name] + +**Container: [container-name]** + +| Resource | Request | Limit | Status | +|----------|---------|-------|--------| +| Memory | [128Mi] | [512Mi] | [OK / WARNING: OOMKilled] | +| CPU | [100m] | [500m] | [OK / WARNING: throttled] | + +**Node Resources (if scheduled):** +| Resource | Allocatable | Allocated | Available | +|----------|-------------|-----------|-----------| +| Memory | [8Gi] | [7.5Gi] | [512Mi] | +| CPU | [4000m] | [3800m] | [200m] | + +**Resource Issues:** +- [Issue 1 - e.g., "Container was OOMKilled - memory limit too low for application"] +- [Issue 2 - e.g., "Pod cannot be scheduled - no nodes have 2Gi available memory"] + +Continue to full diagnosis summary? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Step 6: Present Diagnosis Summary + +```markdown +## Diagnosis Summary: [pod-name] + +### Root Cause + +**Primary Issue:** [Categorized root cause] + +| Category | Status | Details | +|----------|--------|---------| +| Container Start | [OK/FAIL] | [details] | +| Image Pull | [OK/FAIL] | [details] | +| Resource Scheduling | [OK/FAIL] | [details] | +| Application Health | [OK/FAIL] | [details] | +| Volume Mounts | [OK/FAIL] | [details] | + +### Detailed Findings + +**[Category 1: e.g., Image Pull Issues]** +- Problem: [specific problem] +- Evidence: [from events/logs] +- Impact: [how this affects the pod] + +**[Category 2: e.g., Application Crash]** +- Problem: [specific problem] +- Evidence: [from logs] +- Impact: [how this affects the pod] + +### Recommended Actions + +1. **[Action 1]** - [description] + ```bash + [command to fix - e.g., oc create secret docker-registry...] + ``` + +2. **[Action 2]** - [description] + ```bash + [command to fix - e.g., oc set resources deployment/app --limits=memory=1Gi] + ``` + +3. **[Action 3]** - [description] + +### Related Documentation + +- [Link to relevant Red Hat KB article if applicable] +- [Link to OpenShift docs for the specific issue] + +--- + +Would you like me to: +1. Execute one of the recommended fixes +2. Dig deeper into a specific area +3. Debug a related resource (Service, Route, ConfigMap) +4. Exit debugging + +Select an option: +``` + +**WAIT for user to select next action.** + +## Error Patterns Reference + +### Common Pod Failure Categories + +| Status | Likely Cause | Key Indicators | +|--------|--------------|----------------| +| **CrashLoopBackOff** | Application crash | Non-zero exit code, error in logs | +| **ImagePullBackOff** | Image access issue | Unauthorized, not found, registry timeout | +| **Pending** | Scheduling issue | Insufficient resources, node selector, taints | +| **OOMKilled** | Memory exhaustion | Exit code 137, memory limit reached | +| **CreateContainerError** | Container config issue | Invalid command, missing secrets/configmaps | +| **Init:Error** | Init container failed | Check init container logs | + +### Exit Code Reference + +| Exit Code | Meaning | Common Cause | +|-----------|---------|--------------| +| 0 | Success | Normal termination | +| 1 | General error | Application error, exception | +| 126 | Permission denied | Cannot execute entrypoint | +| 127 | Command not found | Invalid entrypoint/command | +| 137 | SIGKILL (OOM) | Memory limit exceeded | +| 139 | SIGSEGV | Segmentation fault | +| 143 | SIGTERM | Graceful shutdown | + +## MCP Tools Used + +| Tool | Purpose | +|------|---------| +| `pod_list` | List pods, find failing pods | +| `resources_get` | Get pod spec, container status, node info | +| `events_list` | Get pod events for scheduling/pull/mount errors | +| `pod_logs` | Get current and previous container logs | +| `resources_list` | Check related resources (secrets, configmaps, PVCs) | + +## Output Variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `POD_NAME` | Debugged pod name | `myapp-5d4f7b8c9-x2k4l` | +| `POD_NAMESPACE` | Pod namespace | `my-project` | +| `FAILURE_CATEGORY` | Categorized failure type | `OOMKilled`, `ImagePull`, `Scheduling` | +| `ROOT_CAUSE` | Identified root cause | `Memory limit 512Mi too low for Java app` | +| `REMEDIATION` | Suggested fix | `Increase memory limit to 1Gi` | + +## Dependencies + +### Required MCP Servers +- `openshift` (kubernetes MCP server) + +### Related Skills +- `/debug-build` - If pod failure is due to bad image from build +- `/debug-network` - If pod is running but service connectivity fails +- `/deploy` - To redeploy after fixing issues + +## Reference Documentation + +For detailed guidance, see: +- [docs/debugging-patterns.md](../../docs/debugging-patterns.md) - Common error patterns and troubleshooting trees +- [docs/prerequisites.md](../../docs/prerequisites.md) - Required tools (oc), cluster access verification diff --git a/rh-developer/skills/debug-rhel/SKILL.md b/rh-developer/skills/debug-rhel/SKILL.md new file mode 100644 index 00000000..be720349 --- /dev/null +++ b/rh-developer/skills/debug-rhel/SKILL.md @@ -0,0 +1,465 @@ +--- +name: debug-rhel +description: | + Diagnose RHEL system issues including systemd service failures, SELinux denials, firewall blocking, and system resource problems. Automates multi-step diagnosis: journalctl log analysis, SELinux denial detection (ausearch), firewall rule inspection, and systemd unit status. Use this skill when applications fail on standalone RHEL/Fedora/CentOS hosts deployed via /rhel-deploy. Triggers on /debug-rhel command or phrases like "service won't start on RHEL", "SELinux blocking", "systemd failed", "firewall blocking". +user_invocable: true +--- + +# /debug-rhel Skill + +Diagnose RHEL system issues by automatically gathering systemd status, journal logs, SELinux denials, and firewall configuration. + +## Overview + +``` +[Connect] → [Identify Service] → [systemd Status] → [Journal Logs] → [SELinux] → [Firewall] → [Summary] +``` + +**This skill diagnoses:** +- systemd service failures +- SELinux access denials (AVC) +- Firewall port blocking +- Permission issues +- Resource constraints + +## Prerequisites + +1. SSH access to target RHEL host +2. sudo privileges on the target host +3. RHEL 8+, CentOS Stream, Rocky Linux, or Fedora + +## Critical: Human-in-the-Loop Requirements + +See [Human-in-the-Loop Requirements](../../docs/human-in-the-loop.md) for mandatory checkpoint behavior. + +**IMPORTANT:** This skill requires explicit user confirmation at each step. You MUST: +1. **Wait for user confirmation** before executing diagnostic commands +2. **Do NOT proceed** to the next step until the user explicitly approves +3. **Present findings clearly** and ask if user wants deeper analysis +4. **Never auto-execute** remediation commands without user approval + +If the user says "no" or wants to focus on specific areas, address their concerns before proceeding. + +## Trigger + +- User types `/debug-rhel` +- User says "service won't start on RHEL", "systemd failed" +- User says "SELinux blocking", "AVC denied" +- User says "firewall blocking", "can't access port" +- User says "permission denied on RHEL" +- After `/rhel-deploy` reports a failure + +## Input Parameters + +| Parameter | Description | Default | +|-----------|-------------|---------| +| `RHEL_HOST` | SSH target (user@host) | From session state | +| `SERVICE_NAME` | systemd service to debug | Auto-detect | + +## Workflow + +### Phase 1: SSH Connection + +```markdown +## RHEL System Debugging + +I'll help you diagnose issues on your RHEL system. + +**SSH Target:** +[If RHEL_HOST in session state from /rhel-deploy:] +- Using previous connection: [user]@[host] + +Is this correct? (yes/no/different host) + +[If no RHEL_HOST:] +Please provide your RHEL host details: + +| Setting | Value | Default | +|---------|-------|---------| +| Host | [required] | - | +| User | [current user] | $USER | +| Port | 22 | 22 | + +**Enter your SSH target:** +``` + +**WAIT for user to confirm or provide host.** + +**Connection verification:** + +```bash +# Test SSH connection +ssh -o BatchMode=yes -o ConnectTimeout=10 [user]@[host] "echo 'Connection successful'" +``` + +If connection fails: +```markdown +**SSH Connection Failed** + +Unable to connect to [host]. + +**Troubleshooting:** +1. Check host is reachable: `ping [host]` +2. Verify SSH key is configured: `ssh-add -l` +3. Check firewall allows SSH: port 22 +4. Verify username is correct + +Would you like to: +1. Try a different host +2. Get help with SSH setup +3. Exit +``` + +### Phase 2: Identify Target Service + +```markdown +## Phase 2: Identify Service + +Which service would you like me to debug? + +1. **Specify service name** - Enter the systemd unit name +2. **List failed services** - Show failed services on the host +3. **From /rhel-deploy** - Debug the last deployed service + +Select an option or enter a service name: +``` + +**WAIT for user response.** + +If user selects "List failed services": + +```bash +# Get failed services +ssh [user]@[host] "systemctl --failed --no-pager" +``` + +```markdown +## Failed Services on [host] + +| Unit | Load | Active | Sub | Description | +|------|------|--------|-----|-------------| +| [myapp.service] | loaded | failed | failed | My Application | +| [other.service] | loaded | failed | failed | Other Service | + +Which service would you like me to debug? +``` + +**WAIT for user to select a service.** + +### Phase 3: Get Service Status + +```bash +# Get detailed service status +ssh [user]@[host] "systemctl status [service] --no-pager -l" +``` + +```markdown +## Service Status: [service-name] + +**Status Overview:** +| Field | Value | +|-------|-------| +| Loaded | [loaded/not-found/masked] | +| Active | [active (running)/inactive (dead)/failed] | +| Main PID | [pid or N/A] | +| Status | [status text] | +| Since | [timestamp] | + +**Recent Activity:** +``` +[systemctl status output - last 10 lines] +``` + +**Quick Assessment:** +[Based on status, provide initial assessment - e.g., "Service failed to start - exit code 1 suggests application error"] + +Continue with journal logs? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Phase 4: Analyze Journal Logs + +```bash +# Get service logs +ssh [user]@[host] "journalctl -u [service] -n 100 --no-pager" +``` + +```markdown +## Journal Logs: [service-name] + +**Last 100 log entries:** +``` +[journalctl output] +``` + +**Log Analysis:** + +[Analyze logs and identify errors:] + +**Errors Found:** +- [timestamp]: [error - e.g., "Permission denied: /var/data/config.yaml"] +- [timestamp]: [error - e.g., "Connection refused: localhost:5432"] +- [timestamp]: [error - e.g., "Port 8080 already in use"] + +**Error Categories:** +| Category | Count | Example | +|----------|-------|---------| +| Permission | [X] | [first occurrence] | +| Connection | [Y] | [first occurrence] | +| Resource | [Z] | [first occurrence] | + +Continue to check SELinux? (yes/no/skip) +``` + +**WAIT for user confirmation before proceeding.** + +### Phase 5: Check SELinux Denials + +```bash +# Check SELinux status +ssh [user]@[host] "getenforce" + +# Get recent AVC denials +ssh [user]@[host] "sudo ausearch -m AVC -ts recent 2>/dev/null || echo 'No recent denials or ausearch not available'" +``` + +```markdown +## SELinux Analysis + +**SELinux Status:** [Enforcing/Permissive/Disabled] + +**Recent AVC Denials:** + +[If denials found:] +| Time | Source | Target | Permission | Denied | +|------|--------|--------|------------|--------| +| [time] | [source_context] | [target_context] | [permission] | [target_file] | +| [time] | [source_context] | [target_context] | [permission] | [target_port] | + +**Denial Analysis:** + +**Denial 1: [description]** +- **What happened:** Process `[name]` tried to [action] on `[target]` +- **Why denied:** SELinux type `[source_type]` cannot [action] `[target_type]` +- **Impact:** [how this affects the application] + +**Recommended Fixes:** + +1. **Set SELinux boolean** (if applicable): + ```bash + sudo setsebool -P [boolean_name] on + ``` + +2. **Change file context** (if file access): + ```bash + sudo semanage fcontext -a -t [correct_type] "[path](/.*)?" + sudo restorecon -Rv [path] + ``` + +3. **Allow port** (if port binding): + ```bash + sudo semanage port -a -t [port_type] -p tcp [port] + ``` + +[If no denials:] +No recent SELinux denials found. SELinux is likely not the issue. + +Continue to check firewall? (yes/no/skip) +``` + +**WAIT for user confirmation before proceeding.** + +### Phase 6: Check Firewall + +```bash +# Get firewall status +ssh [user]@[host] "sudo firewall-cmd --state 2>/dev/null || echo 'firewalld not running'" + +# List firewall rules +ssh [user]@[host] "sudo firewall-cmd --list-all 2>/dev/null" +``` + +```markdown +## Firewall Analysis + +**Firewall Status:** [running/not running] + +**Active Zone:** [zone-name] + +**Current Rules:** +| Type | Value | +|------|-------| +| Services | [ssh, http, https, ...] | +| Ports | [8080/tcp, 3000/tcp, ...] | +| Rich Rules | [count] | + +**Application Port:** [detected-port from logs/config] + +**Port Status:** +| Port | Protocol | Status | +|------|----------|--------| +| [8080] | TCP | [OPEN/BLOCKED] | +| [443] | TCP | [OPEN/BLOCKED] | + +[If port blocked:] +**WARNING: Application port [port] is NOT open in firewall!** + +**To open port:** +```bash +sudo firewall-cmd --permanent --add-port=[port]/tcp +sudo firewall-cmd --reload +``` + +**Or add service:** +```bash +sudo firewall-cmd --permanent --add-service=[service] +sudo firewall-cmd --reload +``` + +Continue to diagnosis summary? (yes/no) +``` + +**WAIT for user confirmation before proceeding.** + +### Phase 7: Present Diagnosis Summary + +```markdown +## Diagnosis Summary: [service-name] on [host] + +### Root Cause + +**Primary Issue:** [Categorized root cause] + +| Category | Status | Details | +|----------|--------|---------| +| Service Unit | [OK/FAIL] | [loaded/enabled status] | +| Application | [OK/FAIL] | [exit code, error] | +| SELinux | [OK/BLOCKED] | [denial count] | +| Firewall | [OK/BLOCKED] | [port status] | +| Permissions | [OK/FAIL] | [file/dir issues] | +| Resources | [OK/FAIL] | [memory/cpu/disk] | + +### Detailed Findings + +**[Category 1: e.g., SELinux Denial]** +- Problem: [specific problem - e.g., "httpd_t cannot bind to port 8080"] +- Evidence: [AVC denial message] +- Impact: [application cannot start] + +**[Category 2: e.g., Missing Dependency]** +- Problem: [specific problem - e.g., "libpq.so.5 not found"] +- Evidence: [error from logs] +- Impact: [application crashes on startup] + +### Recommended Actions + +1. **[Action 1 - Highest Priority]** - [description] + ```bash + ssh [user]@[host] "[command]" + ``` + +2. **[Action 2]** - [description] + ```bash + ssh [user]@[host] "[command]" + ``` + +3. **[Action 3]** - [description] + ```bash + ssh [user]@[host] "[command]" + ``` + +### Verify Fix + +After applying fixes: +```bash +# Restart service +ssh [user]@[host] "sudo systemctl restart [service]" + +# Check status +ssh [user]@[host] "systemctl status [service]" + +# View logs +ssh [user]@[host] "journalctl -u [service] -f" +``` + +--- + +Would you like me to: +1. Execute one of the recommended fixes +2. Dig deeper into a specific area +3. Restart the service +4. View live logs +5. Exit debugging + +Select an option: +``` + +**WAIT for user to select next action.** + +## Common RHEL Issues + +### systemd Service Issues + +| Issue | Symptom | Diagnosis | Fix | +|-------|---------|-----------|-----| +| Unit not found | "not-found" load state | Service file missing | Create or install unit file | +| Exit code 1 | "failed" status | Application error | Check application logs | +| Exit code 126 | Permission issue | Cannot execute | Check ExecStart path/perms | +| Exit code 127 | Command not found | Binary missing | Install dependency | +| Exit code 203 | Exec format error | Wrong architecture | Rebuild for target arch | +| Exit code 217 | User not found | Bad User= directive | Create user or fix unit | + +### SELinux Common Denials + +| Denial Type | Symptom | Common Fix | +|-------------|---------|------------| +| Port binding | Cannot bind to port | `semanage port -a -t http_port_t -p tcp [port]` | +| File read | Cannot read config | `semanage fcontext` + `restorecon` | +| File write | Cannot write data | `semanage fcontext` + `restorecon` | +| Network connect | Cannot connect out | `setsebool -P httpd_can_network_connect on` | +| Container | Podman issues | `setsebool -P container_manage_cgroup on` | + +See [docs/selinux-troubleshooting.md](../../docs/selinux-troubleshooting.md) for detailed guidance. + +### Firewall Issues + +| Issue | Symptom | Fix | +|-------|---------|-----| +| Port not open | Connection refused from outside | `firewall-cmd --add-port=[port]/tcp` | +| Service not enabled | Standard service blocked | `firewall-cmd --add-service=[service]` | +| Zone mismatch | Rules in wrong zone | Check active zone, add to correct zone | +| Rich rule blocking | Specific traffic blocked | Review/remove rich rules | + +## MCP Tools Used + +This skill uses Bash (SSH commands) instead of MCP tools since it operates on remote RHEL hosts. + +## Output Variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `RHEL_HOST` | Target host | `user@192.168.1.100` | +| `SERVICE_NAME` | Debugged service | `myapp.service` | +| `SERVICE_STATUS` | Current status | `failed` | +| `SELINUX_DENIALS` | AVC denial count | `3` | +| `FIREWALL_BLOCKING` | Port blocked | `true` / `false` | +| `ROOT_CAUSE` | Identified root cause | `SELinux port binding denied` | + +## Dependencies + +### Required Tools +- SSH client with key-based authentication +- sudo access on target host + +### Related Skills +- `/rhel-deploy` - To redeploy after fixing issues +- `/debug-container` - To debug Podman containers on the host + +## Reference Documentation + +For detailed guidance, see: +- [docs/selinux-troubleshooting.md](../../docs/selinux-troubleshooting.md) - SELinux denial analysis +- [docs/rhel-deployment.md](../../docs/rhel-deployment.md) - RHEL deployment patterns +- [docs/debugging-patterns.md](../../docs/debugging-patterns.md) - Common error patterns +- [docs/prerequisites.md](../../docs/prerequisites.md) - Required tools and setup diff --git a/rh-developer/skills/deploy/SKILL.md b/rh-developer/skills/deploy/SKILL.md index f93eca04..ab9154f4 100644 --- a/rh-developer/skills/deploy/SKILL.md +++ b/rh-developer/skills/deploy/SKILL.md @@ -273,6 +273,54 @@ Waiting for pods to be ready... [Poll until ready or timeout after 5 minutes] ``` +### Step 6a: Handle Deployment Failure + +If pods do not become ready within the timeout period, or pods are in error states (CrashLoopBackOff, ImagePullBackOff, Pending): + +```markdown +## Deployment Failed + +**Status:** Rollout did not complete successfully + +**Pod Status:** +| Pod | Status | Ready | Restarts | Reason | +|-----|--------|-------|----------|--------| +| [app-name]-xxx-yyy | [CrashLoopBackOff/ImagePullBackOff/Pending] | 0/1 | [count] | [reason] | + +**Events:** +| Time | Type | Message | +|------|------|---------| +| [time] | Warning | [event message] | + +--- + +**Would you like me to diagnose the issue?** + +1. **Debug Pod** - Investigate pod failures (runs `/debug-pod`) + - Analyzes pod status, events, logs, and resource constraints + - Identifies root cause (OOM, image pull issues, crashes, etc.) + +2. **Debug Network** - Investigate connectivity issues (runs `/debug-network`) + - Checks service endpoints, route status, network policies + - Useful if pods are running but service is unreachable + +3. **View logs manually** - Show pod logs without full diagnosis + +4. **Rollback deployment** - Delete created resources and stop + +5. **Continue waiting** - Wait another 5 minutes for rollout + +Select an option: +``` + +**WAIT for user to select an option.** + +- If user selects "Debug Pod" → Invoke `/debug-pod` skill with pod name +- If user selects "Debug Network" → Invoke `/debug-network` skill with service name +- If user selects "View logs" → Show pod logs using `pod_logs` +- If user selects "Rollback" → Delete Deployment, Service, Route +- If user selects "Continue" → Wait another polling cycle + ### Step 7: Deployment Complete ```markdown @@ -334,7 +382,16 @@ Your application is now live! | Create Route | Yes (default: yes) | Yes | | Namespace | Yes (from kubeconfig) | Yes | +## Related Skills + +| Skill | Use When | +|-------|----------| +| `/debug-pod` | Pod failures (CrashLoopBackOff, OOMKilled, ImagePullBackOff) | +| `/debug-network` | Service connectivity issues (no endpoints, 503 errors) | +| `/debug-build` | Build failures before deployment | + ## Reference Documentation For detailed guidance, see: - [docs/prerequisites.md](../docs/prerequisites.md) - Required tools (oc), cluster access verification +- [docs/debugging-patterns.md](../docs/debugging-patterns.md) - Common error patterns and troubleshooting diff --git a/rh-developer/skills/rhel-deploy/SKILL.md b/rh-developer/skills/rhel-deploy/SKILL.md index beaacc78..ae293fc4 100644 --- a/rh-developer/skills/rhel-deploy/SKILL.md +++ b/rh-developer/skills/rhel-deploy/SKILL.md @@ -526,6 +526,53 @@ sudo rm /etc/systemd/system/[app-name].service Your application is live! ``` +### Phase 5a: Handle Deployment Failure + +If the service fails to start or is not accessible: + +```markdown +## Deployment Failed + +The service did not start successfully. + +**Service Status:** +``` +[systemctl status output showing failure] +``` + +**Recent Errors:** +| Time | Error | +|------|-------| +| [time] | [error from journalctl] | + +--- + +**Would you like me to diagnose the issue?** + +1. **Debug RHEL** (`/debug-rhel`) - Full system diagnosis + - Analyzes systemd status, journal logs, SELinux denials, firewall rules + - Identifies root cause and suggests remediation + +2. **Debug Container** (`/debug-container`) - If using container deployment + - Analyzes container state, logs, exit codes + +3. **View full logs** - Show complete journalctl output + +4. **Check SELinux** - Quick SELinux denial check + +5. **Check firewall** - Quick firewall port check + +6. **Stop and clean up** + +Select an option: +``` + +**WAIT for user to select an option.** + +- If user selects "Debug RHEL" → Invoke `/debug-rhel` skill +- If user selects "Debug Container" → Invoke `/debug-container` skill +- After debugging → Offer to retry deployment + ## Delegated Skills This skill delegates to other skills when needed: @@ -539,8 +586,17 @@ When delegating to `/recommend-image`: 2. Receive back: `BUILDER_IMAGE`, `IMAGE_VARIANT`, `SELECTION_RATIONALE` 3. Use `BUILDER_IMAGE` as the FROM image in generated Containerfile +## Related Skills + +| Skill | Use When | +|-------|----------| +| `/debug-rhel` | systemd failures, SELinux denials, firewall blocking | +| `/debug-container` | Container startup issues on RHEL host | + ## Reference Documentation For detailed guidance, see: - [docs/rhel-deployment.md](../docs/rhel-deployment.md) - Comprehensive RHEL deployment reference: systemd unit templates, SELinux configuration, firewall commands, runtime package mapping +- [docs/selinux-troubleshooting.md](../docs/selinux-troubleshooting.md) - SELinux denial analysis and fixes +- [docs/debugging-patterns.md](../docs/debugging-patterns.md) - Common error patterns and troubleshooting - [docs/prerequisites.md](../docs/prerequisites.md) - Required tools (ssh, podman) diff --git a/rh-developer/skills/s2i-build/SKILL.md b/rh-developer/skills/s2i-build/SKILL.md index 492d680e..105a9408 100644 --- a/rh-developer/skills/s2i-build/SKILL.md +++ b/rh-developer/skills/s2i-build/SKILL.md @@ -376,14 +376,20 @@ If build fails: - [relevant troubleshooting tips] **Options:** -1. View full build logs -2. Delete failed build and retry -3. Update BuildConfig and retry -4. Cancel and troubleshoot +1. **Debug Build** (`/debug-build`) - Full build diagnosis + - Analyzes BuildConfig, build logs, source access, registry auth + - Identifies root cause and suggests remediation +2. View full build logs +3. Delete failed build and retry +4. Update BuildConfig and retry +5. Cancel and troubleshoot What would you like to do? ``` +- If user selects "Debug Build" → Invoke `/debug-build` skill with build name +- After debugging → Offer to retry build + ## MCP Tools Used | Tool | Purpose | @@ -415,9 +421,17 @@ On success, these values are available for `/deploy`: | `IMAGESTREAM_TAG` | `[app]:latest` | | `BUILD_NAME` | `[app]-1` | +## Related Skills + +| Skill | Use When | +|-------|----------| +| `/debug-build` | Build failures (source access, dependencies, registry issues) | +| `/deploy` | After successful build, to deploy the image | + ## Reference Documentation For detailed guidance, see: - [docs/builder-images.md](../docs/builder-images.md) - S2I builder image selection, version mapping - [docs/python-s2i-entrypoints.md](../docs/python-s2i-entrypoints.md) - Python APP_MODULE configuration, entry point troubleshooting +- [docs/debugging-patterns.md](../docs/debugging-patterns.md) - Common build error patterns and troubleshooting - [docs/prerequisites.md](../docs/prerequisites.md) - Required tools (oc) From 3cd70cc34b97284b5e1dc9dd4ac27800f2972246 Mon Sep 17 00:00:00 2001 From: ikrispin Date: Mon, 16 Feb 2026 18:10:14 +0200 Subject: [PATCH 2/6] fix: plugin name --- rh-developer/.claude-plugin/plugin.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rh-developer/.claude-plugin/plugin.json b/rh-developer/.claude-plugin/plugin.json index ac66f6e8..5232eb81 100644 --- a/rh-developer/.claude-plugin/plugin.json +++ b/rh-developer/.claude-plugin/plugin.json @@ -1,5 +1,5 @@ { - "name": "Red Hat Developer Agentic Skills Collection", + "name": "rh-developer", "version": "1.0.0", "description": "Plugins for building and deploying applications on Red Hat platforms.", "author": { From b6eca5221b2cf7f9e21775c2fb3c8a69af90d841 Mon Sep 17 00:00:00 2001 From: ikrispin Date: Tue, 17 Feb 2026 11:28:22 +0200 Subject: [PATCH 3/6] make agent prefer MCP tool use other CLI commands --- rh-developer/skills/debug-build/SKILL.md | 6 ++++++ rh-developer/skills/debug-container/SKILL.md | 6 ++++++ rh-developer/skills/debug-network/SKILL.md | 6 ++++++ rh-developer/skills/debug-pod/SKILL.md | 6 ++++++ rh-developer/skills/debug-rhel/SKILL.md | 4 ++++ 5 files changed, 28 insertions(+) diff --git a/rh-developer/skills/debug-build/SKILL.md b/rh-developer/skills/debug-build/SKILL.md index 16b53d75..b4fe4f4d 100644 --- a/rh-developer/skills/debug-build/SKILL.md +++ b/rh-developer/skills/debug-build/SKILL.md @@ -28,6 +28,12 @@ See [Human-in-the-Loop Requirements](../../docs/human-in-the-loop.md) for mandat If the user says "no" or wants to focus on specific areas, address their concerns before proceeding. +## Critical: Prefer MCP Tools + +**IMPORTANT:** Prefer MCP tools over CLI commands for better integration and user experience: +1. **Search for MCP tools first** - Use `ToolSearch` to load OpenShift MCP tools (e.g., `+openshift pods_get`) before diagnostic actions +2. **Use MCP when available** - Prefer `pods_get`, `pods_log`, `events_list`, `resources_get` over `oc`/`kubectl` commands + ## Trigger - User types `/debug-build` diff --git a/rh-developer/skills/debug-container/SKILL.md b/rh-developer/skills/debug-container/SKILL.md index 92d5b146..4a716201 100644 --- a/rh-developer/skills/debug-container/SKILL.md +++ b/rh-developer/skills/debug-container/SKILL.md @@ -40,6 +40,12 @@ See [Human-in-the-Loop Requirements](../../docs/human-in-the-loop.md) for mandat If the user says "no" or wants to focus on specific areas, address their concerns before proceeding. +## Critical: Prefer MCP Tools + +**IMPORTANT:** Prefer MCP tools over CLI commands for better integration and user experience: +1. **Search for MCP tools first** - Use `ToolSearch` to load Podman MCP tools (e.g., `+podman container_inspect`) before diagnostic actions +2. **Use MCP when available** - Prefer `container_list`, `container_inspect`, `container_logs` over `podman`/`docker` commands + ## Trigger - User types `/debug-container` diff --git a/rh-developer/skills/debug-network/SKILL.md b/rh-developer/skills/debug-network/SKILL.md index ab35e557..d18534dd 100644 --- a/rh-developer/skills/debug-network/SKILL.md +++ b/rh-developer/skills/debug-network/SKILL.md @@ -28,6 +28,12 @@ See [Human-in-the-Loop Requirements](../../docs/human-in-the-loop.md) for mandat If the user says "no" or wants to focus on specific areas, address their concerns before proceeding. +## Critical: Prefer MCP Tools + +**IMPORTANT:** Prefer MCP tools over CLI commands for better integration and user experience: +1. **Search for MCP tools first** - Use `ToolSearch` to load OpenShift MCP tools (e.g., `+openshift pods_get`) before diagnostic actions +2. **Use MCP when available** - Prefer `pods_get`, `pods_log`, `events_list`, `resources_get` over `oc`/`kubectl` commands + ## Trigger - User types `/debug-network` diff --git a/rh-developer/skills/debug-pod/SKILL.md b/rh-developer/skills/debug-pod/SKILL.md index 66107525..602f6e6e 100644 --- a/rh-developer/skills/debug-pod/SKILL.md +++ b/rh-developer/skills/debug-pod/SKILL.md @@ -28,6 +28,12 @@ See [Human-in-the-Loop Requirements](../../docs/human-in-the-loop.md) for mandat If the user says "no" or wants to focus on specific areas, address their concerns before proceeding. +## Critical: Prefer MCP Tools + +**IMPORTANT:** Prefer MCP tools over CLI commands for better integration and user experience: +1. **Search for MCP tools first** - Use `ToolSearch` to load OpenShift MCP tools (e.g., `+openshift pods_get`) before diagnostic actions +2. **Use MCP when available** - Prefer `pods_get`, `pods_log`, `events_list`, `resources_get` over `oc`/`kubectl` commands + ## Trigger - User types `/debug-pod` diff --git a/rh-developer/skills/debug-rhel/SKILL.md b/rh-developer/skills/debug-rhel/SKILL.md index be720349..d3984fef 100644 --- a/rh-developer/skills/debug-rhel/SKILL.md +++ b/rh-developer/skills/debug-rhel/SKILL.md @@ -40,6 +40,10 @@ See [Human-in-the-Loop Requirements](../../docs/human-in-the-loop.md) for mandat If the user says "no" or wants to focus on specific areas, address their concerns before proceeding. +## Note: SSH/Bash Required + +This skill operates on **remote RHEL hosts** via SSH, not local MCP servers. Unlike OpenShift/Podman skills, direct Bash commands with SSH are the correct approach here since MCP servers run locally and cannot access remote systems. + ## Trigger - User types `/debug-rhel` From ba42f63cf213fe0719b458db4c73c65e7ae7b176 Mon Sep 17 00:00:00 2001 From: ikrispin Date: Mon, 23 Feb 2026 10:13:55 +0200 Subject: [PATCH 4/6] docs: add troubleshooting skills details --- rh-developer/README.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/rh-developer/README.md b/rh-developer/README.md index cdfd06ff..3e2e7c56 100644 --- a/rh-developer/README.md +++ b/rh-developer/README.md @@ -14,6 +14,22 @@ A Claude Code plugin for building and deploying applications on Red Hat platform | `/rhel-deploy` | Deploy to standalone RHEL/Fedora systems via SSH | | `/containerize-deploy` | End-to-end workflow from source to running app (use if not sure which strategy to choose)) | +### Troubleshooting + +| Command | Description | +| -------------------- | ------------------------------------------------------------------------------------------ | +| `/debug-pod` | Diagnose pod failures on OpenShift (CrashLoopBackOff, ImagePullBackOff, OOMKilled, pending pods) | +| `/debug-build` | Diagnose OpenShift build failures (S2I builds, Docker/Podman builds, BuildConfig issues) | +| `/debug-network` | Diagnose OpenShift service connectivity (DNS, endpoints, routes, network policies) | +| `/debug-container` | Diagnose local Podman/Docker container issues (startup failures, OOM kills, image pull errors) | +| `/debug-rhel` | Diagnose RHEL system issues (systemd failures, SELinux denials, firewall blocking) | + +### Environment + +| Command | Description | +| ------------------------ | -------------------------------------------------------------------------------------- | +| `/validate-environment` | Check required tools and environment setup (oc, helm, podman, git, cluster connectivity) | + ## Prerequisites - OpenShift cluster access (for S2I and OpenShift deployments) From ef6e6f983840e8cd446c4400d9175bac94385f05 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 24 Feb 2026 12:56:17 +0000 Subject: [PATCH 5/6] Initial plan From 883ffdf9a34dc87877082c16d711283e083a5f38 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 24 Feb 2026 13:00:31 +0000 Subject: [PATCH 6/6] fix: move user_invocable to metadata to comply with agentskills.io spec Co-authored-by: dmartinol <86618610+dmartinol@users.noreply.github.com> --- rh-developer/skills/debug-build/SKILL.md | 3 ++- rh-developer/skills/debug-container/SKILL.md | 3 ++- rh-developer/skills/debug-network/SKILL.md | 3 ++- rh-developer/skills/debug-pod/SKILL.md | 3 ++- rh-developer/skills/debug-rhel/SKILL.md | 3 ++- rh-developer/skills/validate-environment/SKILL.md | 3 ++- 6 files changed, 12 insertions(+), 6 deletions(-) diff --git a/rh-developer/skills/debug-build/SKILL.md b/rh-developer/skills/debug-build/SKILL.md index b4fe4f4d..e62c7284 100644 --- a/rh-developer/skills/debug-build/SKILL.md +++ b/rh-developer/skills/debug-build/SKILL.md @@ -2,7 +2,8 @@ name: debug-build description: | Diagnose OpenShift build failures including S2I builds, Docker/Podman builds, and BuildConfig issues. Automates multi-step diagnosis: BuildConfig validation, build pod logs, registry authentication, and source repository access. Use this skill when builds fail, hang, or produce unexpected results. Triggers on /debug-build command or phrases like "build failed", "S2I error", "can't pull builder image", "can't push to registry", "build timeout". -user_invocable: true +metadata: + user_invocable: "true" --- # /debug-build Skill diff --git a/rh-developer/skills/debug-container/SKILL.md b/rh-developer/skills/debug-container/SKILL.md index 4a716201..0669b610 100644 --- a/rh-developer/skills/debug-container/SKILL.md +++ b/rh-developer/skills/debug-container/SKILL.md @@ -2,7 +2,8 @@ name: debug-container description: | Diagnose local container issues with Podman/Docker including image pull errors, container startup failures, OOM kills, and networking problems. Automates multi-step diagnosis: container inspect, logs retrieval, image analysis, and resource constraint checking. Use this skill when containers fail to run locally before deployment. Triggers on /debug-container command or phrases like "container won't start", "podman run fails", "local container crashing", "container exits immediately". -user_invocable: true +metadata: + user_invocable: "true" --- # /debug-container Skill diff --git a/rh-developer/skills/debug-network/SKILL.md b/rh-developer/skills/debug-network/SKILL.md index d18534dd..b084e790 100644 --- a/rh-developer/skills/debug-network/SKILL.md +++ b/rh-developer/skills/debug-network/SKILL.md @@ -2,7 +2,8 @@ name: debug-network description: | Diagnose OpenShift service connectivity issues including DNS resolution, service endpoints, route ingress, and network policies. Automates multi-step diagnosis: service endpoint verification, pod selector matching, route status, and network policy analysis. Use this skill when services can't communicate, routes return 503/502 errors, or external access fails. Triggers on /debug-network command or phrases like "can't reach service", "route returning 503", "pods can't communicate", "no endpoints". -user_invocable: true +metadata: + user_invocable: "true" --- # /debug-network Skill diff --git a/rh-developer/skills/debug-pod/SKILL.md b/rh-developer/skills/debug-pod/SKILL.md index 602f6e6e..ca3cb66a 100644 --- a/rh-developer/skills/debug-pod/SKILL.md +++ b/rh-developer/skills/debug-pod/SKILL.md @@ -2,7 +2,8 @@ name: debug-pod description: | Diagnose pod failures on OpenShift including CrashLoopBackOff, ImagePullBackOff, OOMKilled, and pending pods. Automates multi-step diagnosis: pod status, events, logs (current + previous), and resource constraint analysis. Use this skill when pods are not running, restarting frequently, or stuck in non-ready states. Triggers on /debug-pod command or phrases like "my pod is crashing", "pod won't start", "CrashLoopBackOff", "ImagePullBackOff", "OOMKilled". -user_invocable: true +metadata: + user_invocable: "true" --- # /debug-pod Skill diff --git a/rh-developer/skills/debug-rhel/SKILL.md b/rh-developer/skills/debug-rhel/SKILL.md index d3984fef..cd580999 100644 --- a/rh-developer/skills/debug-rhel/SKILL.md +++ b/rh-developer/skills/debug-rhel/SKILL.md @@ -2,7 +2,8 @@ name: debug-rhel description: | Diagnose RHEL system issues including systemd service failures, SELinux denials, firewall blocking, and system resource problems. Automates multi-step diagnosis: journalctl log analysis, SELinux denial detection (ausearch), firewall rule inspection, and systemd unit status. Use this skill when applications fail on standalone RHEL/Fedora/CentOS hosts deployed via /rhel-deploy. Triggers on /debug-rhel command or phrases like "service won't start on RHEL", "SELinux blocking", "systemd failed", "firewall blocking". -user_invocable: true +metadata: + user_invocable: "true" --- # /debug-rhel Skill diff --git a/rh-developer/skills/validate-environment/SKILL.md b/rh-developer/skills/validate-environment/SKILL.md index 3335846b..74b97072 100644 --- a/rh-developer/skills/validate-environment/SKILL.md +++ b/rh-developer/skills/validate-environment/SKILL.md @@ -2,7 +2,8 @@ name: validate-environment description: | Check and report the status of required tools and environment for rh-developer skills. Validates tool installation (oc, helm, podman, git, skopeo, etc.), cluster connectivity, and permissions. Use this skill before running other deployment skills to ensure prerequisites are met. Triggers on /validate-environment command or when user asks to check their environment setup. -user_invocable: true +metadata: + user_invocable: "true" --- # Validate Environment Skill