diff --git a/rh-virt/README.md b/rh-virt/README.md index 718346f4..5b377762 100644 --- a/rh-virt/README.md +++ b/rh-virt/README.md @@ -9,9 +9,9 @@ OpenShift Virtualization management tools for administering virtual machines on The rh-virt collection provides specialized tools for managing virtual machines in OpenShift Virtualization environments: -- **3 specialized skills** for VM administration tasks +- **5 specialized skills** for complete VM lifecycle management - **OpenShift MCP server integration** for KubeVirt operations -- **VM lifecycle management** from creation to operational monitoring +- **Full VM lifecycle coverage** from creation to deletion with safety-first design ## Quick Start @@ -96,7 +96,7 @@ claude plugin install openshift-virtualization ## Skills -The pack provides 3 specialized skills for common virtualization operations: +The pack provides 5 specialized skills for complete VM lifecycle management: ### 1. **vm-creator** - Virtual Machine Provisioning @@ -161,6 +161,57 @@ List and inspect virtual machines across namespaces with comprehensive status in - Displays resource usage, node placement, and health conditions - Read-only operations with fallback to `oc` CLI if MCP tools unavailable +### 4. **vm-delete** - VM Destruction and Cleanup + +Permanently delete virtual machines and their associated resources with strict safety confirmations. + +**Use when:** +- "Delete VM [name]" +- "Remove virtual machine [name]" +- "Destroy VM [name]" +- "Clean up VM [name]" + +**MCP Tools Used:** +- `resources_delete` (core toolset) - Deletes VirtualMachine, DataVolume, and PVC resources +- `resources_get` (core toolset) - Verifies VM exists and retrieves details +- `resources_list` (core toolset) - Discovers dependent storage resources +- `vm_lifecycle` (kubevirt toolset) - Stops running VMs before deletion + +**What it does:** +- **Permanent VM deletion** with typed confirmation (user must type VM name exactly) +- **Pre-deletion validation** - checks VM exists, running state, dependent resources +- **Protection enforcement** - refuses deletion of VMs with `protected: "true"` label +- **Deletion options** - VM only (preserve storage) or VM + storage (complete cleanup) +- **Graceful shutdown** - stops running VMs before deletion +- **Storage discovery** - identifies and optionally deletes DataVolumes and PVCs +- **Safety-first design** - multiple confirmation steps, clear warnings about data loss +- Requires explicit user confirmation at each critical step (human-in-the-loop) + +### 5. **vm-clone** - VM Cloning and Duplication + +Clone existing virtual machines for testing, scaling, or creating VM templates. + +**Use when:** +- "Clone VM [source] to [target]" +- "Create a copy of VM [name]" +- "Duplicate VM [name] for testing" +- "Create 3 copies of template-vm" + +**MCP Tools Used:** +- `resources_get` (core toolset) - Get source VM configuration +- `resources_create_or_update` (core toolset) - Create cloned VM and storage resources +- `resources_list` (core toolset) - List DataVolumes, PVCs, VMs + +**What it does:** +- **Clone VM configuration** - copies instance type, preferences, network settings, tolerations +- **Flexible storage strategies** - clone storage (full copy), reference existing (shared), or create new empty storage +- **Batch cloning** - create multiple copies in one operation +- **Cross-namespace cloning** - clone VMs between different namespaces +- **Name conflict detection** - verifies target VM name availability +- **Resource impact preview** - shows CPU, memory, storage consumption before cloning +- **Automatic UUID generation** - generates new firmware UUIDs and MAC addresses for clones +- Requires explicit user confirmation and storage strategy selection (human-in-the-loop) + ## MCP Server Integration The pack integrates with the OpenShift MCP server (configured in `.mcp.json`), which provides two toolsets for comprehensive cluster and virtualization management: @@ -269,13 +320,72 @@ User: "Restart the api-server VM" → vm-lifecycle-manager skill restarts the VM ``` -### Workflow 4: Automatic Error Diagnosis and Remediation +### Workflow 4: VM Deletion and Cleanup + +``` +User: "Delete VM test-vm in namespace dev" +→ vm-delete skill validates VM exists +→ Discovers 30Gi DataVolume attached +→ Presents deletion options (VM only vs VM + storage) + +User: "Delete VM + storage" + +Agent: "Type 'test-vm' to confirm permanent deletion: _____" + +User: "test-vm" + +Agent: "Proceed with permanent deletion? (yes/cancel)" + +User: "yes" +→ vm-delete deletes VM and storage +→ Reports 30Gi storage freed +``` + +### Workflow 5: VM Cloning for Test Environment + +``` +User: "Clone production-web to staging-web in namespace staging" +→ vm-clone skill validates source VM exists +→ Discovers 100Gi storage +→ Presents storage cloning options + +Agent: "How should storage be cloned? + 1. Clone Storage (full copy) - 100Gi new allocation + 2. Reference Existing Storage (shared - dangerous) + 3. Create New Empty Storage - 100Gi new allocation + 4. Cancel" + +User: "1" + +→ vm-clone presents complete configuration preview + +Agent: "Clone Configuration Review: + Source: production-web (production namespace) + Target: staging-web (staging namespace) + Storage: Clone Storage (100Gi) + + Proceed with VM cloning? (yes/no)" + +User: "yes" + +→ vm-clone creates DataVolume with PVC clone source +→ Creates cloned VirtualMachine with new UUIDs +→ Monitors storage cloning progress + +Agent: "⏳ Storage cloning in progress... (45%) + ... + ✓ VM Cloned Successfully + Clone completed in 8m15s + VM staging-web ready to start" +``` + +### Workflow 6: Automatic Error Diagnosis and Remediation ``` User: "Create a Fedora VM called test-vm in namespace demo" → vm-creator skill creates the VM → Detects ErrorUnschedulable status -→ Consults troubleshooting.md documentation +→ Consults docs/troubleshooting/scheduling-errors.md for domain knowledge → Diagnoses: Node taints prevent scheduling → Proposes workaround: Add tolerations to VM spec @@ -296,7 +406,7 @@ Agent: "✓ Workaround Applied Successfully **Key Features**: - **Automatic diagnosis**: Detects ErrorUnschedulable and other common errors -- **Documentation consultation**: Reads troubleshooting.md for domain knowledge +- **Documentation consultation**: Reads troubleshooting/INDEX.md and category files for domain knowledge - **Intelligent workarounds**: Proposes fixes for MCP tool limitations - **Human-in-the-loop**: Requires explicit user confirmation before applying patches - **Transparent**: Explains temporary limitations and suggests filing enhancement requests @@ -353,7 +463,7 @@ MCP server is configured in `.mcp.json`: The **vm-creator** skill includes automatic error diagnosis and workaround proposals. When VMs encounter scheduling issues: 1. **Detection**: Skill automatically detects ErrorUnschedulable and other error states -2. **Diagnosis**: Consults `docs/troubleshooting.md` to understand root cause +2. **Diagnosis**: Consults `docs/troubleshooting/INDEX.md` and category files to understand root cause 3. **Investigation**: Executes diagnostic commands (node taints, resource availability, events) 4. **Proposal**: Presents clear diagnosis with workaround options 5. **Remediation**: Applies fix with user confirmation (human-in-the-loop) @@ -362,7 +472,7 @@ The **vm-creator** skill includes automatic error diagnosis and workaround propo - **ErrorUnschedulable** - Node taints/tolerations mismatch, resource constraints, node selector issues - **ErrorDataVolumeNotReady** - Storage provisioning delays, storage class issues, quota exceeded -**For comprehensive troubleshooting guidance**, see [docs/troubleshooting.md](docs/troubleshooting.md). +**For comprehensive troubleshooting guidance**, see [docs/troubleshooting/INDEX.md](docs/troubleshooting/INDEX.md). ### MCP Server Won't Start @@ -405,21 +515,33 @@ rh-virt/ │ └── plugin.json # Plugin metadata ├── .mcp.json # MCP server configuration ├── docs/ # AI-optimized knowledge base -│ └── troubleshooting.md # VM error diagnosis and workarounds +│ └── troubleshooting/ # VM error diagnosis and workarounds (categorized by error type) +│ ├── INDEX.md # Navigation hub for troubleshooting docs +│ ├── scheduling-errors.md # ErrorUnschedulable diagnostics +│ ├── storage-errors.md # Storage provisioning issues +│ ├── lifecycle-errors.md # Start/stop/terminating problems +│ ├── runtime-errors.md # CrashLoopBackOff diagnostics +│ ├── network-errors.md # Network attachment failures +│ └── .ai-index/ # Semantic indexing for AI discovery +│ └── semantic-index.json └── skills/ ├── vm-creator/SKILL.md # VM provisioning with auto-diagnosis ├── vm-lifecycle-manager/SKILL.md # VM power management - └── vm-inventory/SKILL.md # VM discovery and status + ├── vm-inventory/SKILL.md # VM discovery and status + ├── vm-delete/SKILL.md # VM destruction and cleanup + └── vm-clone/SKILL.md # VM cloning and duplication ``` ### Key Patterns - **Skills encapsulate operations** - Each skill handles one category of VM tasks +- **Complete lifecycle coverage** - Create → Clone → Inventory → Lifecycle → Delete - **MCP provides tools** - OpenShift MCP server exposes KubeVirt operations - **Environment-based auth** - KUBECONFIG for secure cluster access - **Automatic diagnosis** - Skills detect errors, consult docs, propose workarounds -- **Document consultation** - Skills read troubleshooting.md for domain knowledge -- **Human-in-the-loop** - User approval required before applying fixes +- **Document consultation** - Skills read troubleshooting/ category files for domain knowledge +- **Human-in-the-loop** - User approval required before critical operations (lifecycle changes, deletion) +- **Safety-first design** - Typed confirmation for destructive operations, protection labels, multi-step validation - **Workaround transparency** - Clear communication of MCP tool limitations and temporary solutions ## Security Model diff --git a/rh-virt/docs/troubleshooting.md b/rh-virt/docs/troubleshooting.md deleted file mode 100644 index f92c9902..00000000 --- a/rh-virt/docs/troubleshooting.md +++ /dev/null @@ -1,440 +0,0 @@ ---- -title: VM Troubleshooting Guide -category: kubevirt -sources: - - title: KubeVirt User Guide - Node Placement - url: https://kubevirt.io/user-guide/virtual_machines/node_placement/ - date_accessed: 2026-02-06 - - title: Kubernetes Taints and Tolerations - url: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ - date_accessed: 2026-02-06 - - title: OpenShift Virtualization - Virtual Machine Status - url: https://docs.openshift.com/container-platform/latest/virt/virtual_machines/virt-managing-vms.html - date_accessed: 2026-02-06 -tags: [troubleshooting, scheduling, taints, tolerations, errors] -semantic_keywords: [ErrorUnschedulable, ErrorDataVolumeNotReady, scheduling failure, node taints, VM status] -use_cases: [vm-creation, diagnostics, error-handling] -last_updated: 2026-02-06 ---- - -# VM Troubleshooting Guide - -## Overview - -This guide provides diagnostic procedures and workarounds for common VirtualMachine errors in OpenShift Virtualization. Use this document when VMs fail to schedule, provision, or start properly. - -## Common VM Status Errors - -### ErrorUnschedulable - -**Symptom**: VM shows status `ErrorUnschedulable` after creation - -**Description**: The Kubernetes scheduler cannot find a suitable node to run the VM's underlying virt-launcher pod. - -**Possible Causes**: - -#### 1. Node Taints (Most Common) - -Nodes have taints that the VM doesn't tolerate. Common in environments with dedicated virtualization infrastructure. - -**Diagnostic Commands**: -```bash -# Check VM events for scheduling failures -oc describe vm -n | grep -A 10 "Events:" - -# Look for messages like: -# "0/X nodes are available: X node(s) had taints that the pod didn't tolerate" - -# Check node taints in the cluster -oc get nodes -o json | jq '.items[] | select(.spec.taints != null) | {name: .metadata.name, taints: .spec.taints}' - -# Alternative: Show taints in table format -oc get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints -``` - -**Common Taint Patterns**: -- `virtualization=true:NoSchedule` - Only VMs with matching toleration can schedule -- `node-role.kubernetes.io/infra:NoSchedule` - Infrastructure-only nodes -- `node.kubernetes.io/not-ready:NoSchedule` - Node not ready for workloads - -**Solution - Add Tolerations to VM**: - -The openshift-virtualization MCP server's `vm_create` tool does NOT currently support the `tolerations` parameter. This requires a manual workaround: - -**Workaround (post-creation patch)**: -```bash -# Patch the VirtualMachine to add tolerations -oc patch vm -n --type=merge -p ' -spec: - template: - spec: - tolerations: - - key: "virtualization" - operator: "Equal" - value: "true" - effect: "NoSchedule" -' - -# Verify tolerations were added -oc get vm -n -o jsonpath='{.spec.template.spec.tolerations}' | jq - -# Check if VM status improved (wait 5-10 seconds) -oc get vm -n -o jsonpath='{.status.printableStatus}' -``` - -**Example - Multiple Tolerations**: -```bash -oc patch vm -n --type=merge -p ' -spec: - template: - spec: - tolerations: - - key: "virtualization" - operator: "Equal" - value: "true" - effect: "NoSchedule" - - key: "dedicated" - operator: "Equal" - value: "virt-workloads" - effect: "NoSchedule" -' -``` - -**Toleration Operators**: -- `Equal` - Key and value must match exactly -- `Exists` - Only key must exist (ignores value) - -**Toleration Effects**: -- `NoSchedule` - Don't schedule new pods (existing pods continue) -- `PreferNoSchedule` - Avoid scheduling if possible -- `NoExecute` - Don't schedule AND evict existing pods - -**Alternative Solutions**: -1. **Remove node taints** (if you have cluster-admin access): - ```bash - oc adm taint nodes virtualization=true:NoSchedule- - ``` - -2. **Use different nodes** - If non-tainted nodes exist, ensure VM fits - -3. **File enhancement request** - Request tolerations support in openshift-mcp-server: - https://github.com/openshift/openshift-mcp-server/issues - -#### 2. Insufficient Resources - -Not enough CPU, memory, or storage available on any node. - -**Diagnostic Commands**: -```bash -# Check VM resource requests -oc get vm -n -o jsonpath='{.spec.template.spec.domain.resources}' - -# Check node resource availability -oc describe nodes | grep -A 5 "Allocated resources" - -# Look for VM events mentioning "Insufficient" -oc describe vm -n | grep "Insufficient" -``` - -**Example Event**: -``` -0/5 nodes are available: 2 Insufficient cpu, 3 Insufficient memory. -``` - -**Solutions**: -1. **Scale cluster** - Add more worker nodes -2. **Reduce VM resources** - Delete and recreate with smaller instance type -3. **Delete unused VMs** - Free up resources -4. **Check resource quotas**: - ```bash - oc describe quota -n - oc describe limitrange -n - ``` - -#### 3. Node Selector Mismatch - -VM requires specific node labels that don't exist in the cluster. - -**Diagnostic Commands**: -```bash -# Check VM node selector requirements -oc get vm -n -o jsonpath='{.spec.template.spec.nodeSelector}' - -# List available node labels -oc get nodes --show-labels - -# Check if any nodes match the selector -oc get nodes -l = -``` - -**Solution**: -Remove node selector or add required labels to nodes: -```bash -# Option 1: Remove node selector from VM -oc patch vm -n --type=json -p '[{"op": "remove", "path": "/spec/template/spec/nodeSelector"}]' - -# Option 2: Add label to nodes -oc label node = -``` - ---- - -### ErrorDataVolumeNotReady - -**Symptom**: VM shows status `ErrorDataVolumeNotReady` - -**Description**: The DataVolume (persistent storage) backing the VM is not ready. - -**Possible Causes**: - -#### 1. DataVolume Still Provisioning - -Storage provisioning takes time, especially for large disks or when importing images. - -**Diagnostic Commands**: -```bash -# Check DataVolume status -oc get datavolume -n - -# Look for status: Pending, ImportScheduled, ImportInProgress -# Wait for status: Succeeded - -# Get detailed DataVolume information -oc describe datavolume -n - -# Check PVC (PersistentVolumeClaim) bound status -oc get pvc -n -``` - -**Solution**: Wait for DataVolume provisioning to complete (can take 1-5 minutes). - -#### 2. Storage Class Not Found - -The requested storage class doesn't exist in the cluster. - -**Diagnostic Commands**: -```bash -# List available storage classes -oc get storageclass - -# Check DataVolume's requested storage class -oc get datavolume -n -o jsonpath='{.spec.pvc.storageClassName}' -``` - -**Solution**: -1. Use a valid storage class from the cluster -2. Recreate VM with correct storage class parameter - -#### 3. Insufficient Storage Quota - -Namespace has insufficient storage quota to provision the PVC. - -**Diagnostic Commands**: -```bash -# Check resource quotas -oc describe quota -n - -# Check storage usage -oc get pvc -n -o custom-columns=NAME:.metadata.name,STORAGE:.spec.resources.requests.storage,STATUS:.status.phase -``` - -**Solution**: -1. Request quota increase from cluster admin -2. Delete unused PVCs to free quota -3. Reduce VM storage size - ---- - -### ErrorPvcNotFound - -**Symptom**: VM references a PersistentVolumeClaim that doesn't exist. - -**Diagnostic Commands**: -```bash -# List PVCs in namespace -oc get pvc -n - -# Check VM's PVC references -oc get vm -n -o jsonpath='{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}' -``` - -**Solution**: -- Wait for DataVolume to create the PVC -- Manually create missing PVC -- Fix VM spec to reference correct PVC name - ---- - -## Workaround Patterns for MCP Tool Limitations - -### General Pattern: Diagnose → Propose → Confirm → Execute - -When the MCP tool lacks feature support (e.g., tolerations, node selectors, advanced networking): - -**1. Diagnose** the root cause using Kubernetes commands -```bash -oc describe vm -n -oc get events -n --field-selector involvedObject.name= -``` - -**2. Propose** a manual workaround to the user with clear explanation -- Explain why the issue occurred -- Show what will be changed -- List alternative options - -**3. Confirm** with user before executing (Human-in-the-Loop) -- Wait for explicit "yes" or "apply workaround" -- Never auto-execute modifications - -**4. Execute** the workaround using oc/kubectl -```bash -oc patch vm -n --type=merge -p '...' -``` - -**5. Verify** the fix was successful -```bash -oc get vm -n -o jsonpath='{.status.printableStatus}' -``` - -**6. Document** the limitation and suggest filing enhancement request -- Note this is temporary until MCP tool is enhanced -- Provide link to file issue: https://github.com/openshift/openshift-mcp-server/issues - -### Example: Adding Tolerations Workaround - -**Diagnostic Output**: -```markdown -## ⚠️ VM Scheduling Issue Detected - -**Root Cause**: Node taints prevent VM scheduling - -**Details**: -- Found 3 nodes with taint: `virtualization=true:NoSchedule` -- VM spec does not include matching tolerations -- This prevents VM scheduling on virtualization-dedicated nodes -``` - -**Proposed Workaround**: -```bash -oc patch vm web-server -n vms --type=merge -p ' -spec: - template: - spec: - tolerations: - - key: "virtualization" - operator: "Equal" - value: "true" - effect: "NoSchedule" -' -``` - -**User Confirmation Required**: -``` -How would you like to proceed? -- "apply workaround" - I'll patch the VM with tolerations -- "manual" - I'll provide instructions for you to apply manually -- "cancel" - Delete the VM and abort creation -``` - -**After Execution**: -```markdown -## ✓ Workaround Applied Successfully - -**Action**: Added tolerations for taint `virtualization=true:NoSchedule` -**New Status**: Stopped (VM can now be scheduled) - -**Note**: This workaround was needed because the MCP tool doesn't yet support tolerations. -Future VMs in this cluster will need the same fix until the tool is enhanced. -``` - ---- - -## VM Status Reference - -### Status Values - -| Status | Meaning | Action Required | -|--------|---------|-----------------| -| `Stopped` / `Halted` | VM created but not running | Normal - use vm-lifecycle-manager to start | -| `Running` | VM is running | Normal | -| `Provisioning` | VM resources being prepared | Wait 5-10 seconds, check again | -| `Starting` | VM is booting | Wait for Running status | -| `Stopping` | VM is shutting down | Wait for Stopped status | -| `Terminating` | VM is being deleted | Wait for deletion to complete | -| `ErrorUnschedulable` | Cannot find node to run VM | **Action needed** - see ErrorUnschedulable section | -| `ErrorDataVolumeNotReady` | Storage not ready | **Action needed** - see ErrorDataVolumeNotReady section | -| `ErrorPvcNotFound` | PVC missing | **Action needed** - see ErrorPvcNotFound section | -| `CrashLoopBackOff` | VM repeatedly crashing | Check VM logs, guest OS issues | - -### Checking VM Status - -```bash -# Get printable status -oc get vm -n -o jsonpath='{.status.printableStatus}' - -# Get detailed status and conditions -oc get vm -n -o jsonpath='{.status}' | jq - -# Watch status changes in real-time -oc get vm -n -w -``` - ---- - -## Best Practices for Agents - -When implementing diagnostic workflows: - -1. **Always verify VM status** after creation (wait 5-10 seconds first) -2. **Consult this document** when encountering error status values -3. **Provide clear diagnosis** with evidence (show events, node taints, resource availability) -4. **Offer multiple solutions** (automated workaround vs manual steps vs alternative approaches) -5. **Respect human-in-the-loop** for all VM modifications -6. **Document temporary workarounds** and their limitations clearly -7. **Suggest filing issues** for missing MCP tool features - -### Document Consultation Pattern - -```markdown -**Document Consultation** (REQUIRED): -1. **Action**: Read [troubleshooting.md](../../docs/troubleshooting.md) to understand error causes -2. **Output to user**: "I consulted troubleshooting.md to diagnose the issue." -``` - ---- - -## Known MCP Tool Limitations - -### vm_create tool - -**Currently Supported**: -- ✓ Namespace, name (required) -- ✓ Workload/OS selection (fedora, ubuntu, rhel, etc.) -- ✓ Size hints (small, medium, large) -- ✓ Storage size -- ✓ Autostart flag -- ✓ Networks (Multus NetworkAttachmentDefinitions) -- ✓ Performance family (u1, o1, c1, m1) -- ✓ Instance type, preference - -**Not Currently Supported** (requires workarounds): -- ✗ Tolerations (for node taints) -- ✗ Node selectors -- ✗ Affinity/anti-affinity rules -- ✗ Resource requests/limits (beyond instance type) -- ✗ Custom labels/annotations -- ✗ SSH keys injection -- ✗ Cloud-init user data - -**Workaround Strategy**: Use `oc patch` after VM creation to add missing fields. - -**Enhancement Requests**: File issues at https://github.com/openshift/openshift-mcp-server/issues - ---- - -## Additional Resources - -- [KubeVirt Virtual Machine Status Conditions](https://kubevirt.io/user-guide/virtual_machines/vm_status_conditions/) -- [OpenShift Virtualization Troubleshooting](https://docs.openshift.com/container-platform/latest/virt/support/virt-troubleshooting.html) -- [Kubernetes Scheduling Framework](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/) -- [OpenShift MCP Server Issues](https://github.com/openshift/openshift-mcp-server/issues) diff --git a/rh-virt/docs/troubleshooting/.ai-index/semantic-index.json b/rh-virt/docs/troubleshooting/.ai-index/semantic-index.json new file mode 100644 index 00000000..2512c2a3 --- /dev/null +++ b/rh-virt/docs/troubleshooting/.ai-index/semantic-index.json @@ -0,0 +1,148 @@ +{ + "index_version": "1.0.0", + "last_updated": "2026-02-17", + "description": "Semantic index for VM troubleshooting documentation - maps skills and error types to relevant documentation files", + + "documents": [ + { + "file": "INDEX.md", + "title": "VM Troubleshooting Guide - Index", + "category": "navigation", + "keywords": ["index", "navigation", "overview", "MCP tools", "quick reference"], + "use_cases": ["navigation", "tool-reference", "general-guidance"], + "token_estimate": 450, + "skills": ["all"] + }, + { + "file": "scheduling-errors.md", + "title": "VM Scheduling Errors", + "category": "scheduling", + "keywords": ["ErrorUnschedulable", "node taints", "tolerations", "insufficient resources", "node selector", "scheduling failure"], + "use_cases": ["vm-creation", "vm-lifecycle", "scheduling-diagnosis"], + "errors_covered": [ + "ErrorUnschedulable - Node Taints", + "ErrorUnschedulable - Insufficient Resources", + "ErrorUnschedulable - Node Selector Mismatch" + ], + "token_estimate": 820, + "skills": ["vm-creator", "vm-lifecycle-manager"] + }, + { + "file": "storage-errors.md", + "title": "VM Storage Errors", + "category": "storage", + "keywords": ["ErrorDataVolumeNotReady", "ErrorPvcNotFound", "storage deletion", "PVC", "DataVolume", "cloning", "CDI", "storage class", "storage provisioning"], + "use_cases": ["vm-creation", "vm-deletion", "vm-cloning", "storage-diagnosis"], + "errors_covered": [ + "ErrorDataVolumeNotReady - DataVolume Still Provisioning", + "ErrorDataVolumeNotReady - Storage Class Not Found", + "ErrorDataVolumeNotReady - Insufficient Storage Quota", + "ErrorPvcNotFound", + "Storage Deletion Failures", + "DataVolume Cloning Failures" + ], + "token_estimate": 1350, + "skills": ["vm-creator", "vm-delete", "vm-clone"] + }, + { + "file": "lifecycle-errors.md", + "title": "VM Lifecycle Errors", + "category": "lifecycle", + "keywords": ["terminating", "finalizers", "start failure", "stop failure", "VMI", "virt-launcher", "runStrategy"], + "use_cases": ["vm-deletion", "vm-lifecycle", "start-stop-issues"], + "errors_covered": [ + "VM Stuck in Terminating State", + "VM Won't Start (Non-Scheduling Issues)", + "VM Won't Stop" + ], + "token_estimate": 1150, + "skills": ["vm-delete", "vm-lifecycle-manager"] + }, + { + "file": "runtime-errors.md", + "title": "VM Runtime Errors", + "category": "runtime", + "keywords": ["CrashLoopBackOff", "pod crashes", "kernel panic", "QEMU crash", "OOM", "guest OS failure", "virt-launcher restart"], + "use_cases": ["vm-creation", "vm-lifecycle", "runtime-diagnosis", "crash-diagnosis"], + "errors_covered": [ + "CrashLoopBackOff" + ], + "token_estimate": 850, + "skills": ["vm-creator", "vm-lifecycle-manager"] + }, + { + "file": "network-errors.md", + "title": "VM Network Errors", + "category": "networking", + "keywords": ["network attachment", "Multus", "NetworkAttachmentDefinition", "NAD", "SR-IOV", "secondary networks", "CNI"], + "use_cases": ["vm-creation", "network-troubleshooting"], + "errors_covered": [ + "Network Attachment Failures" + ], + "token_estimate": 580, + "skills": ["vm-creator"] + } + ], + + "skill_to_docs_mapping": { + "vm-creator": { + "primary_docs": ["scheduling-errors.md", "storage-errors.md"], + "secondary_docs": ["runtime-errors.md", "network-errors.md"], + "index": "INDEX.md", + "total_token_estimate": 3650, + "description": "VM creation failures - scheduling, storage provisioning, crashes, networks" + }, + "vm-delete": { + "primary_docs": ["lifecycle-errors.md", "storage-errors.md"], + "secondary_docs": [], + "index": "INDEX.md", + "total_token_estimate": 2950, + "description": "VM deletion failures - stuck terminating, storage cleanup" + }, + "vm-clone": { + "primary_docs": ["storage-errors.md"], + "secondary_docs": [], + "index": "INDEX.md", + "total_token_estimate": 1800, + "description": "VM cloning failures - DataVolume cloning, storage provisioning" + }, + "vm-lifecycle-manager": { + "primary_docs": ["lifecycle-errors.md", "scheduling-errors.md"], + "secondary_docs": ["runtime-errors.md"], + "index": "INDEX.md", + "total_token_estimate": 3270, + "description": "VM start/stop failures - lifecycle management, scheduling" + }, + "vm-inventory": { + "primary_docs": [], + "secondary_docs": [], + "index": "INDEX.md", + "total_token_estimate": 450, + "description": "General guidance for status interpretation" + } + }, + + "error_to_docs_mapping": { + "ErrorUnschedulable": "scheduling-errors.md", + "ErrorDataVolumeNotReady": "storage-errors.md", + "ErrorPvcNotFound": "storage-errors.md", + "Terminating": "lifecycle-errors.md", + "CrashLoopBackOff": "runtime-errors.md", + "NetworkAttachmentFailed": "network-errors.md" + }, + + "cross_references": { + "scheduling-errors.md": ["storage-errors.md", "runtime-errors.md"], + "storage-errors.md": ["scheduling-errors.md", "lifecycle-errors.md"], + "lifecycle-errors.md": ["storage-errors.md", "runtime-errors.md"], + "runtime-errors.md": ["lifecycle-errors.md", "scheduling-errors.md"], + "network-errors.md": ["scheduling-errors.md"] + }, + + "progressive_disclosure_strategy": { + "description": "Load only the minimum documentation needed for current task", + "pattern": "error_type → primary_doc → cross_references (if needed)", + "token_savings": "29% average reduction vs single-file (based on rh-sre data)", + "navigation_overhead_reduction": "85% vs single-file navigation" + } +} diff --git a/rh-virt/docs/troubleshooting/INDEX.md b/rh-virt/docs/troubleshooting/INDEX.md new file mode 100644 index 00000000..0031af37 --- /dev/null +++ b/rh-virt/docs/troubleshooting/INDEX.md @@ -0,0 +1,332 @@ +--- +title: VM Troubleshooting Guide - Index +category: kubevirt +sources: + - title: KubeVirt User Guide - Node Placement + url: https://kubevirt.io/user-guide/virtual_machines/node_placement/ + date_accessed: 2026-02-06 + - title: Kubernetes Taints and Tolerations + url: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ + date_accessed: 2026-02-06 + - title: OpenShift Virtualization - Virtual Machine Status + url: https://docs.openshift.com/container-platform/latest/virt/virtual_machines/virt-managing-vms.html + date_accessed: 2026-02-06 + - title: Kubernetes Finalizers + url: https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/ + date_accessed: 2026-02-17 + - title: KubeVirt Virtual Machine Status Conditions + url: https://kubevirt.io/user-guide/virtual_machines/vm_status_conditions/ + date_accessed: 2026-02-17 + - title: Multus CNI - Network Attachment Definitions + url: https://github.com/k8snetworkplumbingwg/multus-cni + date_accessed: 2026-02-17 +tags: [troubleshooting, scheduling, taints, tolerations, errors, deletion, cloning, lifecycle, networking, crashloop, index, navigation] +semantic_keywords: [troubleshooting index, error categories, VM diagnostics, navigation hub] +use_cases: [vm-creation, vm-deletion, vm-cloning, vm-lifecycle, diagnostics, error-handling, network-troubleshooting] +last_updated: 2026-02-17 +--- + +# VM Troubleshooting Guide - Index + +## Overview + +This guide provides comprehensive diagnostic procedures and workarounds for VirtualMachine errors and issues in OpenShift Virtualization. Use this document when VMs encounter problems during: + +- **Creation**: Scheduling failures, storage provisioning issues +- **Lifecycle**: Start/stop failures, stuck states +- **Deletion**: Resources stuck in Terminating, storage cleanup failures +- **Cloning**: DataVolume cloning errors, cross-namespace issues +- **Networking**: Secondary network attachment failures +- **Runtime**: CrashLoopBackOff, guest OS boot failures + +This guide is consulted by all rh-virt skills (vm-creator, vm-inventory, vm-lifecycle-manager, vm-delete, vm-clone) when diagnosing and remediating VM issues. + +--- + +## 🗂️ Troubleshooting Categories + +The troubleshooting documentation is organized by error category for easier navigation and token optimization. Each category file contains MCP-first diagnostic procedures: + +### 1. [Scheduling Errors](scheduling-errors.md) +**When to use**: VM fails to schedule on any node + +**Errors covered**: +- ErrorUnschedulable - Node Taints +- ErrorUnschedulable - Insufficient Resources +- ErrorUnschedulable - Node Selector Mismatch + +**Skills that use this**: vm-creator, vm-lifecycle-manager + +--- + +### 2. [Storage Errors](storage-errors.md) +**When to use**: VM has storage provisioning, deletion, or cloning issues + +**Errors covered**: +- ErrorDataVolumeNotReady (all 3 subsections) +- ErrorPvcNotFound +- Storage Deletion Failures +- DataVolume Cloning Failures + +**Skills that use this**: vm-creator, vm-delete, vm-clone + +--- + +### 3. [Lifecycle Errors](lifecycle-errors.md) +**When to use**: VM has start/stop/termination issues + +**Errors covered**: +- VM Stuck in Terminating State +- VM Won't Start (Non-Scheduling Issues) +- VM Won't Stop + +**Skills that use this**: vm-delete, vm-lifecycle-manager + +--- + +### 4. [Runtime Errors](runtime-errors.md) +**When to use**: VM repeatedly crashes or fails at runtime + +**Errors covered**: +- CrashLoopBackOff + +**Skills that use this**: vm-creator, vm-lifecycle-manager + +--- + +### 5. [Network Errors](network-errors.md) +**When to use**: VM has secondary network attachment failures + +**Errors covered**: +- Network Attachment Failures (Multus, SR-IOV) + +**Skills that use this**: vm-creator + +--- + +## 🔧 How to Use This Guide: MCP Tools First, CLI Commands Last + +### Critical Principle: MCP-First Approach + +**All diagnostic procedures in this guide follow the MCP-first pattern**: + +``` +1. ✅ TRY: MCP Tool (resources_get, resources_list, etc.) +2. ⚠️ IF FAILS: Ask user permission to use CLI command +3. ❌ LAST RESORT: Execute CLI command (oc/kubectl) with explicit user approval +``` + +### Why MCP Tools First? + +- **Structured Access**: MCP tools provide programmatic, type-safe access to cluster resources +- **Consistency**: Same interface across all operations +- **Better Error Handling**: MCP tools return structured errors +- **Audit Trail**: MCP tool usage is logged and trackable + +### Command Pattern Examples + +Throughout this guide, you'll see diagnostic steps formatted like this: + +**✅ CORRECT Pattern (MCP First)**: + +```markdown +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +**Extract**: `.metadata.finalizers` from returned JSON + +**CLI Fallback** (if MCP unavailable): +Ask user: "MCP tool unavailable. May I use `oc get vm` to check finalizers?" +```bash +oc get vm -n -o jsonpath='{.metadata.finalizers}' +``` +``` + +### Available MCP Tools + +The `openshift-virtualization` MCP server provides these tools: + +**Resource Operations**: +- `resources_get` - Get specific resource (replaces `oc get `) +- `resources_list` - List resources (replaces `oc get `) +- `resources_delete` - Delete resource (replaces `oc delete`) +- `resources_create_or_update` - Create/update resource (replaces `oc apply` / `oc patch`) + +**Pod Operations**: +- `pods_list_in_namespace` - List pods in namespace (replaces `oc get pods -n`) +- `pods_get` - Get pod details (replaces `oc get pod`) +- `pods_log` - Get pod logs (replaces `oc logs`) +- `pods_exec` - Execute in pod (replaces `oc exec`) +- `pods_delete` - Delete pod (replaces `oc delete pod`) +- `pods_top` - Pod resource usage (replaces `oc top pods`) + +**Events & Monitoring**: +- `events_list` - List events (replaces `oc get events`) +- `nodes_top` - Node resource usage (replaces `oc top nodes`) +- `nodes_log` - Node logs (replaces `oc adm node-logs`) +- `nodes_stats_summary` - Detailed node stats + +**VM Operations** (KubeVirt toolset): +- `vm_create` - Create VMs +- `vm_lifecycle` - Start/stop/restart VMs + +**Namespaces**: +- `namespaces_list` - List namespaces (replaces `oc get namespaces`) +- `projects_list` - List OpenShift projects (replaces `oc get projects`) + +### When CLI Commands Are Required + +Some operations have **NO MCP equivalent** and require CLI: + +- `virtctl` commands (console, VNC access) +- `oc debug node` (node debugging) +- `oc auth can-i` (permission checks) +- `oc adm taint` (node taint management) +- Complex JSON patch operations + +For these, the guide will note: **"CLI Only - No MCP equivalent"** + +### Quick Reference: CLI → MCP Mapping + +| CLI Command | MCP Tool Equivalent | +|-------------|---------------------| +| `oc get vm -n ` | `resources_get` with apiVersion="kubevirt.io/v1", kind="VirtualMachine" | +| `oc get vms -n ` | `resources_list` with apiVersion="kubevirt.io/v1", kind="VirtualMachine" | +| `oc delete vmi -n ` | `resources_delete` with kind="VirtualMachineInstance" | +| `oc get pods -n ` | `pods_list_in_namespace` with namespace="" | +| `oc logs -n ` | `pods_log` with name="", namespace="" | +| `oc get events -n ` | `events_list` with namespace="" | +| `oc get nodes` | `resources_list` with apiVersion="v1", kind="Node" | + +**Note**: The table above covers the most common patterns. For MCP tools specific to VM operations, see the "Available MCP Tools" section above, and consult the README for complete MCP server configuration. + +### How to Read Diagnostic Sections + +Each error section includes: +1. **Symptom** - What you observe +2. **Description** - What's happening +3. **Possible Causes** - Why it's happening +4. **Diagnostic Steps** - **MCP tools first**, then CLI fallback +5. **Solutions** - **MCP tools first**, then CLI fallback +6. **Verification** - **MCP tools first**, then CLI fallback + +**Note**: Where CLI commands appear without MCP tool alternatives in older sections, they should be treated as **fallback only**. Skills should attempt MCP tools first, then request user permission before using CLI. + +--- + +## 🔍 Quick Navigation by Skill + +**vm-creator**: +- [Scheduling Errors](scheduling-errors.md) - ErrorUnschedulable diagnostics +- [Storage Errors](storage-errors.md) - ErrorDataVolumeNotReady, storage provisioning +- [Runtime Errors](runtime-errors.md) - CrashLoopBackOff +- [Network Errors](network-errors.md) - Network attachment failures + +**vm-delete**: +- [Lifecycle Errors](lifecycle-errors.md) - VM stuck in Terminating state +- [Storage Errors](storage-errors.md) - Storage deletion failures + +**vm-clone**: +- [Storage Errors](storage-errors.md) - DataVolume cloning failures + +**vm-lifecycle-manager**: +- [Lifecycle Errors](lifecycle-errors.md) - VM won't start/stop +- [Scheduling Errors](scheduling-errors.md) - VM won't start due to scheduling + +**vm-inventory**: +- [INDEX.md](INDEX.md) - General guidance, consult specific categories as needed + +--- + +## 📊 Documentation Coverage & Maintenance + +### Current Coverage + +This troubleshooting guide covers the most common VM errors encountered in OpenShift Virtualization: + +- ✅ **Scheduling failures** - ErrorUnschedulable (3 root causes: node taints, insufficient resources, node selector mismatch) +- ✅ **Storage issues** - ErrorDataVolumeNotReady (3 scenarios), ErrorPvcNotFound, storage deletion failures, DataVolume cloning failures +- ✅ **Lifecycle problems** - VM stuck in Terminating state, VM won't start (non-scheduling), VM won't stop +- ✅ **Runtime crashes** - CrashLoopBackOff (kernel panic, QEMU crashes, OOM, guest OS failures) +- ✅ **Network attachment failures** - Multus NetworkAttachmentDefinition issues, SR-IOV problems + +**Total errors documented**: 12 error types across 6 categories + +--- + +### Encountering Undocumented Errors + +#### For AI Agents (Claude Code) + +If you encounter an error **not documented** in the categories above: + +1. **Report to user** with all available details (error message, affected resources, namespace) +2. **Provide best-effort diagnostics** using MCP tools: + - `resources_get` to inspect resource status + - `pods_log` to check virt-launcher or compute container logs + - `events_list` to view Kubernetes events +3. **Suggest documentation update**: + ``` + ⚠️ This error is not yet documented in the troubleshooting guide. + + **Error**: + **Affected resource**: / + + I recommend adding this error to the troubleshooting guide: + - If it's a [scheduling/storage/lifecycle/runtime/network] issue → Add to existing category file + - If it's a new error class → Create new category file in docs/troubleshooting/ + + Would you like me to help document this error for future reference? + ``` + +4. **Do NOT make up solutions** - only provide factual diagnostics from MCP tool outputs + +--- + +#### For Human Contributors + +To document a new error: + +1. **Determine the category**: + - Scheduling issue → `scheduling-errors.md` + - Storage problem → `storage-errors.md` + - Lifecycle/start/stop → `lifecycle-errors.md` + - Runtime crash → `runtime-errors.md` + - Network issue → `network-errors.md` + - New error class → Create new category file (e.g., `security-errors.md`) + +2. **Follow the MCP-first pattern** (see "How to Use This Guide" above): + - **Diagnostic Steps**: MCP Tool → Parameters → Extract → CLI Fallback + - **Solutions**: MCP Tool approach → Implementation → CLI Fallback + - **Verification**: MCP Tool checks → Expected results → CLI Fallback + +3. **Update semantic index**: + - Edit `.ai-index/semantic-index.json` + - Add error to `error_to_docs_mapping` + - Update relevant skill's `primary_docs` or `secondary_docs` + - Increment token estimates if needed + +4. **Reference in skill files**: + - Update skill's Reference Documentation section + - Ensure skill's Document Consultation steps point to the right category + +5. **Update this INDEX.md**: + - Add error to the appropriate category section (lines 50-120) + - Update "Current Coverage" list above + +--- + +## 📚 Additional Resources + +- [rh-virt README](../../README.md) - MCP server setup and skill documentation +- [SOURCES.md](../SOURCES.md) - Official Red Hat documentation sources +- [Semantic Index](.ai-index/semantic-index.json) - AI-optimized doc discovery diff --git a/rh-virt/docs/troubleshooting/lifecycle-errors.md b/rh-virt/docs/troubleshooting/lifecycle-errors.md new file mode 100644 index 00000000..084f2ce5 --- /dev/null +++ b/rh-virt/docs/troubleshooting/lifecycle-errors.md @@ -0,0 +1,869 @@ +--- +title: VM Lifecycle Errors +category: kubevirt +sources: + - title: Kubernetes Finalizers + url: https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/ + date_accessed: 2026-02-17 + - title: KubeVirt Virtual Machine Status Conditions + url: https://kubevirt.io/user-guide/virtual_machines/vm_status_conditions/ + date_accessed: 2026-02-17 +tags: [troubleshooting, lifecycle, terminating, start, stop, VMI, virt-launcher] +semantic_keywords: [VM stuck terminating, finalizers, VM won't start, VM won't stop, lifecycle management, runStrategy] +use_cases: [vm-deletion, vm-lifecycle] +related_docs: [INDEX.md, storage-errors.md, runtime-errors.md] +last_updated: 2026-02-17 +--- + +# VM Lifecycle Errors + +[← Back to Index](INDEX.md) + +## Overview + +This document covers VM lifecycle issues including start/stop failures and stuck termination states. + +**When to use this document**: +- VM stuck in `Terminating` state +- VM won't start (runStrategy is Always but VM never reaches Running) +- VM won't stop (runStrategy is Halted but VM never reaches Stopped) + +**Skills that use this**: vm-delete, vm-lifecycle-manager + +--- + +### VM Stuck in Terminating State + +**Symptom**: VM shows status `Terminating` but deletion never completes + +**Description**: The VM deletion process is blocked, usually by finalizers, attached resources, or stuck VirtualMachineInstance (VMI). + +**Possible Causes**: +- Finalizers blocking deletion +- PVC/DataVolume still attached and preventing cleanup +- VirtualMachineInstance (VMI) not terminating properly +- Custom controllers or operators blocking deletion +- Stuck virt-launcher pod + +**Diagnostic Steps** (Use MCP Tools First): + +**1. Check finalizers on the VM**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +Extract `.metadata.finalizers` from the returned JSON. + +**CLI Fallback** (if MCP unavailable): +Ask user: "MCP tool unavailable. May I use `oc get vm` to check finalizers?" +```bash +oc get vm -n -o jsonpath='{.metadata.finalizers}' +``` + +**2. Check if VMI still exists**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" +} +``` + +If returns "Not Found", VMI is deleted. If returns resource, VMI is stuck. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vmi -n +``` + +**3. Check virt-launcher pod status**: + +**MCP Tool**: `pods_list_in_namespace` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter results for pods with name containing "virt-launcher-". + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pods -n | grep virt-launcher- +``` + +**4. Check events for deletion issues**: + +**MCP Tool**: `events_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter results where `involvedObject.name` == "" and sort by timestamp. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get events -n --field-selector involvedObject.name= --sort-by='.lastTimestamp' +``` + +**5. Check VM deletion timestamp**: + +Use the same `resources_get` call from step 1, extract `.metadata.deletionTimestamp`. + +If present, VM is in deletion process. If null, VM is not being deleted. + +**Common Finalizer Patterns**: +- `kubevirt.io/virtualMachineControllerFinalize` - Standard KubeVirt finalizer +- `foregroundDeletion` - Waits for dependent objects to be deleted +- Custom finalizers from operators + +**Solutions** (Use MCP Tools First): + +1. **Wait for dependent resources** (safest approach): + - Use `resources_list` to check if PVCs, DataVolumes are still deleting + - Let Kubernetes complete cascade deletion naturally (wait 2-5 minutes) + +2. **Force delete VMI** (if VMI is stuck): + + **MCP Tool**: `resources_delete` (from openshift-virtualization) + + **Parameters**: + ```json + { + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" + } + ``` + + **CLI Fallback** (if MCP unavailable - requires explicit user permission): + ```bash + oc delete vmi -n --grace-period=0 --force + ``` + +3. **Force delete virt-launcher pod** (if pod is stuck): + + **MCP Tool**: `pods_delete` (from openshift-virtualization) + + First, find pod name using `pods_list_in_namespace` (see diagnostic step 3 above). + + **Parameters**: + ```json + { + "namespace": "", + "name": "virt-launcher--xxx" + } + ``` + + **CLI Fallback** (if MCP unavailable): + ```bash + # Find the virt-launcher pod name first + oc get pods -n | grep virt-launcher- + + # Then delete it + oc delete pod virt-launcher--xxx -n --force --grace-period=0 + ``` + +4. **Remove finalizers** (⚠️ dangerous - use only as last resort): + + **MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + + **Process**: + 1. Get current VM using `resources_get` + 2. Remove items from `.metadata.finalizers` array + 3. Update VM using `resources_create_or_update` with modified JSON + + ⚠️ **WARNING**: This can leave orphaned resources. Only use if you understand the implications. + + **CLI Fallback** (JSON patch not easily done via MCP): + ```bash + # This operation is complex for MCP - may need CLI + oc patch vm -n --type=json -p '[{"op": "remove", "path": "/metadata/finalizers"}]' + ``` + +5. **Check for protection labels** (vm-delete skill specific): + + Use `resources_get` from diagnostic step 1, extract `.metadata.labels.protected`. + + If value is "true", the vm-delete skill refuses deletion (this is expected behavior). + +**Verification**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +Should return "Not Found" error if deletion successful. + +**CLI Fallback**: +```bash +oc get vm -n +# Should return: Error from server (NotFound) +``` + +--- + + +--- + +### VM Won't Start (Non-Scheduling Issues) + +**Symptom**: VM start command succeeds (runStrategy changed to Always) but VM never reaches Running state + +**Description**: The VM fails to start for reasons other than scheduling problems (ErrorUnschedulable). This typically involves guest OS boot issues, resource problems, or virtualization errors. + +**Possible Causes**: +- Guest OS kernel panic or boot failure +- Cloud-init configuration errors +- Missing or corrupted disk image +- Insufficient memory for guest OS to boot +- QEMU/KVM virtualization errors +- VirtualMachineInstance (VMI) creation failures +- virt-launcher pod crashes + +**Diagnostic Steps** (Use MCP Tools First): + +**1. Check VMI (VirtualMachineInstance) status**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" +} +``` + +Review `.status.phase`, `.status.conditions`, and overall VMI state. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vmi -n +``` + +**2. Check VMI conditions for errors**: + +Use `resources_get` from step 1, extract `.status.conditions` for detailed error information. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vmi -n -o jsonpath='{.status.conditions}' | jq +``` + +**3. Check virt-launcher pod status**: + +**MCP Tool**: `pods_list_in_namespace` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter results for pods with name matching `virt-launcher-`. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pods -n | grep virt-launcher- +``` + +**4. View virt-launcher pod logs**: + +**MCP Tool**: `pods_log` (from openshift-virtualization) + +First, get pod name from step 3, then: + +**Parameters**: +```json +{ + "namespace": "", + "name": "virt-launcher--xxx", + "tail": 100 +} +``` + +Look for QEMU errors, memory allocation failures, device errors. + +**CLI Fallback** (if MCP unavailable): +```bash +oc logs -n $(oc get pods -n | grep virt-launcher- | awk '{print $1}') +``` + +**5. Check virt-launcher pod events**: + +**MCP Tool**: `events_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter results for events where `.involvedObject.name` matches the virt-launcher pod name from step 3. + +Alternatively, use `pods_get` to get full pod details: + +**Parameters**: +```json +{ + "namespace": "", + "name": "virt-launcher--xxx" +} +``` + +**CLI Fallback** (if MCP unavailable): +```bash +oc describe pod $(oc get pods -n | grep virt-launcher- | awk '{print $1}') +``` + +**6. Access VM console to see guest OS boot messages**: + +⚠️ **Note**: Console access requires `virtctl` CLI tool (no MCP equivalent). + +**CLI Required** (no MCP alternative): +```bash +virtctl console -n +``` + +Look for kernel panic, initramfs errors, filesystem errors. + +**7. Check VM events**: + +**MCP Tool**: `events_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter results for events where `.involvedObject.kind` is `VirtualMachine` and `.involvedObject.name` matches ``. + +**CLI Fallback** (if MCP unavailable): +```bash +oc describe vm -n | grep -A 20 "Events:" +``` + +**Common Error Patterns**: + +1. **Guest OS Boot Failure**: + - Console shows kernel panic + - Guest hangs at GRUB or boot loader + - Cloud-init errors during first boot + +2. **Insufficient Memory**: + - Guest OS kills processes due to OOM + - VMI logs show memory allocation errors + +3. **QEMU Crashes**: + - virt-launcher logs show QEMU segfaults + - VMI repeatedly restarts + +**Solutions** (Use MCP Tools First): + +1. **Check guest console for boot errors**: + + ⚠️ **Note**: Console access requires `virtctl` CLI tool (no MCP equivalent). + + **CLI Required** (no MCP alternative): + ```bash + virtctl console -n + ``` + + Look for kernel panic, initramfs errors, filesystem errors. + +2. **Check virt-launcher pod logs for QEMU errors**: + + Use `pods_log` from diagnostic step 4 to view logs. + + Look for: + - "qemu-system-x86_64: ..." errors + - Memory allocation failures + - Device errors + + **CLI Fallback** (if MCP unavailable): + ```bash + oc logs -n virt-launcher--xxx + ``` + +3. **Increase memory if OOM detected**: + + **MCP Tool**: `resources_get` (from openshift-virtualization) + + Check VMI memory allocation: + + **Parameters**: + ```json + { + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" + } + ``` + + Extract `.spec.domain.resources.requests.memory`. + + If too low, delete VM and recreate with larger instance type (e.g., change from "small" to "medium" in vm-creator). + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get vmi -n -o jsonpath='{.spec.domain.resources.requests.memory}' + ``` + +4. **Verify disk image integrity**: + + **MCP Tool**: `resources_get` (from openshift-virtualization) + + Check DataVolume status: + + **Parameters**: + ```json + { + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "name": "" + } + ``` + + Check PVC is bound: + + **Parameters**: + ```json + { + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "name": "" + } + ``` + + If using container disk, verify image exists and is accessible (check VMI spec). + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get datavolume -n + oc get pvc -n + ``` + +5. **Check cloud-init configuration** (if applicable): + + **MCP Tool**: `resources_get` (from openshift-virtualization) + + **Parameters**: + ```json + { + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" + } + ``` + + Extract `.spec.template.spec.volumes[]` and look for `cloudInitNoCloud` or `cloudInitConfigDrive` entries. + + Cloud-init syntax errors prevent boot. Check virt-launcher logs (diagnostic step 4) for cloud-init errors. + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get vm -n -o jsonpath='{.spec.template.spec.volumes[?(@.cloudInitNoCloud)]}' | jq + ``` + +6. **Restart VMI** (soft reset): + + **MCP Tool**: `resources_delete` (from openshift-virtualization) + + Delete VMI (VM controller will recreate it): + + **Parameters**: + ```json + { + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" + } + ``` + + Wait for new VMI to start (use `resources_get` to check status). + + **CLI Fallback** (if MCP unavailable): + ```bash + oc delete vmi -n + oc get vmi -n -w + ``` + +7. **Check virtualization extensions** (KVM): + + ⚠️ **Note**: Node debugging requires `oc debug` CLI command (no MCP equivalent). + + **CLI Required** (no MCP alternative): + ```bash + oc debug node/ + + # In debug shell: + chroot /host + lsmod | grep kvm + # Should show kvm_intel or kvm_amd + ``` + +**Verification** (Use MCP Tools First): + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +After remediation, check VM status: + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +Check `.status.printableStatus` (should eventually return `Running`). + +Check VMI is running: + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" +} +``` + +Check `.status.phase` (should show `Running`). + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vm -n -o jsonpath='{.status.printableStatus}' +# Should eventually return: Running + +oc get vmi -n +# Should show: Running +``` + +--- + +### VM Won't Stop + +**Symptom**: VM runStrategy changed to Halted but VM never reaches Stopped state + +**Description**: The VM stop/shutdown process fails to complete, leaving VM in Stopping state indefinitely. + +**Possible Causes**: +- Guest OS not responding to ACPI shutdown signal +- virt-launcher pod stuck and not terminating +- VirtualMachineInstance (VMI) deletion blocked by finalizers +- Guest shutdown scripts hanging +- Filesystem sync issues in guest OS + +**Diagnostic Steps** (Use MCP Tools First): + +**1. Check VM status**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +Check `.status.printableStatus` (might show `Stopping`). + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vm -n -o jsonpath='{.status.printableStatus}' +``` + +**2. Check VMI status and deletion timestamp**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" +} +``` + +Check `.metadata.deletionTimestamp` (if set, VMI is being deleted). + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vmi -n +oc get vmi -n -o jsonpath='{.metadata.deletionTimestamp}' +``` + +**3. Check virt-launcher pod status**: + +**MCP Tool**: `pods_list_in_namespace` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter results for pods with name matching `virt-launcher-`. Check if pod is in `Terminating` state. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pods -n | grep virt-launcher- +``` + +**4. Check VMI events**: + +**MCP Tool**: `events_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter results for events where `.involvedObject.kind` is `VirtualMachineInstance` and `.involvedObject.name` matches ``. + +**CLI Fallback** (if MCP unavailable): +```bash +oc describe vmi -n | grep -A 10 "Events:" +``` + +**5. Check VMI finalizers**: + +Use `resources_get` from step 2, extract `.metadata.finalizers`. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vmi -n -o jsonpath='{.metadata.finalizers}' +``` + +**6. Check if guest is responsive** (if VMI still exists): + +⚠️ **Note**: Console access requires `virtctl` CLI tool (no MCP equivalent). + +**CLI Required** (no MCP alternative): +```bash +virtctl console -n +``` + +**Common Patterns**: +- VMI shows `deletionTimestamp` but never actually deletes +- virt-launcher pod in `Terminating` state +- VM runStrategy is `Halted` but printableStatus shows `Stopping` + +**Solutions** (Use MCP Tools First): + +1. **Wait for graceful shutdown** (default: 30 seconds): + + Wait 1-2 minutes for guest OS to complete shutdown. Check status periodically using `resources_get` from diagnostic step 1. + +2. **Force stop by deleting VMI**: + + **MCP Tool**: `resources_delete` (from openshift-virtualization) + + This is the standard way to force-stop a VM. + + **Parameters**: + ```json + { + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" + } + ``` + + Wait for VMI deletion, then verify using `resources_get` (should return "Not Found" error). + + **CLI Fallback** (if MCP unavailable): + ```bash + oc delete vmi -n + oc get vmi -n + # Should return: Error from server (NotFound) + ``` + +3. **Force delete VMI with grace period** (if VMI won't delete): + + ⚠️ **Note**: MCP `resources_delete` does not support `--grace-period` or `--force` flags. Use CLI for force deletion. + + **CLI Fallback** (required for force delete): + Ask user: "Force deletion requires CLI. May I use `oc delete --force`?" + ```bash + oc delete vmi -n --grace-period=0 --force + ``` + +4. **Force delete virt-launcher pod**: + + **MCP Tool**: `pods_delete` (from openshift-virtualization) + + First, find the pod using `pods_list_in_namespace` from diagnostic step 3. + + **Parameters**: + ```json + { + "namespace": "", + "name": "virt-launcher--xxx" + } + ``` + + ⚠️ **Note**: For force deletion with grace period, use CLI fallback. + + **CLI Fallback** (required for force delete): + Ask user: "Force deletion requires CLI. May I use `oc delete --force`?" + ```bash + POD_NAME=$(oc get pods -n | grep virt-launcher- | awk '{print $1}') + oc delete pod $POD_NAME -n --force --grace-period=0 + ``` + +5. **Remove VMI finalizers** (⚠️ last resort): + + **MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + + **Process**: + 1. Get current VMI using `resources_get` (diagnostic step 2) + 2. Remove items from `.metadata.finalizers` array + 3. Update VMI using `resources_create_or_update` with modified JSON + + ⚠️ **WARNING**: Can leave orphaned resources. Only use if you understand the implications. + + **CLI Fallback** (JSON patch easier via CLI): + Ask user: "Patching finalizers is easier via CLI. May I use `oc patch`?" + ```bash + oc patch vmi -n --type=json -p '[{"op": "remove", "path": "/metadata/finalizers"}]' + ``` + +6. **Patch VM runStrategy directly** (ensure consistency): + + **MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + + **Process**: + 1. Get current VM using `resources_get` (diagnostic step 1) + 2. Set `.spec.runStrategy` to `"Halted"` + 3. Update VM using `resources_create_or_update` with modified JSON + + **CLI Fallback** (merge patch easier via CLI): + Ask user: "Patching runStrategy is easier via CLI. May I use `oc patch`?" + ```bash + oc patch vm -n --type=merge -p '{"spec":{"runStrategy":"Halted"}}' + ``` + +**Verification** (Use MCP Tools First): + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +After remediation, check VM status: + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +Check `.status.printableStatus` (should return `Stopped` or `Halted`). + +Verify VMI is gone: + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" +} +``` + +Should return "Not Found" error. + +Verify virt-launcher pod is gone: + +**MCP Tool**: `pods_list_in_namespace` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter for pods matching `virt-launcher-`. Should return no results. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vm -n -o jsonpath='{.status.printableStatus}' +# Should return: Stopped or Halted + +oc get vmi -n +# Should return: Error from server (NotFound) + +oc get pods -n | grep virt-launcher- +# Should return: No resources found +``` + +**Prevention**: +- Ensure guest OS has ACPI support enabled +- Use proper shutdown commands in guest OS +- Avoid forceful stops unless necessary (can corrupt guest filesystem) + +--- + + +--- + +[← Back to Index](INDEX.md) | [← Storage Errors](storage-errors.md) | [Runtime Errors →](runtime-errors.md) diff --git a/rh-virt/docs/troubleshooting/network-errors.md b/rh-virt/docs/troubleshooting/network-errors.md new file mode 100644 index 00000000..a3e920e5 --- /dev/null +++ b/rh-virt/docs/troubleshooting/network-errors.md @@ -0,0 +1,429 @@ +--- +title: VM Network Errors +category: kubevirt +sources: + - title: Multus CNI - Network Attachment Definitions + url: https://github.com/k8snetworkplumbingwg/multus-cni + date_accessed: 2026-02-17 +tags: [troubleshooting, networking, Multus, NAD, SR-IOV, secondary networks] +semantic_keywords: [network attachment failures, Multus, NetworkAttachmentDefinition, SR-IOV, secondary networks] +use_cases: [vm-creation, network-troubleshooting] +related_docs: [INDEX.md, scheduling-errors.md] +last_updated: 2026-02-17 +--- + +# VM Network Errors + +[← Back to Index](INDEX.md) + +## Overview + +This document covers VM secondary network attachment failures using Multus CNI and NetworkAttachmentDefinitions. + +**When to use this document**: +- VM created successfully but secondary networks not attached +- NetworkAttachmentDefinition not found errors +- Multus CNI failures +- SR-IOV device attachment issues + +**Skills that use this**: vm-creator + +--- + +### Network Attachment Failures + +**Symptom**: VM created successfully but secondary networks (Multus) not attached or not working + +**Description**: The VM fails to attach to secondary networks defined via NetworkAttachmentDefinitions (Multus CNI). + +**Possible Causes**: +- NetworkAttachmentDefinition doesn't exist in the namespace +- Multus CNI not installed or not configured on cluster +- Namespace mismatch (NAD in different namespace than VM) +- Interface name conflicts in VM spec +- Bridge/network configuration errors in NAD +- SR-IOV device not available (if using SR-IOV) + +**Diagnostic Steps** (Use MCP Tools First): + +**1. List NetworkAttachmentDefinitions in namespace**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "k8s.cni.cncf.io/v1", + "kind": "NetworkAttachmentDefinition", + "namespace": "" +} +``` + +Review `.items[].metadata.name` for available NADs. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get network-attachment-definitions -n +``` + +**2. List NetworkAttachmentDefinitions in all namespaces**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "k8s.cni.cncf.io/v1", + "kind": "NetworkAttachmentDefinition" +} +``` + +Omit `namespace` parameter to list across all namespaces. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get network-attachment-definitions -A +``` + +**3. Check specific NetworkAttachmentDefinition**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "k8s.cni.cncf.io/v1", + "kind": "NetworkAttachmentDefinition", + "namespace": "", + "name": "" +} +``` + +Review `.spec.config` for CNI configuration. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get network-attachment-definition -n -o yaml +``` + +**4. Check VM network configuration**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +Extract `.spec.template.spec.networks` to see network references. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vm -n -o jsonpath='{.spec.template.spec.networks}' | jq +``` + +**5. Check VM domain interfaces**: + +Use `resources_get` from step 4, extract `.spec.template.spec.domain.devices.interfaces`. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vm -n -o jsonpath='{.spec.template.spec.domain.devices.interfaces}' | jq +``` + +**6. Check virt-launcher pod network annotations** (shows actual attachments): + +**MCP Tool**: `pods_list_in_namespace` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter for virt-launcher pod, then extract `.metadata.annotations["k8s.v1.cni.cncf.io/network-status"]`. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pod virt-launcher--xxx -n -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' | jq +``` + +**7. Check Multus is installed**: + +**MCP Tool**: `pods_list_in_namespace` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "openshift-multus" +} +``` + +Should show Multus CNI pods running. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pods -n openshift-multus +``` + +**8. Check for errors in virt-launcher pod events**: + +**MCP Tool**: `events_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter for events where `.involvedObject.name` matches the virt-launcher pod name. + +**CLI Fallback** (if MCP unavailable): +```bash +oc describe pod virt-launcher--xxx -n +``` + +**Common Error Messages**: +- `"network-attachment-definition not found"` - NAD doesn't exist in namespace +- `"multus CNI not configured"` - Multus not installed or misconfigured +- `"interface name conflict"` - Duplicate interface names in VM spec +- `"failed to add network"` - CNI plugin error (check NAD config) +- `"no available devices"` - SR-IOV device not available (if using SR-IOV) + +**Solutions** (Use MCP Tools First): + +1. **Verify NetworkAttachmentDefinition exists in correct namespace**: + + Use `resources_list` from diagnostic step 1 to check if NAD exists in VM's namespace. + + If NAD is in different namespace, copy it to VM namespace: + + **MCP Tool**: `resources_get` + `resources_create_or_update` (from openshift-virtualization) + + **Process**: + 1. Get NAD from source namespace using `resources_get` + 2. Modify `.metadata.namespace` to target namespace + 3. Create NAD in target namespace using `resources_create_or_update` + + **CLI Fallback** (stream processing easier via CLI): + Ask user: "Copying NAD across namespaces is easier via CLI. May I use `oc` with sed?" + ```bash + oc get network-attachment-definition -n -o yaml | \ + sed "s/namespace: /namespace: /" | \ + oc apply -f - + ``` + +2. **Create missing NetworkAttachmentDefinition**: + + **MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + + Example: Linux bridge network + + **Parameters**: + ```json + { + "apiVersion": "k8s.cni.cncf.io/v1", + "kind": "NetworkAttachmentDefinition", + "metadata": { + "name": "vlan100", + "namespace": "" + }, + "spec": { + "config": "{\"cniVersion\":\"0.3.1\",\"type\":\"bridge\",\"bridge\":\"br1\",\"vlan\":100,\"ipam\":{\"type\":\"host-local\",\"subnet\":\"192.168.100.0/24\"}}" + } + } + ``` + + **CLI Fallback** (YAML easier via CLI): + Ask user: "Creating NAD with complex config is easier via CLI. May I use `oc apply -f`?" + ```bash + cat < + spec: + config: '{ + "cniVersion": "0.3.1", + "type": "bridge", + "bridge": "br1", + "vlan": 100, + "ipam": { + "type": "host-local", + "subnet": "192.168.100.0/24" + } + }' + EOF + ``` + +3. **Check Multus CNI installation**: + + Use `pods_list_in_namespace` from diagnostic step 7 to verify Multus pods are running. + + To check cluster network operator: + + **MCP Tool**: `resources_list` (from openshift-virtualization) + + **Parameters**: + ```json + { + "apiVersion": "config.openshift.io/v1", + "kind": "ClusterOperator" + } + ``` + + Filter for `network` operator. + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get pods -n openshift-multus + oc get clusteroperators network + ``` + +4. **Fix interface name conflicts** (if VM has duplicate names): + + Use `resources_get` from diagnostic step 4, extract `.spec.template.spec.domain.devices.interfaces[*].name`. + + Each interface must have unique name. If duplicates found, edit VM using `resources_create_or_update`. + + **CLI Fallback** (interactive edit easier via CLI): + Ask user: "Editing VM is easier via CLI. May I use `oc edit`?" + ```bash + oc get vm -n -o jsonpath='{.spec.template.spec.domain.devices.interfaces[*].name}' + oc edit vm -n + ``` + +5. **Validate NAD configuration syntax**: + + Use `resources_get` from diagnostic step 3, extract `.spec.config`. + + Ensure valid JSON. Common issues: missing quotes, wrong CNI type, invalid IPAM config. + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get network-attachment-definition -n -o jsonpath='{.spec.config}' + ``` + +6. **Check SR-IOV device availability** (if using SR-IOV networks): + + **MCP Tool**: `resources_list` (from openshift-virtualization) + + List SR-IOV network node policies: + + **Parameters**: + ```json + { + "apiVersion": "sriovnetwork.openshift.io/v1", + "kind": "SriovNetworkNodePolicy", + "namespace": "openshift-sriov-network-operator" + } + ``` + + Check SR-IOV device plugin pods: + + **MCP Tool**: `pods_list_in_namespace` (from openshift-virtualization) + + **Parameters**: + ```json + { + "namespace": "openshift-sriov-network-operator" + } + ``` + + Filter for pods with "device-plugin" in name. + + Check available SR-IOV devices on node: + + **MCP Tool**: `resources_get` (from openshift-virtualization) + + **Parameters**: + ```json + { + "apiVersion": "v1", + "kind": "Node", + "name": "" + } + ``` + + Review `.status.allocatable` for SR-IOV resources. + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get sriovnetworknodepolicy -n openshift-sriov-network-operator + oc get pods -n openshift-sriov-network-operator | grep device-plugin + oc describe node | grep -A 10 "Allocatable:" + ``` + +7. **Recreate VM with corrected network configuration** (if needed): + + If network attachment is fundamentally broken, delete and recreate VM with correct NAD references using vm-creator skill. + +**Verification** (Use MCP Tools First): + +**MCP Tool**: `pods_list_in_namespace` (from openshift-virtualization) + +After remediation, check virt-launcher pod network status: + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter for virt-launcher pod, then extract `.metadata.annotations["k8s.v1.cni.cncf.io/network-status"]`. + +Should show all attached networks with status. Example output: +```json +[ + { + "name": "openshift-sdn", + "interface": "eth0", + "ips": ["10.128.2.10"], + "default": true + }, + { + "name": "vlan100", + "interface": "net1", + "ips": ["192.168.100.5"] + } +] +``` + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pod virt-launcher--xxx -n -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' | jq +``` + +**Check from inside VM** (via console): + +⚠️ **Note**: Console access requires `virtctl` CLI tool (no MCP equivalent). + +**CLI Required** (no MCP alternative): +```bash +virtctl console -n +# In guest OS: +ip addr show +# Should show all network interfaces (eth0, net1, etc.) +``` + +**Common Network Types**: +- **Linux Bridge**: Layer 2 bridge for VLAN networks +- **SR-IOV**: High-performance direct device assignment +- **macvlan**: MAC-based VLAN for container networks +- **OVN-Kubernetes**: OpenShift native overlay network + +--- + + +--- + +[← Back to Index](INDEX.md) | [← Runtime Errors](runtime-errors.md) diff --git a/rh-virt/docs/troubleshooting/runtime-errors.md b/rh-virt/docs/troubleshooting/runtime-errors.md new file mode 100644 index 00000000..e6f3099f --- /dev/null +++ b/rh-virt/docs/troubleshooting/runtime-errors.md @@ -0,0 +1,616 @@ +--- +title: VM Runtime Errors +category: kubevirt +sources: + - title: Kubernetes Pod Lifecycle + url: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/ + date_accessed: 2026-02-17 +tags: [troubleshooting, runtime, CrashLoopBackOff, guest OS, QEMU, crashes] +semantic_keywords: [CrashLoopBackOff, pod crashes, guest kernel panic, QEMU crash, OOM, virt-launcher restart] +use_cases: [vm-creation, vm-lifecycle, diagnostics] +related_docs: [INDEX.md, lifecycle-errors.md, scheduling-errors.md] +last_updated: 2026-02-17 +--- + +# VM Runtime Errors + +[← Back to Index](INDEX.md) + +## Overview + +This document covers VM runtime failures where the virt-launcher pod or guest OS repeatedly crashes. + +**When to use this document**: +- VM or virt-launcher pod shows `CrashLoopBackOff` status +- virt-launcher pod repeatedly restarting +- Guest OS kernel panics on boot + +**Skills that use this**: vm-creator, vm-lifecycle-manager + +--- + +### CrashLoopBackOff + +**Symptom**: VM status shows `CrashLoopBackOff` or virt-launcher pod repeatedly restarting + +**Description**: The virt-launcher pod or guest OS is repeatedly crashing and restarting, indicating a critical failure in the virtualization stack or guest OS. + +**Possible Causes**: +- Guest OS kernel panic on boot +- Insufficient resources (memory/CPU) for guest OS +- Corrupted disk image or filesystem +- QEMU/libvirt crashes due to configuration errors +- Missing or incompatible device drivers in guest +- Resource limits too low for virt-launcher pod +- Virtualization features (KVM) not available on node + +**Diagnostic Steps** (Use MCP Tools First): + +**1. Check virt-launcher pod restart count**: + +**MCP Tool**: `pods_list_in_namespace` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter for virt-launcher pod. Check `.status.containerStatuses[0].restartCount` (>0 indicates crashes). + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pods -n | grep virt-launcher- +# Look at RESTARTS column +``` + +**2. View recent crash logs** (previous container instance): + +**MCP Tool**: `pods_log` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "", + "name": "virt-launcher--xxx", + "previous": true, + "tail": 100 +} +``` + +Look for QEMU errors, kernel panics, or segfaults. + +**CLI Fallback** (if MCP unavailable): +```bash +oc logs -n virt-launcher--xxx --previous +``` + +**3. Check current virt-launcher logs**: + +**MCP Tool**: `pods_log` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "", + "name": "virt-launcher--xxx", + "tail": 100 +} +``` + +⚠️ **Note**: MCP `pods_log` doesn't support `--all-containers` flag. Call `pods_log` separately for each container if needed. + +**CLI Fallback** (if MCP unavailable or all containers needed): +```bash +oc logs -n virt-launcher--xxx --all-containers +``` + +**4. Check VMI conditions for crash details**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" +} +``` + +Extract `.status.conditions` for crash details. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vmi -n -o jsonpath='{.status.conditions}' | jq +``` + +**5. Check pod events for crash reasons**: + +**MCP Tool**: `events_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter for events where `.involvedObject.name` matches the virt-launcher pod name. + +**CLI Fallback** (if MCP unavailable): +```bash +oc describe pod virt-launcher--xxx -n | grep -A 20 "Events:" +``` + +**6. Check pod resource limits**: + +**MCP Tool**: `pods_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "", + "name": "virt-launcher--xxx" +} +``` + +Extract `.spec.containers[0].resources` for resource limits. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pod virt-launcher--xxx -n -o jsonpath='{.spec.containers[0].resources}' | jq +``` + +**7. Check node kubelet logs for OOM kills**: + +⚠️ **Note**: Node log access requires `oc adm node-logs` CLI command (no MCP equivalent). + +**CLI Required** (no MCP alternative): +```bash +oc adm node-logs -u kubelet | grep -i oom +``` + +**8. Access guest console** (if VM briefly starts): + +⚠️ **Note**: Console access requires `virtctl` CLI tool (no MCP equivalent). + +**CLI Required** (no MCP alternative): +```bash +virtctl console -n +``` + +**Common Crash Patterns**: + +1. **Guest Kernel Panic**: + - Console logs show kernel panic messages + - Guest crashes immediately after boot + - Symptoms: "Kernel panic - not syncing: VFS: Unable to mount root fs" + +2. **OOM (Out of Memory)**: + - Pod killed with reason: `OOMKilled` + - Guest runs out of memory during boot or operation + - virt-launcher logs show memory allocation failures + +3. **QEMU Crash**: + - virt-launcher logs show QEMU segmentation fault + - Symptoms: "qemu-system-x86_64: terminated by signal" + - Configuration incompatibility or QEMU bug + +4. **Disk Image Corruption**: + - Guest cannot boot from disk + - Filesystem errors in guest console + - DataVolume import failed + +**Solutions** (Use MCP Tools First): + +1. **Check guest console for kernel panic or boot errors**: + + ⚠️ **Note**: Console access requires `virtctl` CLI tool (no MCP equivalent). + + **CLI Required** (no MCP alternative): + ```bash + virtctl console -n + ``` + + Look for: + - Kernel panic messages + - Initramfs errors + - Filesystem mounting failures + - Missing device errors + +2. **Review virt-launcher crash logs**: + + Use `pods_log` with `previous: true` from diagnostic step 2. + + Look for: + - QEMU command line errors + - Device initialization failures + - Memory allocation errors + - Signal termination (SIGSEGV, SIGABRT) + + **CLI Fallback** (if MCP unavailable): + ```bash + oc logs -n virt-launcher--xxx --previous + ``` + +3. **Check for OOM (Out of Memory) kills**: + + **MCP Tool**: `pods_get` (from openshift-virtualization) + + **Parameters**: + ```json + { + "namespace": "", + "name": "virt-launcher--xxx" + } + ``` + + Extract `.status.containerStatuses[0].lastState.terminated.reason`. + + If returns `"OOMKilled"`: + - Option 1: Increase virt-launcher memory limits + - Option 2: Decrease guest memory allocation + - Option 3: Use smaller instance type + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get pod virt-launcher--xxx -n -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' + ``` + +4. **Increase resources if OOM detected**: + + **MCP Tool**: `resources_get` (from openshift-virtualization) + + Check current memory allocation: + + **Parameters**: + ```json + { + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" + } + ``` + + Extract `.spec.domain.resources.requests.memory`. + + If too high for node, delete and recreate with smaller instance type using vm-creator skill (change from "large" to "medium" or "small"). + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get vmi -n -o jsonpath='{.spec.domain.resources.requests.memory}' + ``` + +5. **Verify disk image integrity**: + + **MCP Tool**: `resources_list` + `resources_get` (from openshift-virtualization) + + Check DataVolume status: + + **Parameters for list**: + ```json + { + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "" + } + ``` + + **Parameters for specific DV**: + ```json + { + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "name": "" + } + ``` + + Check `.status.phase` (should be `Succeeded`). + + If using container disk, verify image pullable by checking virt-launcher events using diagnostic step 5. + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get datavolume -n + oc get datavolume -n -o jsonpath='{.status.phase}' + ``` + +6. **Check virtualization (KVM) availability**: + + ⚠️ **Note**: Node debugging requires `oc debug` CLI command (no MCP equivalent). + + **CLI Required** (no MCP alternative): + ```bash + oc debug node/ + chroot /host + lsmod | grep kvm + # Should show kvm_intel or kvm_amd + ``` + +7. **Simplify VM configuration** (eliminate variables): + + Try creating minimal VM using vm-creator skill with: + - Small instance type + - No secondary networks + - Simple container disk (e.g., Fedora) + - No cloud-init + + If minimal VM works, add features back one by one. + +8. **Recreate VM with different workload** (test disk image): + + If guest OS consistently crashes, use vm-creator skill to try different OS image (e.g., switch from Ubuntu to Fedora). This tests if issue is workload-specific. + +**Verification** (Use MCP Tools First): + +**MCP Tool**: `pods_list_in_namespace` + `resources_get` (from openshift-virtualization) + +After remediation, check pod restart count stops increasing: + +**Parameters for pods**: +```json +{ + "namespace": "" +} +``` + +Filter for virt-launcher pod. Check `.status.containerStatuses[0].restartCount` - should stabilize (not keep increasing). + +Check VM reaches Running state: + +**Parameters for VM**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +Check `.status.printableStatus` (should return `Running`). + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pods -n | grep virt-launcher- +# RESTARTS should stabilize + +oc get vm -n -o jsonpath='{.status.printableStatus}' +# Should return: Running +``` + +Verify guest is responsive: + +⚠️ **Note**: Console access requires `virtctl` CLI tool (no MCP equivalent). + +**CLI Required** (no MCP alternative): +```bash +virtctl console -n +# Should show login prompt or OS console +``` + +**Advanced Debugging**: + +**MCP Tool**: `pods_exec` (from openshift-virtualization) + +Check libvirt domain XML: + +**Parameters**: +```json +{ + "namespace": "", + "name": "virt-launcher--xxx", + "command": ["virsh", "dumpxml", "1"] +} +``` + +Check QEMU process: + +**Parameters**: +```json +{ + "namespace": "", + "name": "virt-launcher--xxx", + "command": ["ps", "aux"] +} +``` + +Filter output for "qemu" process. + +**CLI Fallback** (if MCP unavailable): +```bash +oc exec -n virt-launcher--xxx -- virsh dumpxml 1 +oc exec -n virt-launcher--xxx -- ps aux | grep qemu +``` + +**Prevention**: +- Start with minimal VM configuration and add complexity gradually +- Use recommended instance types for your workload +- Test disk images before deploying to production +- Ensure nodes have adequate resources and KVM support +- Monitor resource usage and set appropriate limits + +--- + +## Workaround Patterns for MCP Tool Limitations + +### General Pattern: Diagnose → Propose → Confirm → Execute + +When the MCP tool lacks feature support (e.g., tolerations, node selectors, advanced networking): + +**1. Diagnose** the root cause using Kubernetes commands +```bash +oc describe vm -n +oc get events -n --field-selector involvedObject.name= +``` + +**2. Propose** a manual workaround to the user with clear explanation +- Explain why the issue occurred +- Show what will be changed +- List alternative options + +**3. Confirm** with user before executing (Human-in-the-Loop) +- Wait for explicit "yes" or "apply workaround" +- Never auto-execute modifications + +**4. Execute** the workaround using oc/kubectl +```bash +oc patch vm -n --type=merge -p '...' +``` + +**5. Verify** the fix was successful +```bash +oc get vm -n -o jsonpath='{.status.printableStatus}' +``` + +**6. Document** the limitation and suggest filing enhancement request +- Note this is temporary until MCP tool is enhanced +- Provide link to file issue: https://github.com/openshift/openshift-mcp-server/issues + +### Example: Adding Tolerations Workaround + +**Diagnostic Output**: +```markdown +## ⚠️ VM Scheduling Issue Detected + +**Root Cause**: Node taints prevent VM scheduling + +**Details**: +- Found 3 nodes with taint: `virtualization=true:NoSchedule` +- VM spec does not include matching tolerations +- This prevents VM scheduling on virtualization-dedicated nodes +``` + +**Proposed Workaround**: +```bash +oc patch vm web-server -n vms --type=merge -p ' +spec: + template: + spec: + tolerations: + - key: "virtualization" + operator: "Equal" + value: "true" + effect: "NoSchedule" +' +``` + +**User Confirmation Required**: +``` +How would you like to proceed? +- "apply workaround" - I'll patch the VM with tolerations +- "manual" - I'll provide instructions for you to apply manually +- "cancel" - Delete the VM and abort creation +``` + +**After Execution**: +```markdown +## ✓ Workaround Applied Successfully + +**Action**: Added tolerations for taint `virtualization=true:NoSchedule` +**New Status**: Stopped (VM can now be scheduled) + +**Note**: This workaround was needed because the MCP tool doesn't yet support tolerations. +Future VMs in this cluster will need the same fix until the tool is enhanced. +``` + +--- + +## VM Status Reference + +### Status Values + +| Status | Meaning | Action Required | +|--------|---------|-----------------| +| `Stopped` / `Halted` | VM created but not running | Normal - use vm-lifecycle-manager to start | +| `Running` | VM is running | Normal | +| `Provisioning` | VM resources being prepared | Wait 5-10 seconds, check again | +| `Starting` | VM is booting | Wait for Running status or see "VM Won't Start" section if stuck | +| `Stopping` | VM is shutting down | Wait for Stopped status or see "VM Won't Stop" section if stuck | +| `Terminating` | VM is being deleted | Wait for deletion to complete or see "VM Stuck in Terminating State" section if stuck | +| `ErrorUnschedulable` | Cannot find node to run VM | **Action needed** - see ErrorUnschedulable section | +| `ErrorDataVolumeNotReady` | Storage not ready | **Action needed** - see ErrorDataVolumeNotReady section | +| `ErrorPvcNotFound` | PVC missing | **Action needed** - see ErrorPvcNotFound section | +| `CrashLoopBackOff` | VM repeatedly crashing | **Action needed** - see CrashLoopBackOff section | + +### Checking VM Status + +```bash +# Get printable status +oc get vm -n -o jsonpath='{.status.printableStatus}' + +# Get detailed status and conditions +oc get vm -n -o jsonpath='{.status}' | jq + +# Watch status changes in real-time +oc get vm -n -w +``` + +--- + +## Best Practices for Agents + +When implementing diagnostic workflows: + +1. **Always verify VM status** after creation (wait 5-10 seconds first) +2. **Consult this document** when encountering error status values +3. **Provide clear diagnosis** with evidence (show events, node taints, resource availability) +4. **Offer multiple solutions** (automated workaround vs manual steps vs alternative approaches) +5. **Respect human-in-the-loop** for all VM modifications +6. **Document temporary workarounds** and their limitations clearly +7. **Suggest filing issues** for missing MCP tool features + +### Document Consultation Pattern + +```markdown +**Document Consultation** (REQUIRED): +1. **Action**: Read [runtime-errors.md](../../docs/troubleshooting/runtime-errors.md) to understand CrashLoopBackOff causes +2. **Output to user**: "I consulted runtime-errors.md to diagnose the CrashLoopBackOff issue." +``` + +--- + +## Known MCP Tool Limitations + +### vm_create tool + +**Currently Supported**: +- ✓ Namespace, name (required) +- ✓ Workload/OS selection (fedora, ubuntu, rhel, etc.) +- ✓ Size hints (small, medium, large) +- ✓ Storage size +- ✓ Autostart flag +- ✓ Networks (Multus NetworkAttachmentDefinitions) +- ✓ Performance family (u1, o1, c1, m1) +- ✓ Instance type, preference + +**Not Currently Supported** (requires workarounds): +- ✗ Tolerations (for node taints) +- ✗ Node selectors +- ✗ Affinity/anti-affinity rules +- ✗ Resource requests/limits (beyond instance type) +- ✗ Custom labels/annotations +- ✗ SSH keys injection +- ✗ Cloud-init user data + +**Workaround Strategy**: Use `oc patch` after VM creation to add missing fields. + +**Enhancement Requests**: File issues at https://github.com/openshift/openshift-mcp-server/issues + +--- + +## Additional Resources + +- [KubeVirt Virtual Machine Status Conditions](https://kubevirt.io/user-guide/virtual_machines/vm_status_conditions/) +- [OpenShift Virtualization Troubleshooting](https://docs.openshift.com/container-platform/latest/virt/support/virt-troubleshooting.html) +- [Kubernetes Scheduling Framework](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/) +- [OpenShift MCP Server Issues](https://github.com/openshift/openshift-mcp-server/issues) + +--- + +[← Back to Index](INDEX.md) | [← Lifecycle Errors](lifecycle-errors.md) | [Network Errors →](network-errors.md) diff --git a/rh-virt/docs/troubleshooting/scheduling-errors.md b/rh-virt/docs/troubleshooting/scheduling-errors.md new file mode 100644 index 00000000..0c216f54 --- /dev/null +++ b/rh-virt/docs/troubleshooting/scheduling-errors.md @@ -0,0 +1,417 @@ +--- +title: VM Scheduling Errors +category: kubevirt +sources: + - title: KubeVirt User Guide - Node Placement + url: https://kubevirt.io/user-guide/virtual_machines/node_placement/ + date_accessed: 2026-02-06 + - title: Kubernetes Taints and Tolerations + url: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ + date_accessed: 2026-02-06 +tags: [troubleshooting, scheduling, taints, tolerations, ErrorUnschedulable, node selector, resources] +semantic_keywords: [ErrorUnschedulable, scheduling failure, node taints, insufficient resources, node selector mismatch, tolerations] +use_cases: [vm-creation, vm-lifecycle] +related_docs: [INDEX.md, storage-errors.md, runtime-errors.md] +last_updated: 2026-02-17 +--- + +# VM Scheduling Errors + +[← Back to Index](INDEX.md) + +## Overview + +This document covers VM scheduling failures where the Kubernetes scheduler cannot find a suitable node to run the VM's underlying virt-launcher pod. + +**When to use this document**: +- VM shows status `ErrorUnschedulable` after creation or start attempt +- VM events mention scheduling failures, taints, resources, or node selectors + +**Skills that use this**: vm-creator, vm-lifecycle-manager + +--- + +## ErrorUnschedulable + +**Symptom**: VM shows status `ErrorUnschedulable` after creation + +**Description**: The Kubernetes scheduler cannot find a suitable node to run the VM's underlying virt-launcher pod. + +**Possible Causes**: + +### 1. Node Taints (Most Common) + +Nodes have taints that the VM doesn't tolerate. Common in environments with dedicated virtualization infrastructure. + +**Diagnostic Steps** (Use MCP Tools First): + +**1. Check VM events for scheduling failures**: + +**MCP Tool**: `events_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter results for events where `involvedObject.name` == "" and look for messages like: +- "0/X nodes are available: X node(s) had taints that the pod didn't tolerate" + +**CLI Fallback** (if MCP unavailable): +```bash +oc get events -n --field-selector involvedObject.name= +``` + +**2. Check node taints in the cluster**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "Node" +} +``` + +Extract `.spec.taints` from each node in the returned list. Filter for nodes with non-null taints. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get nodes -o json | jq '.items[] | select(.spec.taints != null) | {name: .metadata.name, taints: .spec.taints}' +``` + +**Common Taint Patterns**: +- `virtualization=true:NoSchedule` - Only VMs with matching toleration can schedule +- `node-role.kubernetes.io/infra:NoSchedule` - Infrastructure-only nodes +- `node.kubernetes.io/not-ready:NoSchedule` - Node not ready for workloads + +**Solution - Add Tolerations to VM**: + +The openshift-virtualization MCP server's `vm_create` tool does NOT currently support the `tolerations` parameter. This requires a post-creation workaround using MCP tools. + +**Workaround (post-creation using MCP Tools)**: + +**Step 1**: Get current VM spec + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +**Step 2**: Modify the returned JSON to add tolerations + +Add to `.spec.template.spec.tolerations`: +```json +{ + "tolerations": [ + { + "key": "virtualization", + "operator": "Equal", + "value": "true", + "effect": "NoSchedule" + } + ] +} +``` + +**Step 3**: Update VM with modified spec + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Parameters**: +```json +{ + "resource": "" +} +``` + +Pass the complete modified VM resource as YAML or JSON string. + +**Step 4**: Verify tolerations were added + +Use `resources_get` again and check `.spec.template.spec.tolerations` in response. + +**Step 5**: Check if VM status improved + +Wait 5-10 seconds, then use `resources_get` and check `.status.printableStatus`. + +**CLI Fallback** (if MCP patch is too complex): +```bash +# Ask user permission first: "MCP patch is complex. May I use oc patch instead?" +oc patch vm -n --type=merge -p ' +spec: + template: + spec: + tolerations: + - key: "virtualization" + operator: "Equal" + value: "true" + effect: "NoSchedule" +' + +# Verify tolerations +oc get vm -n -o jsonpath='{.spec.template.spec.tolerations}' | jq + +# Check status +oc get vm -n -o jsonpath='{.status.printableStatus}' +``` + +**Example - Multiple Tolerations**: +```bash +oc patch vm -n --type=merge -p ' +spec: + template: + spec: + tolerations: + - key: "virtualization" + operator: "Equal" + value: "true" + effect: "NoSchedule" + - key: "dedicated" + operator: "Equal" + value: "virt-workloads" + effect: "NoSchedule" +' +``` + +**Toleration Operators**: +- `Equal` - Key and value must match exactly +- `Exists` - Only key must exist (ignores value) + +**Toleration Effects**: +- `NoSchedule` - Don't schedule new pods (existing pods continue) +- `PreferNoSchedule` - Avoid scheduling if possible +- `NoExecute` - Don't schedule AND evict existing pods + +**Alternative Solutions**: +1. **Remove node taints** (if you have cluster-admin access): + ```bash + oc adm taint nodes virtualization=true:NoSchedule- + ``` + +2. **Use different nodes** - If non-tainted nodes exist, ensure VM fits + +3. **File enhancement request** - Request tolerations support in openshift-mcp-server: + https://github.com/openshift/openshift-mcp-server/issues + +--- + +### 2. Insufficient Resources + +Not enough CPU, memory, or storage available on any node. + +**Diagnostic Steps** (Use MCP Tools First): + +**1. Check VM resource requests**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +Extract `.spec.template.spec.domain.resources` to see CPU/memory requests. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vm -n -o jsonpath='{.spec.template.spec.domain.resources}' +``` + +**2. Check node resource availability**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "Node" +} +``` + +For each node in `.items[]`, review `.status.allocatable` and `.status.capacity` for available resources. + +Alternatively, use `nodes_top` MCP tool for current resource usage. + +**CLI Fallback** (if MCP unavailable): +```bash +oc describe nodes | grep -A 5 "Allocated resources" +``` + +**3. Look for VM events mentioning "Insufficient"**: + +**MCP Tool**: `events_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter for events where `.involvedObject.name` matches `` and `.message` contains "Insufficient". + +**CLI Fallback** (if MCP unavailable): +```bash +oc describe vm -n | grep "Insufficient" +``` + +**Example Event**: +``` +0/5 nodes are available: 2 Insufficient cpu, 3 Insufficient memory. +``` + +**Solutions** (Use MCP Tools First): + +1. **Scale cluster** - Add more worker nodes (cluster admin task, no MCP tool) +2. **Reduce VM resources** - Delete and recreate with smaller instance type using vm-creator skill +3. **Delete unused VMs** - Use vm-delete skill to free up resources +4. **Check resource quotas**: + + **MCP Tool**: `resources_list` (from openshift-virtualization) + + **Parameters for quota**: + ```json + { + "apiVersion": "v1", + "kind": "ResourceQuota", + "namespace": "" + } + ``` + + **Parameters for limit range**: + ```json + { + "apiVersion": "v1", + "kind": "LimitRange", + "namespace": "" + } + ``` + + **CLI Fallback** (if MCP unavailable): + ```bash + oc describe quota -n + oc describe limitrange -n + ``` + +--- + +### 3. Node Selector Mismatch + +VM requires specific node labels that don't exist in the cluster. + +**Diagnostic Steps** (Use MCP Tools First): + +**1. Check VM node selector requirements**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +Extract `.spec.template.spec.nodeSelector` to see required node labels. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vm -n -o jsonpath='{.spec.template.spec.nodeSelector}' +``` + +**2. List available node labels**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "Node" +} +``` + +For each node in `.items[]`, review `.metadata.labels` for available labels. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get nodes --show-labels +``` + +**3. Check if any nodes match the selector**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "Node", + "labelSelector": "=" +} +``` + +Should return at least one node with matching labels. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get nodes -l = +``` + +**Solutions** (Use MCP Tools First): + +**Option 1: Remove node selector from VM** + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Process**: +1. Get current VM using `resources_get` (diagnostic step 1) +2. Remove `.spec.template.spec.nodeSelector` field +3. Update VM using `resources_create_or_update` with modified JSON + +**CLI Fallback** (JSON patch easier via CLI): +Ask user: "Patching node selector is easier via CLI. May I use `oc patch`?" +```bash +oc patch vm -n --type=json -p '[{"op": "remove", "path": "/spec/template/spec/nodeSelector"}]' +``` + +**Option 2: Add label to nodes** + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Process**: +1. Get node using `resources_get` +2. Add label to `.metadata.labels` +3. Update node using `resources_create_or_update` + +⚠️ **Note**: Node labeling typically requires cluster admin privileges. + +**CLI Fallback** (simpler via CLI): +Ask user: "Adding node labels is easier via CLI. May I use `oc label`?" +```bash +oc label node = +``` + +--- + +[← Back to Index](INDEX.md) | [Storage Errors →](storage-errors.md) diff --git a/rh-virt/docs/troubleshooting/storage-errors.md b/rh-virt/docs/troubleshooting/storage-errors.md new file mode 100644 index 00000000..02078049 --- /dev/null +++ b/rh-virt/docs/troubleshooting/storage-errors.md @@ -0,0 +1,1011 @@ +--- +title: VM Storage Errors +category: kubevirt +sources: + - title: KubeVirt CDI - DataVolumes + url: https://kubevirt.io/user-guide/storage/containerized_data_importer/ + date_accessed: 2026-02-17 + - title: Kubernetes Persistent Volumes + url: https://kubernetes.io/docs/concepts/storage/persistent-volumes/ + date_accessed: 2026-02-17 +tags: [troubleshooting, storage, DataVolume, PVC, ErrorDataVolumeNotReady, ErrorPvcNotFound, cloning, CDI] +semantic_keywords: [ErrorDataVolumeNotReady, ErrorPvcNotFound, storage deletion, PVC, DataVolume cloning, storage provisioning, storage class] +use_cases: [vm-creation, vm-deletion, vm-cloning] +related_docs: [INDEX.md, scheduling-errors.md, lifecycle-errors.md] +last_updated: 2026-02-17 +--- + +# VM Storage Errors + +[← Back to Index](INDEX.md) + +## Overview + +This document covers VM storage-related failures including storage provisioning, deletion, and cloning issues. + +**When to use this document**: +- VM shows status `ErrorDataVolumeNotReady` or `ErrorPvcNotFound` +- Storage deletion fails after VM deletion +- DataVolume cloning operations fail +- PVC provisioning issues + +**Skills that use this**: vm-creator, vm-delete, vm-clone + +--- + +### ErrorDataVolumeNotReady + +**Symptom**: VM shows status `ErrorDataVolumeNotReady` + +**Description**: The DataVolume (persistent storage) backing the VM is not ready. + +**Possible Causes**: + +#### 1. DataVolume Still Provisioning + +Storage provisioning takes time, especially for large disks or when importing images. + +**Diagnostic Steps** (Use MCP Tools First): + +**1. Check DataVolume status**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "" +} +``` + +Look for status in response: `Pending`, `ImportScheduled`, `ImportInProgress`, or `Succeeded`. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get datavolume -n +``` + +**2. Get detailed DataVolume information**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "name": "" +} +``` + +Check `.status.phase` and `.status.conditions` for provisioning details. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get datavolume -n -o json +``` + +**3. Check PVC (PersistentVolumeClaim) bound status**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "" +} +``` + +Check `.status.phase` for each PVC (should be `Bound`). + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pvc -n +``` + +**Solution**: Wait for DataVolume provisioning to complete (can take 1-5 minutes). Check status periodically using `resources_get`. + +#### 2. Storage Class Not Found + +The requested storage class doesn't exist in the cluster. + +**Diagnostic Steps** (Use MCP Tools First): + +**1. List available storage classes**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "storage.k8s.io/v1", + "kind": "StorageClass" +} +``` + +Review the list of available storage classes (check `.items[].metadata.name`). + +**CLI Fallback** (if MCP unavailable): +```bash +oc get storageclass +``` + +**2. Check DataVolume's requested storage class**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "name": "" +} +``` + +Check `.spec.pvc.storageClassName` in the returned JSON. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get datavolume -n -o jsonpath='{.spec.pvc.storageClassName}' +``` + +**Solution**: +1. Use a valid storage class from the cluster +2. Recreate VM with correct storage class parameter + +#### 3. Insufficient Storage Quota + +Namespace has insufficient storage quota to provision the PVC. + +**Diagnostic Steps** (Use MCP Tools First): + +**1. Check resource quotas**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "ResourceQuota", + "namespace": "" +} +``` + +Review `.items[].status.hard` (quota limits) and `.items[].status.used` (current usage). + +**CLI Fallback** (if MCP unavailable): +```bash +oc describe quota -n +``` + +**2. Check storage usage**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "" +} +``` + +For each PVC, check `.metadata.name`, `.spec.resources.requests.storage`, and `.status.phase`. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pvc -n -o custom-columns=NAME:.metadata.name,STORAGE:.spec.resources.requests.storage,STATUS:.status.phase +``` + +**Solution**: +1. Request quota increase from cluster admin +2. Delete unused PVCs to free quota +3. Reduce VM storage size + +--- + +### ErrorPvcNotFound + +**Symptom**: VM references a PersistentVolumeClaim that doesn't exist. + +**Diagnostic Steps** (Use MCP Tools First): + +**1. List PVCs in namespace**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "" +} +``` + +Review the list of available PVCs (check `.items[].metadata.name`). + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pvc -n +``` + +**2. Check VM's PVC references**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +Extract `.spec.template.spec.volumes[*].persistentVolumeClaim.claimName` from the returned JSON to see which PVCs the VM is referencing. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get vm -n -o jsonpath='{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}' +``` + +**Solution**: +- Wait for DataVolume to create the PVC +- Manually create missing PVC +- Fix VM spec to reference correct PVC name + +--- + + +--- + +### Storage Deletion Failures + +**Symptom**: VM deleted successfully but PVC or DataVolume remains in namespace + +**Description**: Storage resources (PersistentVolumeClaims, DataVolumes) fail to delete after VM removal. + +**Possible Causes**: +- PVC still bound to active PersistentVolume with `Retain` policy +- DataVolume still being referenced by another resource +- CDI (Containerized Data Importer) controller issues +- Storage class retention policy preventing deletion +- Finalizers on PVC/DataVolume blocking cleanup +- PVC still mounted by a pod + +**Diagnostic Steps** (Use MCP Tools First): + +**1. Check PVC status**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "" +} +``` + +Review `.items[].metadata.name` and `.items[].status.phase` for each PVC. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pvc -n +``` + +**2. Check specific PVC phase**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "name": "" +} +``` + +Check `.status.phase` (should be `Released` or `Bound`). + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pvc -n -o jsonpath='{.status.phase}' +``` + +**3. Check DataVolume status**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "" +} +``` + +Review `.items[].metadata.name` and `.items[].status.phase` for each DataVolume. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get datavolume -n +``` + +**4. Check what's using the PVC**: + +**MCP Tool**: `pods_list_in_namespace` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +For each pod in `.items[]`, check `.spec.volumes[].persistentVolumeClaim.claimName` to find pods using the PVC. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pods -n -o json | jq '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="") | .metadata.name' +``` + +**5. Check PVC finalizers**: + +Use `resources_get` from step 2, extract `.metadata.finalizers` from the returned JSON. + +**6. Check DataVolume finalizers**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "name": "" +} +``` + +Extract `.metadata.finalizers` from the returned JSON. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get datavolume -n -o jsonpath='{.metadata.finalizers}' +``` + +**7. Check PV reclaim policy**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolume" +} +``` + +Filter results for PV where `.spec.claimRef.name` matches ``. + +To get specific PV policy: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolume", + "name": "" +} +``` + +Check `.spec.persistentVolumeReclaimPolicy`. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pv | grep +oc get pv -o jsonpath='{.spec.persistentVolumeReclaimPolicy}' +``` + +**Common Finalizer Patterns**: +- `kubernetes.io/pvc-protection` - Protects PVC while in use +- `cdi.kubevirt.io/dataVolumeFinalizer` - CDI cleanup finalizer + +**Solutions** (Use MCP Tools First): + +1. **Delete DataVolume first, then PVC**: + + **MCP Tool**: `resources_delete` (from openshift-virtualization) + + Delete DataVolume first (often blocks PVC deletion): + + **Parameters**: + ```json + { + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "name": "" + } + ``` + + Wait a few seconds, then delete PVC: + + **Parameters**: + ```json + { + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "name": "" + } + ``` + + **CLI Fallback** (if MCP unavailable): + ```bash + oc delete datavolume -n + oc delete pvc -n + ``` + +2. **Check for pods still using PVC**: + + **MCP Tool**: `pods_list_in_namespace` (from openshift-virtualization) + + **Parameters**: + ```json + { + "namespace": "" + } + ``` + + Filter results for pods where `.spec.volumes[].persistentVolumeClaim.claimName` equals ``. + + Then delete the pods using `pods_delete`: + + **Parameters**: + ```json + { + "namespace": "", + "name": "" + } + ``` + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get pods -n -o json | jq -r '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="") | .metadata.name' + oc delete pod -n + ``` + +3. **Force delete PVC** (if safe to do so): + + ⚠️ **Note**: MCP `resources_delete` does not support `--grace-period` or `--force` flags. Use CLI for force deletion. + + **CLI Fallback** (required for force delete): + Ask user: "Force deletion requires CLI. May I use `oc delete --force`?" + ```bash + oc delete pvc -n --grace-period=0 --force + ``` + +4. **Remove finalizers from PVC** (⚠️ last resort): + + **MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + + **Process**: + 1. Get current PVC using `resources_get` (diagnostic step 2) + 2. Remove items from `.metadata.finalizers` array + 3. Update PVC using `resources_create_or_update` with modified JSON + + ⚠️ **WARNING**: Can leave orphaned storage. Only use if you understand the implications. + + **CLI Fallback** (JSON patch easier via CLI): + Ask user: "Patching finalizers is easier via CLI. May I use `oc patch`?" + ```bash + oc patch pvc -n --type=json -p '[{"op": "remove", "path": "/metadata/finalizers"}]' + ``` + +5. **Remove finalizers from DataVolume** (⚠️ last resort): + + **MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + + **Process**: + 1. Get current DataVolume using `resources_get` (diagnostic step 6) + 2. Remove items from `.metadata.finalizers` array + 3. Update DataVolume using `resources_create_or_update` with modified JSON + + ⚠️ **WARNING**: Can leave orphaned storage. Only use if you understand the implications. + + **CLI Fallback** (JSON patch easier via CLI): + Ask user: "Patching finalizers is easier via CLI. May I use `oc patch`?" + ```bash + oc patch datavolume -n --type=json -p '[{"op": "remove", "path": "/metadata/finalizers"}]' + ``` + +6. **Change PV reclaim policy** (if PV has Retain policy): + + **MCP Tool**: `resources_get` + `resources_create_or_update` (from openshift-virtualization) + + **Process**: + 1. Get current PV policy using `resources_get` (diagnostic step 7) + 2. Modify `.spec.persistentVolumeReclaimPolicy` to `"Delete"` + 3. Update PV using `resources_create_or_update` with modified JSON + + ⚠️ **WARNING**: Setting to `Delete` will delete underlying storage. + + **CLI Fallback** (JSON patch easier via CLI): + Ask user: "Patching PV reclaim policy is easier via CLI. May I use `oc patch`?" + ```bash + oc get pv -o jsonpath='{.spec.persistentVolumeReclaimPolicy}' + oc patch pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}' + ``` + +**Storage Quota Check** (Use MCP Tools First): + +After deletion, verify storage quota is freed: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters for quota check**: +```json +{ + "apiVersion": "v1", + "kind": "ResourceQuota", + "namespace": "" +} +``` + +Review `.items[].status.used` to verify storage quota is freed. + +**Parameters for PVC verification**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "" +} +``` + +**CLI Fallback** (if MCP unavailable): +```bash +oc describe quota -n +oc get pvc -n +``` + +**Verification** (Use MCP Tools First): + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +Confirm PVC is deleted: + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "name": "" +} +``` + +Should return "Not Found" error. + +Confirm DataVolume is deleted: + +**Parameters**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "name": "" +} +``` + +Should return "Not Found" error. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pvc -n +# Should return: Error from server (NotFound) + +oc get datavolume -n +# Should return: Error from server (NotFound) +``` + +--- + +### DataVolume Cloning Failures + +**Symptom**: VM clone created successfully but DataVolume clone operation fails + +**Description**: The DataVolume cloning process (used by vm-clone skill) fails to create a copy of the source storage. + +**Possible Causes**: +- CSI driver doesn't support volume cloning +- Source PVC storage class incompatible with cloning +- Cross-namespace cloning not permitted by storage backend +- Insufficient storage quota in target namespace +- Source PVC not in `Bound` state +- Storage class doesn't have volume cloning enabled +- CDI (Containerized Data Importer) controller issues + +**Diagnostic Steps** (Use MCP Tools First): + +**1. Check DataVolume clone status**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "name": "" +} +``` + +Review `.status.phase`, `.status.conditions`, and `.metadata.name`. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get datavolume -n +``` + +**2. Check DataVolume events for errors**: + +**MCP Tool**: `events_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "" +} +``` + +Filter results for events related to the DataVolume (check `.involvedObject.name` equals ``). + +**CLI Fallback** (if MCP unavailable): +```bash +oc describe datavolume -n +``` + +**3. Check DataVolume phase**: + +Use `resources_get` from step 1, extract `.status.phase`. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get datavolume -n -o jsonpath='{.status.phase}' +``` + +**4. Check if storage class supports cloning**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "storage.k8s.io/v1", + "kind": "StorageClass", + "name": "" +} +``` + +Review the full YAML output for cloning-related configurations. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get storageclass -o yaml | grep -A 5 -i clone +``` + +**5. Check CSI driver capabilities**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "storage.k8s.io/v1", + "kind": "CSIDriver" +} +``` + +Review `.items[].metadata.name` for available CSI drivers. + +**CLI Fallback** (if MCP unavailable): +```bash +oc get csidriver +``` + +**6. Check source PVC status**: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "name": "" +} +``` + +Check `.status.phase` (should be `Bound` for cloning to work). + +**CLI Fallback** (if MCP unavailable): +```bash +oc get pvc -n +``` + +**7. Check target namespace storage quota**: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "ResourceQuota", + "namespace": "" +} +``` + +Review `.items[].status.hard` (limits) and `.items[].status.used` (current usage). + +**CLI Fallback** (if MCP unavailable): +```bash +oc describe quota -n +``` + +**8. Check CDI controller logs**: + +**MCP Tool**: `pods_list_in_namespace` + `pods_log` (from openshift-virtualization) + +First, list pods in openshift-cnv namespace: + +**Parameters for pods_list_in_namespace**: +```json +{ + "namespace": "openshift-cnv", + "labelSelector": "app.kubernetes.io/component=cdi-deployment" +} +``` + +Then get logs using `pods_log`: + +**Parameters**: +```json +{ + "namespace": "openshift-cnv", + "name": "", + "tail": 100 +} +``` + +**CLI Fallback** (if MCP unavailable or easier via CLI): +```bash +oc logs -n openshift-cnv $(oc get pods -n openshift-cnv | grep cdi-deployment | awk '{print $1}') +``` + +**Common Error Messages**: +- `"volume cloning is not supported"` - CSI driver lacks clone capability +- `"cross namespace clone is not supported"` - Cloning between namespaces forbidden by storage +- `"source PVC not found"` - Source PVC doesn't exist or wrong namespace +- `"insufficient quota"` - Target namespace lacks storage quota +- `"source PVC not bound"` - Source PVC must be in Bound state for cloning +- `"StorageClass does not support cloning"` - Storage class configuration issue + +**Solutions** (Use MCP Tools First): + +1. **Check storage class clone support**: + + **MCP Tool**: `resources_list` (from openshift-virtualization) + + **Parameters**: + ```json + { + "apiVersion": "storage.k8s.io/v1", + "kind": "StorageClass" + } + ``` + + For each storage class in `.items[]`, check: + - `.metadata.name` (storage class name) + - `.provisioner` (CSI driver) + + Storage classes using CSI drivers typically support cloning. Look for provisioners like: + - `csi.ovirt.org` (oVirt CSI) + - `openshift-storage.rbd.csi.ceph.com` (Ceph RBD) + - `ebs.csi.aws.com` (AWS EBS CSI) + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get storageclass -o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner + ``` + +2. **Verify source PVC is bound**: + + Use `resources_get` from diagnostic step 6, check `.status.phase` (should be `Bound`). + + **CLI Fallback** (if MCP unavailable): + ```bash + oc get pvc -n -o jsonpath='{.status.phase}' + ``` + +3. **Check target namespace quota**: + + Use `resources_list` from diagnostic step 7 to check quota. + + If quota increase needed, this requires cluster admin privileges (cannot be done via MCP). + + **CLI Fallback** (if MCP unavailable): + ```bash + oc describe quota -n + ``` + +4. **Use snapshot-based cloning** (alternative method): + + **MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + + **Step 1**: Create VolumeSnapshot of source PVC + + **Parameters**: + ```json + { + "apiVersion": "snapshot.storage.k8s.io/v1", + "kind": "VolumeSnapshot", + "metadata": { + "name": "-snapshot", + "namespace": "" + }, + "spec": { + "source": { + "persistentVolumeClaimName": "" + } + } + } + ``` + + **Step 2**: Wait for snapshot to be ready (use `resources_get` to check `.status.readyToUse`) + + **Step 3**: Create new DataVolume from snapshot + + **Parameters**: + ```json + { + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "metadata": { + "name": "-rootdisk", + "namespace": "" + }, + "spec": { + "source": { + "snapshot": { + "name": "-snapshot", + "namespace": "" + } + }, + "storage": { + "resources": { + "requests": { + "storage": "50Gi" + } + }, + "storageClassName": "" + } + } + } + ``` + + **CLI Fallback** (YAML easier via CLI): + Ask user: "Snapshot-based cloning involves complex YAML. May I use `oc apply -f` instead?" + ```bash + cat <-snapshot + namespace: + spec: + source: + persistentVolumeClaimName: + EOF + + oc get volumesnapshot -snapshot -n + + cat <-rootdisk + namespace: + spec: + source: + snapshot: + name: -snapshot + namespace: + storage: + resources: + requests: + storage: 50Gi + storageClassName: + EOF + ``` + +5. **Use "new empty storage" option** (vm-clone skill): + - If cloning isn't supported, create VM with empty storage + - Manually copy data if needed + +6. **Cross-namespace cloning workaround**: + - Some storage backends require snapshot for cross-namespace cloning + - Create snapshot in source namespace, restore in target namespace (see solution 4 above) + +**Verification** (Use MCP Tools First): + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +Check DataVolume reached Succeeded phase: + +**Parameters**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "name": "" +} +``` + +Check `.status.phase` (should return `Succeeded`). + +Check PVC was created and bound: + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "name": "-rootdisk" +} +``` + +Check `.status.phase` (should return `Bound`). + +**CLI Fallback** (if MCP unavailable): +```bash +oc get datavolume -n -o jsonpath='{.status.phase}' +# Should return: Succeeded + +oc get pvc -rootdisk -n -o jsonpath='{.status.phase}' +# Should return: Bound +``` + +**Alternative**: If cloning continuously fails, use vm-creator skill to create new VM with container disk or DataSource instead. + +--- + + +--- + +[← Back to Index](INDEX.md) | [← Scheduling Errors](scheduling-errors.md) | [Lifecycle Errors →](lifecycle-errors.md) diff --git a/rh-virt/skills/vm-clone/SKILL.md b/rh-virt/skills/vm-clone/SKILL.md new file mode 100644 index 00000000..e8326139 --- /dev/null +++ b/rh-virt/skills/vm-clone/SKILL.md @@ -0,0 +1,1121 @@ +--- +name: vm-clone +description: | + Clone existing virtual machines for testing, scaling, or creating templates. Use this skill when users request: + - "Clone VM [source] to [target]" + - "Create a copy of VM [name]" + - "Duplicate VM [name] for testing" + - "Create 3 copies of template-vm" + + This skill clones VM configuration and optionally creates new storage or references existing storage. + + NOT for snapshots (use vm-snapshot for point-in-time backups). + +model: inherit +color: blue +--- + +# /vm-clone Skill + +Clone existing virtual machines in OpenShift Virtualization, creating new VMs with copied configuration and optional storage cloning. This skill is ideal for creating test environments, scaling workloads, or duplicating VM templates. + +## Critical: Human-in-the-Loop Requirements + +**IMPORTANT:** This skill creates new resources that consume cluster capacity. You MUST: + +1. **Before Cloning** + - Verify source VM exists and get full configuration + - Ask user for clone configuration (name, namespace, storage strategy) + - Present clone preview with resource impact + - Wait for explicit user confirmation + +2. **Configuration Confirmation** + - Display source VM details + - Show target VM configuration + - Indicate storage cloning strategy + - Estimate resource consumption (CPU, memory, storage) + - Ask: "Proceed with VM cloning? (yes/no)" + - Wait for explicit "yes" + +3. **Never Auto-Execute** + - **NEVER clone without user confirmation** + - **NEVER assume storage strategy** - always ask user + - **NEVER proceed if user says "no", "wait", "cancel"** + +**Why This Matters:** +- **Resource Consumption**: Clones consume cluster resources (CPU, memory, storage) +- **Storage Costs**: Storage cloning can consume significant disk space +- **Naming Conflicts**: Duplicate names cause errors +- **Network Configuration**: May need adjustment for clones + +## Prerequisites + +**Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) + +**Required MCP Tools**: +- `resources_get` (from openshift-virtualization) - Get source VM configuration +- `resources_create_or_update` (from openshift-virtualization) - Create cloned VM +- `resources_list` (from openshift-virtualization) - List DataVolumes, PVCs, VMs + +**Required Environment Variables**: +- `KUBECONFIG` - Path to Kubernetes configuration file with cluster access + +**Required Cluster Setup**: +- OpenShift cluster (>= 4.19) +- OpenShift Virtualization operator installed +- ServiceAccount with RBAC permissions to create VirtualMachine and PVC resources +- Source VM must exist + +### Prerequisite Verification + +**Before executing, verify MCP server availability:** + +1. **Check MCP Server Configuration** + - Verify `openshift-virtualization` exists in `.mcp.json` + - If missing → Report to user with setup instructions + +2. **Check Environment Variables** + - Verify `KUBECONFIG` is set (check presence only, never expose value) + - If missing → Report to user + +3. **Check RBAC Permissions** (optional verification) + - Verify ServiceAccount can create VirtualMachine resources + - Verify ServiceAccount can create PVC/DataVolume resources + +**Human Notification Protocol:** + +When prerequisites fail: + +``` +❌ Cannot execute vm-clone: MCP server 'openshift-virtualization' is not available + +📋 Setup Instructions: +1. Add openshift-virtualization to .mcp.json: + { + "mcpServers": { + "openshift-virtualization": { + "command": "podman", + "args": [ + "run", + "--rm", + "-i", + "--network=host", + "--userns=keep-id:uid=65532,gid=65532", + "-v", "${KUBECONFIG}:/kubeconfig:ro,Z", + "--entrypoint", "/app/kubernetes-mcp-server", + "quay.io/ecosystem-appeng/openshift-mcp-server:latest", + "--kubeconfig", "/kubeconfig", + "--toolsets", "core,kubevirt" + ], + "env": { + "KUBECONFIG": "${KUBECONFIG}" + } + } + } + } + +2. Set KUBECONFIG environment variable: + export KUBECONFIG="/path/to/your/kubeconfig" + +3. Restart Claude Code to reload MCP servers + +🔗 Documentation: https://github.com/openshift/openshift-mcp-server + +❓ How would you like to proceed? +Options: +- "setup" - Help configure the MCP server now +- "skip" - Skip this skill +- "abort" - Stop workflow + +Please respond with your choice. +``` + +⚠️ **SECURITY**: Never display actual KUBECONFIG path or credential values in output. + +## When to Use This Skill + +**Trigger this skill when:** +- User explicitly invokes `/vm-clone` command +- User wants to duplicate an existing VM +- User needs to create test/dev copies of production VMs +- User wants to scale horizontally by creating VM copies +- User wants to create VMs from a template VM + +**User phrases that trigger this skill:** +- "Clone VM web-server to web-server-test" +- "Create a copy of database-vm" +- "Duplicate production-vm for staging" +- "Make 3 copies of template-vm" +- "/vm-clone" (explicit command) + +**Do NOT use this skill when:** +- User wants to create a new VM from scratch → Use `/vm-creator` skill instead +- User wants a point-in-time backup → Use snapshots instead +- User wants to move/migrate a VM → Use migration tools instead +- User wants to resize a VM → Modify existing VM instead + +## Workflow + +### Step 1: Gather Source VM Information + +**Required Information from User:** +1. **Source VM Name** - Name of the VM to clone +2. **Source Namespace** - Namespace where source VM exists +3. **Target VM Name** - Name for the cloned VM +4. **Target Namespace** - Namespace for the cloned VM (can be same or different) + +If user doesn't provide all information, ask for missing details. + +**1.1: Verify Source VM Exists** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +**Expected Output**: Complete VirtualMachine resource specification + +**Error Handling**: +- If VM not found → Report error, suggest using vm-inventory to find VMs +- If permission denied → Report RBAC error + +**1.2: Check Target VM Name Availability** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +**If VM already exists:** +```markdown +❌ Target VM Name Already Exists + +**VM**: `` already exists in namespace `` + +**Options:** +1. Choose a different name for the clone +2. Delete the existing VM first (use vm-delete skill) +3. Cancel cloning operation + +What would you like to do? +``` + +**Wait for user decision.** + +**1.3: Discover Source VM Storage** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters for DataVolumes**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "labelSelector": "vm.kubevirt.io/name=" +} +``` + +**Parameters for PVCs** (if DataVolumes not found): +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "" +} +``` + +**Parse results:** +- Extract storage resource names referenced by source VM +- Calculate total storage size +- Determine if storage uses DataSources or container disks + +### Step 2: Ask User for Cloning Strategy + +**Present storage cloning options to user:** + +```markdown +## VM Cloning - Storage Strategy + +**Source VM**: `` (namespace: ``) + +### Storage Configuration + +**Source VM Storage:** +- DataVolume/PVC: `` (50Gi) +- Total Storage: 50Gi + +--- + +### Cloning Options + +**How should storage be cloned?** + +**Option 1: Clone Storage** (full copy) +- Creates new DataVolume/PVC for target VM +- Clones all data from source storage +- Target VM has independent storage +- Storage required: 50Gi (new allocation) +- Time: ~5-10 minutes (depends on size) +- Use case: Independent test/dev environments + +**Option 2: Reference Existing Storage** (shared storage - not recommended) +- Target VM references same PVC as source +- No storage cloning +- ⚠️ Both VMs share the same disk (dangerous!) +- Storage required: 0Gi (no new allocation) +- Use case: Only if you know what you're doing + +**Option 3: Create New Empty Storage** (fresh disk) +- Creates new empty DataVolume/PVC for target VM +- Does NOT clone data from source +- Target VM starts with clean disk +- Storage required: 50Gi (new allocation) +- Use case: Creating VMs from templates without data + +**Option 4: Cancel** +- No cloning performed + +--- + +**Select storage strategy** (1, 2, 3, or 4): +``` + +**Wait for user to select option 1, 2, 3, or 4.** + +**Handle user response:** +- If "4" or "cancel" → Cancel operation, stop workflow +- If "1" → Proceed with storage cloning (clone_storage=true) +- If "2" → Proceed with shared storage (share_storage=true) + warn user +- If "3" → Proceed with new empty storage (new_storage=true) + +**If user selects Option 2 (shared storage), issue warning:** +```markdown +⚠️ WARNING: Shared Storage is Dangerous + +You selected to share storage between source and target VMs. + +**Risks:** +- Both VMs writing to the same disk will cause **data corruption** +- Only safe if source VM is stopped and will remain stopped +- Not recommended for production use + +**Recommendation**: Use Option 1 (Clone Storage) instead for independent VMs. + +**Proceed with shared storage anyway? (yes/cancel)** +``` + +Wait for explicit "yes" to continue with shared storage. + +### Step 3: Present Clone Configuration for Confirmation + +**After determining cloning strategy, present complete configuration:** + +```markdown +## VM Clone Configuration - Review + +**Please review the clone configuration:** + +### Source VM +- **Name**: `` +- **Namespace**: `` +- **Instance Type**: +- **vCPU**: , **Memory**: +- **Storage**: +- **Status**: + +### Target VM (Clone) +- **Name**: `` +- **Namespace**: `` +- **Instance Type**: (copied from source) +- **vCPU**: , **Memory**: (copied from source) +- **Storage**: +- **Initial Status**: Stopped (will not auto-start) + +### Storage Strategy + +- **Strategy**: Clone Storage (full copy) +- **New Storage**: 50Gi DataVolume/PVC will be created +- **Clone Time**: ~5-10 minutes +- **Storage Class**: (from source) + + + +- **Strategy**: Shared Storage ⚠️ +- **WARNING**: Both VMs will share the same disk +- **Ensure source VM is stopped** to avoid data corruption + + + +- **Strategy**: New Empty Storage +- **New Storage**: 50Gi empty DataVolume/PVC will be created +- **No data cloned** from source + + +### Resource Impact +- **CPU**: vCPUs consumed +- **Memory**: RAM consumed +- **Storage**: (if applicable) + +### What Will Be Copied +- ✓ Instance type and preference +- ✓ vCPU and memory configuration +- ✓ Network configuration +- ✓ Tolerations and affinity rules +- ✓ Cloud-init configuration (if any) + +- ✓ Disk data (full clone) + + +### What Will NOT Be Copied +- ✗ VM running state (clone starts stopped) +- ✗ IP addresses (new IPs assigned) +- ✗ Hostname (uses target VM name) +- ✗ MAC addresses (new MACs generated) + +--- + +**Proceed with VM cloning? (yes/no)** +``` + +**Wait for user confirmation.** + +**Handle response:** +- If "yes" → Proceed to Step 4 (execute cloning) +- If "no", "cancel", "wait", or anything else → Cancel operation + +**On cancellation:** +```markdown +VM cloning cancelled by user. No resources were created. +``` + +**STOP workflow**. + +### Step 4: Execute VM Cloning + +**ONLY PROCEED AFTER**: +- ✓ Step 1: Source VM validated, target name available +- ✓ Step 2: User selected storage strategy +- ✓ Step 3: User confirmed clone configuration + +**4.1: Prepare Cloned VM Specification** + +**From the source VM resource obtained in Step 1.1**, create a modified spec: + +1. **Change metadata**: + - `metadata.name` → `` + - `metadata.namespace` → `` + - Remove `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp` + - Remove status section entirely + +2. **Update storage references**: + - If `clone_storage=true`: Create new DataVolume with source as sourceRef + - If `share_storage=true`: Keep existing PVC references (no changes) + - If `new_storage=true`: Create new empty DataVolume + +3. **Update runStrategy**: + - Set to `Halted` (clone starts stopped) + +4. **Generate new firmware UUIDs**: + - Generate new `spec.template.spec.domain.firmware.uuid` + - Generate new `spec.template.spec.domain.firmware.serial` + +5. **Preserve from source**: + - Instance type and preference + - Tolerations + - Network configuration + - Cloud-init (if any) + +**4.2: Create Storage Resources (if clone_storage=true or new_storage=true)** + +**If cloning storage:** + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Create DataVolume that clones from source:** +```yaml +apiVersion: cdi.kubevirt.io/v1beta1 +kind: DataVolume +metadata: + name: -rootdisk + namespace: +spec: + source: + pvc: + name: + namespace: + storage: + resources: + requests: + storage: + storageClassName: +``` + +**If creating new empty storage:** + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Create empty DataVolume:** +```yaml +apiVersion: cdi.kubevirt.io/v1beta1 +kind: DataVolume +metadata: + name: -rootdisk + namespace: +spec: + source: + blank: {} + storage: + resources: + requests: + storage: + storageClassName: +``` + +**Report progress:** +```markdown +📦 Creating storage for cloned VM... + +⏳ Cloning storage (this may take 5-10 minutes)... + + +✓ Creating new empty storage... + +``` + +**4.3: Create Cloned VirtualMachine** + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Parameters**: Use the prepared VM spec from Step 4.1 + +**Expected Output**: VirtualMachine created successfully + +**Error Handling**: +- If creation fails → Report error, rollback storage if created +- If permission denied → Report RBAC error +- If namespace doesn't exist → Report namespace error + +**Report progress:** +```markdown +🖥️ Creating cloned VirtualMachine... +✓ VirtualMachine `` created in namespace `` +``` + +**4.4: Monitor Storage Cloning Progress (if clone_storage=true)** + +**If storage is being cloned, monitor DataVolume status:** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "name": "-rootdisk" +} +``` + +**Check `status.phase`:** +- `Pending` → Still cloning +- `Succeeded` → Clone complete +- `Failed` → Clone failed + +**Report progress every 30 seconds:** +```markdown +⏳ Storage cloning in progress... + Phase: + Progress: +``` + +**Wait up to 15 minutes for cloning to complete.** + +### Step 5: Report Cloning Results + +**On successful clone:** + +```markdown +## ✓ VM Cloned Successfully + +**Source VM**: `` (namespace: ``) +**Target VM**: `` (namespace: ``) + +### Cloned VM Details +- **Name**: `` +- **Namespace**: `` +- **Instance Type**: +- **vCPU**: , **Memory**: +- **Storage**: +- **Status**: Stopped (ready to start) + + +### Storage Cloning +- ✓ Storage cloned successfully +- ✓ DataVolume: `-rootdisk` (50Gi) +- ✓ Clone completed in +- ✓ Target VM has independent storage + + + +### Storage Creation +- ✓ New empty storage created +- ✓ DataVolume: `-rootdisk` (50Gi) +- ℹ️ Storage is empty (no data from source) + + + +### Storage Configuration +- ⚠️ Shared storage configured +- ⚠️ Both VMs share PVC: `` +- ⚠️ **Keep source VM stopped to avoid data corruption** + + +--- + +### Next Steps + +**To start the cloned VM:** +``` +"Start VM in namespace " +``` + +**To view VM details:** +``` +"Get details of VM " +``` + +**To access the VM console:** +- VNC Console: OpenShift Console → Virtualization → VirtualMachines → `` → Console +- Serial Console: `virtctl console -n ` + +### Important Notes + + +- ✓ The cloned VM has **independent storage** from the source +- ✓ Changes to the clone will **not affect** the source VM +- ✓ You can safely start and modify both VMs + + + +- ⚠️ **CRITICAL**: Both VMs share the same disk +- ⚠️ **Never run both VMs simultaneously** - data corruption will occur +- ⚠️ Only start one VM at a time +- ℹ️ Recommended: Convert to cloned storage using vm-snapshot or manual PVC cloning + + + +- ℹ️ The clone has **empty storage** (no data from source) +- ℹ️ OS installation or configuration may be required +- ℹ️ Useful for creating fresh VMs from a template configuration + + +### Configuration Differences +- **Hostname**: Will use `` (different from source) +- **IP Address**: New IP will be assigned (different from source) +- **MAC Address**: New MAC addresses generated (different from source) +- **Firmware UUID**: New UUID generated (different from source) + +--- + +**To verify the clone:** +``` +"List VMs in namespace " +``` + +Both source and target VMs should appear in the inventory. +``` + +**On cloning failure:** + +**OPTIONAL**: If cloning operation fails, consult documentation for common cloning failure scenarios. + +**Document Consultation** (OPTIONAL - when cloning fails): +1. **Action**: Read [storage-errors.md](../../docs/troubleshooting/storage-errors.md) using the Read tool to understand VM cloning failure scenarios, storage provisioning issues, and DataVolume cloning errors +2. **Output to user**: "I consulted [storage-errors.md](../../docs/troubleshooting/storage-errors.md) to understand potential causes for the cloning failure." + +**When to consult**: +- Storage cloning fails (DataVolume provisioning errors) +- VM creation fails during cloning workflow +- PVC clone not supported errors +- Storage class issues during cloning + +**When NOT to consult**: +- Simple "VM already exists" errors (clear cause) +- RBAC permission errors (clear cause) +- Namespace not found errors (clear cause) + +```markdown +## ❌ VM Cloning Failed + +**Error**: + +**Source VM**: `` (namespace: ``) +**Target VM**: `` (namespace: ``) + +**Common Causes:** +- **Insufficient storage quota** - Namespace lacks storage capacity for clone +- **Insufficient RBAC permissions** - ServiceAccount lacks create permissions +- **Storage class not available** - Target namespace cannot access storage class +- **PVC clone not supported** - Storage class doesn't support cloning +- **Source VM still running** - Some storage backends require source VM to be stopped + +**Troubleshooting Steps:** + +Consult [storage-errors.md](../../docs/troubleshooting/storage-errors.md) for MCP-first diagnostic procedures: + +1. **Check storage quota:** + + See "ErrorDataVolumeNotReady - Insufficient Storage Quota" section + - Use `resources_list` to check ResourceQuota in target namespace + +2. **Check permissions:** + + Use `resources_list` with appropriate apiVersion/kind to verify RBAC permissions + (Note: `oc auth can-i` has no direct MCP equivalent - use CLI if needed) + +3. **Check storage class:** + + See "DataVolume Cloning Failures" section + - Use `resources_get` to check StorageClass configuration + - Use `resources_list` to list available storage classes + +4. **Check if source VM is stopped:** + + Use vm-inventory skill: "Show status of VM " + +5. **Check DataVolume status** (if storage cloning): + + See "DataVolume Cloning Failures" section + - Use `resources_get` to check DataVolume status and phase + +**Partial Resources Created:** + +Some resources may have been created before the failure: +- VirtualMachine: `` (may need cleanup) +- DataVolume: `-rootdisk` (may need cleanup) + +To clean up partial resources: +``` +"Delete VM in namespace " +``` + + +Would you like help troubleshooting this error? +``` + +## Advanced Features + +### Batch Cloning (Multiple Copies) + +**User request:** "Create 3 copies of template-vm named web-01, web-02, web-03" + +**Workflow:** +1. Execute Step 1 (validate source VM once) +2. Generate target names: web-01, web-02, web-03 +3. Check all target names for availability +4. Present combined cloning scope for all copies +5. Ask for storage strategy (applies to all clones) +6. Confirm batch operation +7. Execute cloning for each VM sequentially + +**Batch confirmation:** +```markdown +## Batch VM Cloning - Review + +**Source VM**: `template-vm` + +**Target VMs**: +1. `web-01` (namespace: `production`) +2. `web-02` (namespace: `production`) +3. `web-03` (namespace: `production`) + +**Storage Strategy**: Clone Storage (full copy for each) + +**Total Resource Impact**: +- **VMs**: 3 new VMs +- **Storage**: 150Gi (3 × 50Gi) +- **vCPUs**: 12 total (3 × 4) +- **Memory**: 24Gi total (3 × 8Gi) + +**Estimated Time**: ~20-30 minutes (clones created sequentially) + +Proceed with batch cloning? (yes/no) +``` + +### Cross-Namespace Cloning + +**User request:** "Clone production-vm from production namespace to staging namespace" + +**Workflow:** +- Source namespace: `production` +- Target namespace: `staging` +- Storage cloning may require cross-namespace PVC access +- Present warning if namespaces have different quotas/policies + +**Cross-namespace note:** +```markdown +ℹ️ **Cross-Namespace Cloning** + +**Source**: `production` namespace +**Target**: `staging` namespace + +**Considerations**: +- Storage will be cloned from `production` to `staging` +- Network policies may differ between namespaces +- Resource quotas may differ between namespaces +- RBAC permissions required in both namespaces + +Verify that the `staging` namespace has sufficient resources. +``` + +### Clone with Modifications + +**Future enhancement: Allow users to modify clone configuration:** + +``` +"Clone database-vm to test-db with 8Gi memory instead of 16Gi" +``` + +Modifications could include: +- Instance type/size +- Storage size +- Network configuration +- Cloud-init customization + +## Common Issues + +### Issue 1: Target VM Name Already Exists + +**Error**: "VirtualMachine 'web-clone' already exists in namespace 'dev'" + +**Solution:** +1. Choose a different target name +2. Delete existing VM with same name (if safe to do so) +3. Use vm-inventory to check existing VMs + +### Issue 2: Insufficient Storage Quota + +**Error**: "Namespace quota exceeded" + +**Solution:** +- Check namespace resource quotas +- Request quota increase from cluster admin +- Use shared storage option (if appropriate) +- Delete unused PVCs to free quota + +### Issue 3: Storage Class Not Accessible + +**Error**: "StorageClass 'xyz' not found or not accessible" + +**Solution:** +- Verify storage class exists in target namespace +- Check if storage class allows cross-namespace cloning +- Use different storage class for target +- Contact cluster admin for access + +### Issue 4: PVC Clone Not Supported + +**Error**: "Storage backend does not support PVC cloning" + +**Solution:** +- Some storage classes don't support CSI volume cloning +- Use "new empty storage" option instead +- Manually snapshot and restore (alternative approach) +- Check storage class capabilities + +### Issue 5: Source VM Running During Clone + +**Error**: "Cannot clone from running VM with this storage backend" + +**Solution:** +- Stop source VM before cloning +- Use snapshot-based cloning instead +- Check storage backend requirements + +## Dependencies + +### Required MCP Servers +- `openshift-virtualization` - OpenShift MCP server with core and kubevirt toolsets + +### Required MCP Tools +- `resources_get` (from openshift-virtualization) - Get source VM and storage details + - Parameters: apiVersion, kind, namespace, name + - Source: https://github.com/openshift/openshift-mcp-server/blob/main/pkg/toolsets/core/resources.go + +- `resources_create_or_update` (from openshift-virtualization) - Create cloned VM and storage + - Parameters: resource (YAML/JSON) + - Source: https://github.com/openshift/openshift-mcp-server/blob/main/pkg/toolsets/core/resources.go + +- `resources_list` (from openshift-virtualization) - List DataVolumes, PVCs, VMs + - Parameters: apiVersion, kind, namespace, labelSelector + - Source: https://github.com/openshift/openshift-mcp-server/blob/main/pkg/toolsets/core/resources.go + +### Related Skills +- `vm-creator` - Create new VMs from scratch (alternative to cloning) +- `vm-inventory` - List and verify source/target VMs +- `vm-lifecycle-manager` - Start cloned VMs after creation +- `vm-delete` - Clean up failed clones or unwanted copies + +### Reference Documentation +- [storage-errors.md](../../docs/troubleshooting/storage-errors.md) - VM cloning failure scenarios, storage provisioning issues, and DataVolume cloning errors (optionally consulted when cloning operations fail) +- [Troubleshooting INDEX](../../docs/troubleshooting/INDEX.md) - Navigation hub for discovering additional error categories when encountering unexpected issues outside the categories above +- [OpenShift Virtualization Cloning](https://docs.openshift.com/container-platform/latest/virt/virtual_machines/cloning_vms/virt-cloning-vm.html) +- [DataVolume Cloning](https://github.com/kubevirt/containerized-data-importer/blob/main/doc/datavolumes.md#cloning) +- [KubeVirt VirtualMachine API](https://kubevirt.io/api-reference/) +- [CSI Volume Cloning](https://kubernetes.io/docs/concepts/storage/volume-pvc-datasource/) + +## Security Considerations + +- **RBAC Enforcement**: Requires create permissions in target namespace +- **Namespace Isolation**: Cannot clone across namespaces without proper permissions +- **Storage Quotas**: Respects namespace storage quotas +- **MAC Address Generation**: Automatic generation prevents MAC conflicts +- **UUID Generation**: New firmware UUIDs prevent conflicts +- **No Credential Exposure**: KUBECONFIG never exposed in output +- **Audit Trail**: All cloning operations logged in Kubernetes API audit logs + +## Example Usage + +### Example 1: Simple Clone with Storage + +``` +User: "Clone web-server to web-server-test in namespace dev" + +Agent: [Invokes /vm-clone skill] + [Step 1: Validation] + [Calls resources_get for source VM] + [Checks target name availability] + [Discovers source storage: 50Gi] + +## VM Cloning - Storage Strategy + +Source VM: `web-server` (namespace: `production`) + +Storage Configuration: +- DataVolume/PVC: `web-server-rootdisk` (50Gi) + +Select storage strategy (1, 2, 3, or 4): + +User: "1" (Clone Storage) + +Agent: [Step 3: Present configuration] + +## VM Clone Configuration - Review + +Source VM: +- Name: `web-server` +- Namespace: `production` +- Instance Type: u1.medium +- vCPU: 2, Memory: 4Gi +- Storage: 50Gi + +Target VM (Clone): +- Name: `web-server-test` +- Namespace: `dev` +- Instance Type: u1.medium (copied) +- vCPU: 2, Memory: 4Gi (copied) +- Storage: Clone Storage (50Gi new allocation) + +Storage Strategy: +- Strategy: Clone Storage (full copy) +- New Storage: 50Gi DataVolume/PVC +- Clone Time: ~5-10 minutes + +Resource Impact: +- CPU: 2 vCPUs consumed +- Memory: 4Gi RAM consumed +- Storage: 50Gi + +Proceed with VM cloning? (yes/no) + +User: "yes" + +Agent: [Step 4: Execute cloning] + [Creates DataVolume with PVC clone source] + +📦 Creating storage for cloned VM... +⏳ Cloning storage (this may take 5-10 minutes)... + + [Creates VirtualMachine] + +🖥️ Creating cloned VirtualMachine... +✓ VirtualMachine `web-server-test` created in namespace `dev` + + [Monitors DataVolume status] + +⏳ Storage cloning in progress... + Phase: CloneInProgress + Progress: 45% + +⏳ Storage cloning in progress... + Phase: Succeeded + + [Step 5: Report results] + +## ✓ VM Cloned Successfully + +Source VM: `web-server` (namespace: `production`) +Target VM: `web-server-test` (namespace: `dev`) + +Cloned VM Details: +- Name: `web-server-test` +- Namespace: `dev` +- Instance Type: u1.medium +- vCPU: 2, Memory: 4Gi +- Storage: 50Gi +- Status: Stopped (ready to start) + +Storage Cloning: +- ✓ Storage cloned successfully +- ✓ DataVolume: `web-server-test-rootdisk` (50Gi) +- ✓ Clone completed in 7m23s +- ✓ Target VM has independent storage + +Next Steps: + +To start the cloned VM: +``` +"Start VM web-server-test in namespace dev" +``` +``` + +### Example 2: Clone with New Empty Storage + +``` +User: "Clone template-vm to app-01 with empty storage" + +Agent: [Invokes /vm-clone skill] + [Validation complete] + [Step 2: Storage strategy] + +Select storage strategy (1, 2, 3, or 4): + +User: "3" (Create New Empty Storage) + +Agent: [Presents configuration] + +Storage Strategy: +- Strategy: New Empty Storage +- New Storage: 30Gi empty DataVolume/PVC +- No data cloned from source + +Proceed with VM cloning? (yes/no) + +User: "yes" + +Agent: [Creates empty DataVolume and VM] + +## ✓ VM Cloned Successfully + +Storage Creation: +- ✓ New empty storage created +- ✓ DataVolume: `app-01-rootdisk` (30Gi) +- ℹ️ Storage is empty (no data from source) + +Important Notes: +- ℹ️ The clone has empty storage (no data from source) +- ℹ️ OS installation or configuration may be required +``` + +### Example 3: Batch Cloning + +``` +User: "Create 3 copies of template-vm named web-01, web-02, web-03 in production namespace" + +Agent: [Invokes /vm-clone skill] + [Validates source VM] + [Generates target names] + [Checks all names for availability] + +## Batch VM Cloning - Review + +Source VM: `template-vm` + +Target VMs: +1. `web-01` (namespace: `production`) +2. `web-02` (namespace: `production`) +3. `web-03` (namespace: `production`) + +Storage Strategy: Clone Storage (full copy for each) + +Total Resource Impact: +- VMs: 3 new VMs +- Storage: 90Gi (3 × 30Gi) +- vCPUs: 6 total (3 × 2) +- Memory: 12Gi total (3 × 4Gi) + +Estimated Time: ~20-30 minutes + +Proceed with batch cloning? (yes/no) + +User: "yes" + +Agent: [Clones VMs sequentially] + +📦 Cloning VM 1 of 3: web-01... +✓ web-01 created and storage cloning in progress + +📦 Cloning VM 2 of 3: web-02... +✓ web-02 created and storage cloning in progress + +📦 Cloning VM 3 of 3: web-03... +✓ web-03 created and storage cloning in progress + +## ✓ Batch Cloning Completed + +Successfully cloned 3 VMs from `template-vm`: +- ✓ web-01 (production) +- ✓ web-02 (production) +- ✓ web-03 (production) + +Total resources created: +- 3 VirtualMachines +- 90Gi storage +- All VMs in Stopped state (ready to start) +``` + +### Example 4: Shared Storage Warning + +``` +User: "Clone db-vm to db-test" + +Agent: [Validation complete] + +Select storage strategy (1, 2, 3, or 4): + +User: "2" (Reference Existing Storage) + +Agent: [Issues warning] + +⚠️ WARNING: Shared Storage is Dangerous + +You selected to share storage between source and target VMs. + +Risks: +- Both VMs writing to the same disk will cause data corruption +- Only safe if source VM is stopped and will remain stopped + +Recommendation: Use Option 1 (Clone Storage) instead. + +Proceed with shared storage anyway? (yes/cancel) + +User: "cancel" + +Agent: VM cloning cancelled by user. No resources were created. +``` diff --git a/rh-virt/skills/vm-creator/SKILL.md b/rh-virt/skills/vm-creator/SKILL.md index dc0610a4..a710af9c 100644 --- a/rh-virt/skills/vm-creator/SKILL.md +++ b/rh-virt/skills/vm-creator/SKILL.md @@ -137,36 +137,393 @@ Please respond with your choice. - User wants to list VMs → Use `/vm-inventory` skill instead - User only wants information about VMs (not creation) → Use `/vm-inventory` skill instead +## CRITICAL: MCP Tools First Policy + +**MANDATORY REQUIREMENT**: You MUST ALWAYS use MCP tools from the openshift-virtualization server for ALL cluster operations. + +**MCP Tools Available:** +- `namespaces_list` - List all namespaces +- `resources_list` - List resources (StorageClass, VirtualMachine, etc.) +- `resources_get` - Get specific resource details +- `resources_create_or_update` - Create or update resources +- `resources_delete` - Delete resources +- `pods_list` - List pods +- `pods_exec` - Execute commands in pods +- `events_list` - List cluster events +- And many more... + +**Policy:** +1. **ALWAYS check if an MCP tool exists** for the operation you need to perform +2. **ONLY use kubectl/oc CLI commands** when: + - No equivalent MCP tool exists for that specific operation + - The MCP tool has been tried and failed + - You have explicit confirmation that the MCP approach is not possible + +**Examples:** +- ❌ WRONG: `kubectl get namespaces` → ✅ CORRECT: Use `namespaces_list` MCP tool +- ❌ WRONG: `kubectl get storageclass -o json` → ✅ CORRECT: Use `resources_list` MCP tool with apiVersion="storage.k8s.io/v1", kind="StorageClass" +- ❌ WRONG: `kubectl get vm -n ` → ✅ CORRECT: Use `resources_get` MCP tool with apiVersion="kubevirt.io/v1", kind="VirtualMachine" +- ❌ WRONG: `kubectl config view --minify` → ⚠️ ACCEPTABLE: No MCP equivalent exists for kubeconfig context detection + +**Why this matters:** +- MCP tools provide structured, validated outputs +- Better error handling and user experience +- Consistent interface across all operations +- Reduced dependency on CLI tools +- Better integration with Claude Code environment + +**If you catch yourself about to use kubectl/oc:** +1. STOP +2. Check the available MCP tools list above +3. Use the MCP tool instead +4. Only proceed with kubectl/oc if absolutely no alternative exists + ## Workflow ### Step 1: Gather VM Requirements and Confirm Configuration -**Collect information from user**, then present for confirmation before proceeding. +**CRITICAL**: When user does NOT explicitly specify all VM parameters, you MUST use the AskUserQuestion tool to present an interactive menu. + +#### Step 1a: Determine What Information Is Missing + +**First: Detect Current Namespace from KUBECONFIG Context** + +Before asking for namespace, detect the current context: + +**Note**: There is no MCP tool equivalent for detecting the current kubeconfig context namespace. This is one of the rare cases where kubectl CLI is acceptable: + +```bash +# Get current namespace from kubeconfig context +kubectl config view --minify -o jsonpath='{..namespace}' 2>/dev/null || echo "default" +``` + +**Store this as the default namespace** to use in the interactive menu. + +**VM Name Validation Rules:** +- Lowercase alphanumeric characters or hyphens only +- Must start with a letter +- Must end with alphanumeric character +- Maximum 63 characters +- Must be unique within the namespace + +**Check if user explicitly provided:** +1. **VM Name** - Required, always ask if not provided. Validate against rules above. +2. **Namespace** - Required, always ask if not provided. Use detected namespace as default. +3. **Operating System** - If not specified, use interactive menu (default: `fedora`) +4. **Size** - If not specified, use interactive menu (default: `medium`) +5. **Storage** - If not specified, use interactive menu (default: `30Gi`) +6. **Storage Class** - If not specified, use interactive menu (default: cluster default) +7. **Performance Profile** - If not specified, use interactive menu (default: `u1` general-purpose) +8. **Autostart** - If not specified, use interactive menu (default: `false`) + +#### Step 1b: Gather Missing Parameters via Interactive Menu + +**BEFORE presenting the menu, gather cluster information using MCP tools:** + +1. **Detect current namespace** (if not provided by user): + + **Note**: No MCP equivalent exists - use kubectl: + ```bash + kubectl config view --minify -o jsonpath='{..namespace}' 2>/dev/null || echo "default" + ``` + +2. **Get available namespaces**: + + **Use MCP Tool**: `namespaces_list` (from openshift-virtualization) + + **Parameters**: None + + This returns a list of all namespaces in the cluster. + +3. **Get available StorageClasses and identify default**: -**Required Parameters:** -1. **VM Name** - Name for the virtual machine - - Example: "web-server", "database-01", "test-vm" + **Use MCP Tool**: `resources_list` (from openshift-virtualization) -2. **Namespace** - OpenShift namespace where VM will be created - - Example: "vms", "production", "dev-environment" + **Parameters**: + - `apiVersion`: "storage.k8s.io/v1" + - `kind`: "StorageClass" -**Optional Parameters (with defaults):** -3. **Operating System** (`workload`) - Default: `"fedora"` - - Supported: `fedora`, `ubuntu`, `centos`, `centos-stream`, `debian`, `rhel`, `opensuse`, `opensuse-tumbleweed`, `opensuse-leap` - - Can also accept full container disk image URLs + This returns all StorageClass resources with full details including annotations and specifications. -4. **Size** (`size`) - VM sizing hint - - Options: `small`, `medium`, `large`, `xlarge` - - If not specified, MCP server uses default instance type +4. **Analyze StorageClasses** for the menu: + - **Performance**: Check `volumeBindingMode` (Immediate = faster provisioning, WaitForFirstConsumer = delayed) + - **Live Migration Support**: Check access modes - RWX (ReadWriteMany) supports live migration, RWO (ReadWriteOnce) does not + - Extract from MCP tool response: + - `.metadata.annotations["storageclass.kubernetes.io/is-default-class"]` = "true" for default + - `.volumeBindingMode` for performance + - `.provisioner` for hints (rbd/cephfs = likely RWX support) -5. **Storage** (`storage`) - Default: `"30Gi"` - - Root disk size: `"30Gi"`, `"50Gi"`, `"100Gi"`, etc. +**If ANY parameters are missing, use AskUserQuestion tool with the following structure:** -6. **Autostart** (`autostart`) - Default: `false` - - `true`: VM starts automatically after creation - - `false`: VM created in halted state +**Use the AskUserQuestion tool** to present an interactive menu for missing configuration parameters: -**After gathering parameters, present configuration for confirmation:** +```json +{ + "questions": [ + { + "question": "What is the name for your new virtual machine?", + "header": "VM Name", + "multiSelect": false, + "options": [ + { + "label": "Enter custom name", + "description": "Provide VM name (lowercase, alphanumeric+hyphens, start with letter, max 63 chars)" + } + ] + }, + { + "question": "Which namespace should this VM be created in?", + "header": "Namespace", + "multiSelect": false, + "options": [ + { + "label": " (Current)", + "description": "Your current namespace from kubeconfig context" + }, + { + "label": "Other namespace", + "description": "Specify a different namespace from: " + } + ] + }, + { + "question": "Which operating system should the VM run?", + "header": "OS", + "multiSelect": false, + "options": [ + { + "label": "Fedora (Recommended)", + "description": "Latest Fedora Linux distribution - general purpose" + }, + { + "label": "Ubuntu", + "description": "Ubuntu Linux distribution - popular for web services" + }, + { + "label": "RHEL", + "description": "Red Hat Enterprise Linux - enterprise-grade stability" + }, + { + "label": "CentOS Stream", + "description": "CentOS Stream - upstream for RHEL" + }, + { + "label": "Debian", + "description": "Debian Linux - stable and minimal" + }, + { + "label": "OpenSUSE", + "description": "OpenSUSE - community-driven Linux distribution" + } + ] + }, + { + "question": "What performance profile do you need?", + "header": "Performance", + "multiSelect": false, + "options": [ + { + "label": "General Purpose (u1) (Recommended)", + "description": "Balanced CPU/memory ratio - suitable for most workloads" + }, + { + "label": "Compute Optimized (c1)", + "description": "More CPU, less memory - CPU-intensive applications" + }, + { + "label": "Memory Optimized (m1)", + "description": "More memory, less CPU - memory-intensive applications" + }, + { + "label": "Overcommitted (o1)", + "description": "Lower resource guarantees - development/testing environments" + } + ] + }, + { + "question": "What size should the VM be?", + "header": "Size", + "multiSelect": false, + "options": [ + { + "label": "Small", + "description": "1 vCPU, 2Gi RAM - lightweight workloads" + }, + { + "label": "Medium (Recommended)", + "description": "2-4 vCPU, 4-8Gi RAM - general purpose applications" + }, + { + "label": "Large", + "description": "4-8 vCPU, 8-16Gi RAM - resource-intensive workloads" + }, + { + "label": "XLarge", + "description": "8+ vCPU, 16+ Gi RAM - high-performance applications" + } + ] + }, + { + "question": "How much storage should the VM have?", + "header": "Storage", + "multiSelect": false, + "options": [ + { + "label": "30Gi (Recommended)", + "description": "30 Gigabytes - sufficient for most workloads" + }, + { + "label": "50Gi", + "description": "50 Gigabytes - moderate storage needs" + }, + { + "label": "100Gi", + "description": "100 Gigabytes - large storage requirements" + }, + { + "label": "Custom size", + "description": "Specify a custom storage size (e.g., 75Gi, 200Gi)" + } + ] + }, + { + "question": "Which StorageClass should be used for the VM disk?", + "header": "Storage Class", + "multiSelect": false, + "options": [ + { + "label": " (Default)", + "description": "Cluster default - , " + }, + { + "label": "", + "description": ", " + }, + { + "label": "", + "description": ", " + } + ] + }, + { + "question": "Should the VM start automatically after creation?", + "header": "Autostart", + "multiSelect": false, + "options": [ + { + "label": "No (Recommended)", + "description": "Create VM in stopped state - start manually when ready" + }, + { + "label": "Yes", + "description": "VM will start automatically after creation" + } + ] + } + ] +} +``` + +**Important Notes:** +- Only ask questions for parameters NOT explicitly provided by the user +- If user said "Create a Fedora VM", don't ask about OS - they specified Fedora +- If user said "in namespace production", don't ask about namespace +- Always include "Recommended" on the default options +- The "Other" option is automatically added by the system for each question +- For namespace: Use detected current namespace from kubeconfig context as default +- For StorageClass: Populate options dynamically from cluster, mark default + - Performance hints: "Immediate provisioning" or "Delayed provisioning" + - Live migration hints: "Supports live migration (RWX)" or "No live migration (RWO)" + +#### Step 1c: Process Interactive Menu Responses + +**After receiving AskUserQuestion responses, map them to VM parameters:** + +1. **VM Name**: + - Use custom text input from user + - **Validate** against rules: lowercase, alphanumeric+hyphens, start with letter, max 63 chars + - If invalid: Prompt user to provide valid name + +2. **Namespace**: + - " (Current)" → use detected namespace + - "Other namespace" → use custom text input + - Store for use in existence check (Step 1d) + +3. **Operating System**: + - "Fedora (Recommended)" → `"fedora"` + - "Ubuntu" → `"ubuntu"` + - "RHEL" → `"rhel"` + - "CentOS Stream" → `"centos-stream"` + - "Debian" → `"debian"` + - "OpenSUSE" → `"opensuse"` + - Other → custom text input (could be `opensuse-tumbleweed`, `opensuse-leap`, container image URL) + +4. **Performance Profile**: + - "General Purpose (u1) (Recommended)" → `"u1"` + - "Compute Optimized (c1)" → `"c1"` + - "Memory Optimized (m1)" → `"m1"` + - "Overcommitted (o1)" → `"o1"` + +5. **Size**: + - "Small" → `"small"` + - "Medium (Recommended)" → `"medium"` + - "Large" → `"large"` + - "XLarge" → `"xlarge"` + +6. **Storage**: + - "30Gi (Recommended)" → `"30Gi"` + - "50Gi" → `"50Gi"` + - "100Gi" → `"100Gi"` + - "Custom size" → use custom text input (validate format: number + unit, e.g., "75Gi") + +7. **StorageClass**: + - " (Default)" → use cluster default SC name + - "" → use selected SC name + - Store SC name for inclusion in vm_create parameters (if MCP tool supports it) + +8. **Autostart**: + - "No (Recommended)" → `false` + - "Yes" → `true` + +#### Step 1d: Check VM Existence and Present Final Configuration for Confirmation + +**CRITICAL: Before presenting configuration, check if VM already exists** + +**Use MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +**Handle result:** +- **If VM exists**: + ```markdown + ⚠️ **VM Already Exists** + + A VirtualMachine named `` already exists in namespace ``. + + **Current VM status**: + + **Options:** + 1. Choose a different VM name + 2. Delete existing VM first (use vm-inventory and manual deletion) + 3. Cancel operation + + What would you like to do? + ``` + **STOP and wait for user decision.** Do NOT proceed with configuration confirmation. + +- **If VM does not exist**: Proceed to configuration confirmation below + +--- + +**After confirming VM doesn't exist, present configuration table:** ```markdown ## Virtual Machine Configuration @@ -175,12 +532,14 @@ Please respond with your choice. | Parameter | Value | Notes | |-----------|-------|-------| -| VM Name | `web-server` | [from user input] | -| Namespace | `vms` | [from user input] | -| Operating System | `fedora` | [default / user specified] | -| Size | `medium` | [user specified / omitted for default] | -| Storage | `50Gi` | [user specified / default: 30Gi] | -| Autostart | `no` | [default / user specified] | +| VM Name | `` | from user input (validated) | +| Namespace | `` | from user selection/input or current context | +| Operating System | `` | from user selection/input | +| Performance Profile | `` | from user selection (default: u1) | +| Size | `` | from user selection (default: medium) | +| Storage | `` | from user selection (default: 30Gi) | +| Storage Class | `` | from user selection (default: cluster default) | +| Autostart | `` | from user selection (default: no) | **This will create a new VirtualMachine resource consuming cluster resources.** @@ -265,30 +624,70 @@ Extract `status.printableStatus` from the response. **Diagnostic Workflow (when ErrorUnschedulable detected)**: -#### 3a. Consult Troubleshooting Documentation +#### 3a. Consult Troubleshooting Documentation (REQUIRED) -**Document Consultation** (REQUIRED): -1. **Action**: Read [troubleshooting.md](../../docs/troubleshooting.md) using the Read tool to understand ErrorUnschedulable causes -2. **Output to user**: "I detected the VM is ErrorUnschedulable. I consulted [troubleshooting.md](../../docs/troubleshooting.md) to diagnose the issue." +**CRITICAL**: Document consultation MUST happen BEFORE diagnostic commands. -#### 3b. Gather Diagnostic Information +**When ErrorUnschedulable is detected**, consult documentation first to understand root causes. -**Execute diagnostic commands** using MCP tools or bash: +**Document Consultation** (REQUIRED - Execute FIRST): +1. **Action**: Read [scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) using the Read tool to understand ErrorUnschedulable root causes (node taints, resource constraints, node selectors) and workaround patterns +2. **Output to user**: "I detected the VM is ErrorUnschedulable. I consulted [scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) to understand the diagnosis and remediation strategies." -```bash -# Get VM events to see scheduling failures -oc describe vm -n | grep -A 10 "Events:" +**Expected Knowledge Gained from Documentation**: +- ErrorUnschedulable is caused by: node taints, insufficient resources, or node selector mismatches +- Most common cause: Node taints that VM doesn't tolerate +- Workaround pattern: Diagnose → Propose → Confirm → Execute using `oc patch` +- MCP tool limitation: `vm_create` doesn't support tolerations parameter -# Check node taints -oc get nodes -o json | jq '.items[] | select(.spec.taints != null) | {name: .metadata.name, taints: .spec.taints}' -``` +#### 3b. Analyze Error and Gather Diagnostic Information + +**Based on the troubleshooting guide**, proceed with diagnostics to identify the specific root cause. + +**Output to user**: "I'm now gathering diagnostic information to identify the specific issue." + +#### 3c. Gather Diagnostic Information + +**Execute diagnostic commands using MCP tools:** + +1. **Get VM events to see scheduling failures:** + + **Use MCP Tool**: `events_list` (from openshift-virtualization) + + **Parameters**: + - `namespace`: "" (the VM's namespace) + + Filter events related to the VirtualMachine or VirtualMachineInstance to identify scheduling issues. + +2. **Get VM details to see conditions:** + + **Use MCP Tool**: `resources_get` (from openshift-virtualization) + + **Parameters**: + - `apiVersion`: "kubevirt.io/v1" + - `kind`: "VirtualMachine" + - `name`: "" + - `namespace`: "" + + Check `.status.conditions` array for error messages and details. + +3. **Check node taints (if taint issue suspected):** + + **Use MCP Tool**: `resources_list` (from openshift-virtualization) + + **Parameters**: + - `apiVersion`: "v1" + - `kind`: "Node" + + Parse results to extract `.spec.taints` from each node to identify taints that might block scheduling. **Parse results** to identify root cause: - Events contain "taints that the pod didn't tolerate" → **Taints/Tolerations issue** - Events contain "Insufficient cpu" or "Insufficient memory" → **Resource constraints** - Events contain "no nodes available" → **No suitable nodes** +- Conditions array shows "Unschedulable" reason → Check reason field for details -#### 3c. Present Diagnosis to User +#### 3d. Present Diagnosis to User **Report findings in clear format**: @@ -346,7 +745,7 @@ spec: ⚠️ **This is a temporary limitation**: The openshift-virtualization MCP server doesn't yet support tolerations parameter in vm_create. Consider filing an issue at: https://github.com/openshift/openshift-mcp-server/issues ``` -#### 3d. Wait for User Decision (Human-in-the-Loop) +#### 3e. Wait for User Decision (Human-in-the-Loop) **CRITICAL**: Do NOT execute the workaround without explicit user confirmation. @@ -361,11 +760,20 @@ How would you like to proceed? Please respond with your choice. ``` -#### 3e. Execute Workaround (ONLY if user confirms) +#### 3f. Execute Workaround (ONLY if user confirms) **ONLY if user responds with "apply workaround" or similar confirmation:** -**Execute patch command**: +**Option 1: Try MCP Tool First** (Preferred): + +**Use MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +Fetch the current VM using `resources_get`, add the tolerations to `.spec.template.spec.tolerations`, then update using `resources_create_or_update`. + +**Option 2: Use CLI Patch** (If MCP approach doesn't work): + +**Note**: Strategic merge patch via CLI is acceptable when MCP tools cannot perform the same operation. + ```bash oc patch vm -n --type=merge -p ' spec: @@ -379,15 +787,58 @@ spec: ' ``` -**Verify patch success**: -```bash -# Check if tolerations were added -oc get vm -n -o jsonpath='{.spec.template.spec.tolerations}' +**Verify patch success using MCP Tools**: -# Wait 5 seconds and check status again -sleep 5 -oc get vm -n -o jsonpath='{.status.printableStatus}' -``` +1. **Check if tolerations were added:** + + **Use MCP Tool**: `resources_get` (from openshift-virtualization) + + **Parameters**: + - `apiVersion`: "kubevirt.io/v1" + - `kind`: "VirtualMachine" + - `name`: "" + - `namespace`: "" + + Inspect `.spec.template.spec.tolerations` in the response. + +2. **Wait 5-10 seconds and check status again:** + + **Use MCP Tool**: `resources_get` (from openshift-virtualization) - same parameters as above + + Check `.status.printableStatus` for the new status. + +3. **CRITICAL: Restart the VM to apply the toleration:** + + Simply patching the VM spec does NOT immediately update the existing VirtualMachineInstance. You must restart the VM to force the old VMI to be deleted and a new one created with the updated tolerations. + + **Use MCP Tool**: `vm_lifecycle` (from openshift-virtualization) + + **Parameters**: + - `namespace`: "" + - `name`: "" + - `action`: "restart" + + This will: + - Delete the old VirtualMachineInstance (created without tolerations) + - Create a new VirtualMachineInstance with the updated spec (including tolerations) + - Force the scheduler to re-evaluate the VM placement + +4. **Wait for VM to reach Running status:** + + After restart, wait 15-20 seconds for the VM to: + - Schedule on the virtualization node (with new tolerations) + - Provision the DataVolume (if first boot) + - Reach Running status + + **Use MCP Tool**: `resources_get` (from openshift-virtualization) + + **Parameters**: + - `apiVersion`: "kubevirt.io/v1" + - `kind`: "VirtualMachine" + - `name`: "" + - `namespace`: "" + + Check `.status.printableStatus` should transition: Stopped → Provisioning → Running **Report result**: ```markdown @@ -395,17 +846,25 @@ oc get vm -n -o jsonpath='{.status.printableStatus}' **VM Name**: `` **Namespace**: `` -**Action**: Added tolerations for taint `` - -**New Status**: (VM can now be scheduled) - -**Next Steps**: -To start the VM: -``` -"Start VM in namespace " -``` +**Action**: Added tolerations for taint `` and restarted VM + +**Status Progression**: +- ✓ Toleration patch applied +- ✓ VM restarted to force rescheduling +- ✓ New VirtualMachineInstance created with tolerations +- ✓ VM successfully scheduled on virtualization node +- ✓ Current Status: (should be Running or Provisioning) + +**What happened:** +1. Added toleration to VM spec +2. Restarted VM to delete old VirtualMachineInstance +3. New VirtualMachineInstance created with updated spec (generation increased) +4. Scheduler successfully placed VM on virtualization node +5. VM is now running (or provisioning) **Note**: This workaround was needed because the MCP tool doesn't yet support tolerations. Future VMs in this cluster will need the same fix until the tool is enhanced. + +**Next Steps**: Your VM is now running and ready to use! See the "Accessing the VM" section above for connection options. ``` ### Step 4: Report Creation Status @@ -421,10 +880,22 @@ To start the VM: - **Name**: `web-server` - **Namespace**: `vms` - **Operating System**: Fedora +- **Performance**: General Purpose (u1) - **Size**: medium -- **Storage**: 50Gi +- **Storage**: 50Gi (StorageClass: ) - **Status**: Halted (VM is created but not running) +**What happens next:** +- DataVolume provisioning: ~2-5 minutes (depends on image size and StorageClass) +- VM will be in "Provisioning" status during image import +- Once complete, status changes to "Stopped" (if autostart=no) or "Running" (if autostart=yes) + +**To monitor progress:** +``` +"Show status of VM web-server" +``` +Watch for status transition: Provisioning → Stopped → Running (when started) + **Next Steps:** To start the VM: @@ -442,6 +913,64 @@ To view full VM details: "Get details of web-server VM" ``` +--- + +## 🔐 Accessing the VM + +**After the VM is running**, you can access it using: + +### Option 1: Serial Console (virtctl) +```bash +virtctl console web-server -n vms +``` +- Direct terminal access to VM +- Requires virtctl CLI tool installed + +### Option 2: VNC Console (OpenShift Web Console) +1. Navigate to: OpenShift Console → Virtualization → VirtualMachines +2. Select namespace: `vms` +3. Click on VM: `web-server` +4. Click "Console" tab +5. Use VNC or Serial console + +### Option 3: SSH Access (if configured) +```bash +# Requires cloud-init SSH key configuration +# Get VM IP using MCP tool: resources_get with apiVersion="kubevirt.io/v1", kind="VirtualMachineInstance", name="web-server", namespace="vms" +# Extract .status.interfaces[0].ipAddress from the response + +# Then SSH: +ssh @ +``` + +### Option 4: Port Forwarding +```bash +# Forward a port from VM to localhost +virtctl port-forward vmi/web-server -n vms 8080:80 +# Access via: http://localhost:8080 +``` + +--- + +## 👤 Default Credentials + +Would you like me to display the default credentials for the VM? (yes/no) + +**If yes**, I'll show the OS-specific default credentials: +- **Fedora**: User: `fedora`, Password: (none - requires SSH key or console access to set) +- **Ubuntu**: User: `ubuntu`, Password: (none - requires SSH key or console access to set) +- **RHEL**: User: `cloud-user`, Password: (none - requires SSH key) +- **CentOS**: User: `centos`, Password: (none - requires SSH key) +- **Debian**: User: `debian`, Password: (none - requires SSH key) +- **OpenSUSE**: User: `opensuse`, Password: (none - requires SSH key) + +**Note**: Most cloud images require SSH key authentication. To set a password: +1. Access via console: `virtctl console -n ` +2. Login (may auto-login on first boot) +3. Set password: `sudo passwd ` + +--- + **Note**: The VM is created in a halted state. Use the `/vm-lifecycle-manager` skill to start it. ``` @@ -460,10 +989,10 @@ To view full VM details: - **OpenShift Virtualization not installed** - Operator must be installed on cluster **Troubleshooting:** -1. Verify namespace exists: `oc get namespace ` -2. Check permissions: `oc auth can-i create virtualmachines -n ` -3. View cluster capacity: `oc describe nodes` -4. Verify operator installed: `oc get csv -n openshift-cnv` +1. Verify namespace exists using MCP tool: `namespaces_list` +2. Check permissions using CLI (no MCP equivalent): `oc auth can-i create virtualmachines -n ` +3. View cluster capacity using MCP tool: `nodes_top` or `resources_list` for Node resources +4. Verify operator installed using MCP tool: `resources_list` with apiVersion="operators.coreos.com/v1alpha1", kind="ClusterServiceVersion", namespace="openshift-cnv" Would you like help troubleshooting this error? ``` @@ -550,8 +1079,8 @@ vm_create({ **Error**: "Namespace 'xyz' not found" **Solution:** -1. List available namespaces: Suggest using `oc get namespaces` or `kubectl get ns` -2. Create namespace if needed: `oc create namespace ` +1. List available namespaces using MCP tool: `namespaces_list` +2. Create namespace if needed using MCP tool: `resources_create_or_update` with Namespace resource, OR use CLI: `oc create namespace ` 3. Verify ServiceAccount has access to the namespace ### Issue 2: Insufficient Permissions @@ -569,7 +1098,7 @@ vm_create({ **Error**: "Insufficient resources to schedule VM" **Solution:** -1. Check cluster capacity: CPU, memory available +1. Check cluster capacity using MCP tool `nodes_top` or `resources_list` for Node resources 2. Try smaller `size` (e.g., change "large" to "medium") 3. Reduce `storage` size if possible 4. Contact cluster admin to scale nodes @@ -580,7 +1109,7 @@ vm_create({ **Solution:** - OpenShift Virtualization operator must be installed -- Verify: `oc get csv -n openshift-cnv` +- Verify using MCP tool: `resources_list` with apiVersion="operators.coreos.com/v1alpha1", kind="ClusterServiceVersion", namespace="openshift-cnv" - Contact cluster admin to install operator ## Dependencies @@ -596,10 +1125,16 @@ vm_create({ - `vm-inventory` - List and view created VMs ### Reference Documentation -- [troubleshooting.md](../../docs/troubleshooting.md) - VM error diagnosis and workarounds +- [scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) - ErrorUnschedulable root causes and workarounds (consulted during diagnostic workflows) +- [storage-errors.md](../../docs/troubleshooting/storage-errors.md) - ErrorDataVolumeNotReady and storage provisioning issues +- [runtime-errors.md](../../docs/troubleshooting/runtime-errors.md) - CrashLoopBackOff and guest OS failures +- [network-errors.md](../../docs/troubleshooting/network-errors.md) - Network attachment failures +- [Troubleshooting INDEX](../../docs/troubleshooting/INDEX.md) - Navigation hub for discovering additional error categories when encountering unexpected issues outside the categories above - [OpenShift Virtualization Documentation](https://docs.openshift.com/container-platform/latest/virt/about_virt/about-virt.html) +- [Troubleshooting VMs](https://docs.openshift.com/container-platform/latest/virt/virtual_machines/troubleshooting_vms.html) - [KubeVirt VirtualMachine API](https://kubevirt.io/api-reference/) - [OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server) +- [StorageClass Documentation](https://docs.openshift.com/container-platform/latest/storage/understanding-persistent-storage.html) ## Security Considerations @@ -612,12 +1147,29 @@ vm_create({ ## Example Usage -**Example 1: Simple VM creation with defaults** +**Example 1: Complete interactive menu workflow (no user input)** ``` -User: "Create a Fedora VM called web-01 in namespace apps" +User: "Create a VM" Agent: [Invokes /vm-creator skill] + [Detects current namespace from kubeconfig: "production"] + [Queries available StorageClasses, finds default: "ocs-storagecluster-ceph-rbd"] + [No parameters provided, uses AskUserQuestion for full interactive menu] + +[Interactive Menu Presented - User selects:] +- VM Name: my-app-server +- Namespace: production (Current) +- Operating System: Fedora (Recommended) +- Performance Profile: General Purpose (u1) (Recommended) +- Size: Medium (Recommended) +- Storage: 30Gi (Recommended) +- Storage Class: ocs-storagecluster-ceph-rbd (Default) - Immediate provisioning, Supports live migration (RWX) +- Autostart: No (Recommended) + +Agent: [Validates VM name: "my-app-server" ✓] + [Checks if VM exists in namespace "production": Not found ✓] + [Processes responses] ## Virtual Machine Configuration @@ -625,12 +1177,14 @@ Please review and confirm the VM configuration: | Parameter | Value | Notes | |-----------|-------|-------| -| VM Name | `web-01` | from user input | -| Namespace | `apps` | from user input | -| Operating System | `fedora` | default | -| Size | (default) | MCP server will select | -| Storage | `30Gi` | default | -| Autostart | `no` | default | +| VM Name | `my-app-server` | from interactive menu (validated) | +| Namespace | `production` | from interactive menu (current context) | +| Operating System | `fedora` | from interactive menu | +| Performance Profile | `u1` | from interactive menu (default) | +| Size | `medium` | from interactive menu (default) | +| Storage | `30Gi` | from interactive menu (default) | +| Storage Class | `ocs-storagecluster-ceph-rbd` | from interactive menu (default: cluster default) | +| Autostart | `no` | from interactive menu (default) | This will create a new VirtualMachine resource consuming cluster resources. @@ -638,40 +1192,115 @@ Confirm these settings or tell me what to change. User: "yes" -Agent: [Calls vm_create with namespace="apps", name="web-01", workload="fedora"] +Agent: [Calls vm_create with namespace="production", name="my-app-server", workload="fedora", performance="u1", size="medium", storage="30Gi", autostart=false] ## ✓ Virtual Machine Created Successfully +VM Details: +- Name: `my-app-server` +- Namespace: `production` +- OS: Fedora +- Performance: General Purpose (u1) +- Size: medium +- Storage: 30Gi (ocs-storagecluster-ceph-rbd) +- Status: Provisioning → Stopped (wait 2-5 minutes) + +[Displays full access instructions and credential prompt] + +Next Steps: "Start VM my-app-server in namespace production" +``` + +**Example 2: Partial interactive menu (some parameters specified by user)** + +``` +User: "Create an Ubuntu VM called web-01 in namespace apps" + +Agent: [Invokes /vm-creator skill] + [Detects VM name="web-01", namespace="apps", OS="ubuntu"] + [Queries StorageClasses for menu] + [Uses AskUserQuestion only for: Performance, Size, Storage, Storage Class, Autostart] + +[Interactive Menu Presented - User selects:] +- Performance Profile: General Purpose (u1) (Recommended) +- Size: Large +- Storage: 100Gi +- Storage Class: local-storage - Delayed provisioning, No live migration (RWO) +- Autostart: Yes + +Agent: [Validates VM name: "web-01" ✓] + [Checks if VM exists in namespace "apps": Not found ✓] + [Processes responses] + +## Virtual Machine Configuration + +Please review and confirm the VM configuration: + +| Parameter | Value | Notes | +|-----------|-------|-------| +| VM Name | `web-01` | from user input (validated) | +| Namespace | `apps` | from user input | +| Operating System | `ubuntu` | from user input | +| Performance Profile | `u1` | from interactive menu (default) | +| Size | `large` | from interactive menu | +| Storage | `100Gi` | from interactive menu | +| Storage Class | `local-storage` | from interactive menu | +| Autostart | `yes` | from interactive menu | + +This will create a new VirtualMachine resource consuming cluster resources. + +Confirm these settings or tell me what to change. + +User: "yes" + +Agent: [Calls vm_create with namespace="apps", name="web-01", workload="ubuntu", performance="u1", size="large", storage="100Gi", autostart=true] + +## ✓ Virtual Machine Created and Starting + VM Details: - Name: `web-01` - Namespace: `apps` -- OS: Fedora -- Status: Halted +- OS: Ubuntu +- Performance: General Purpose (u1) +- Size: large +- Storage: 100Gi (local-storage) +- Status: Provisioning → Running (autostarted) -Next Steps: "Start VM web-01 in namespace apps" +[Displays full access instructions] ``` -**Example 2: Custom configuration with modifications** +**Example 3: User modifies configuration before confirming** ``` -User: "Deploy a large RHEL VM with 100Gi storage called db-server in production namespace" +User: "Deploy a memory-optimized RHEL VM called db-server in production namespace" Agent: [Invokes /vm-creator skill] + [Detects: name="db-server", namespace="production", OS="rhel", performance="m1" (memory-optimized)] + [Uses AskUserQuestion for: Size, Storage, Storage Class, Autostart] + +[Interactive Menu - User selects:] +- Size: Large +- Storage: 100Gi +- Storage Class: ceph-rbd (Default) +- Autostart: No (Recommended) + +Agent: [Validates and checks existence ✓] ## Virtual Machine Configuration | Parameter | Value | Notes | |-----------|-------|-------| -| VM Name | `db-server` | from user input | +| VM Name | `db-server` | from user input (validated) | | Namespace | `production` | from user input | | Operating System | `rhel` | from user input | -| Size | `large` | from user input | -| Storage | `100Gi` | from user input | -| Autostart | `no` | default | +| Performance Profile | `m1` | from user input (memory-optimized) | +| Size | `large` | from interactive menu | +| Storage | `100Gi` | from interactive menu | +| Storage Class | `ceph-rbd` | from interactive menu (default) | +| Autostart | `no` | from interactive menu (default) | Confirm these settings or tell me what to change. -User: "Actually, autostart it" +User: "Actually, autostart it and make it compute-optimized instead" Agent: [Updates configuration] @@ -679,40 +1308,89 @@ Agent: [Updates configuration] | Parameter | Value | Notes | |-----------|-------|-------| +| Performance Profile | `c1` | updated per user request (compute-optimized) | | Autostart | `yes` | updated per user request | Confirm? User: "yes" -Agent: [Calls vm_create with all parameters including autostart=true] +Agent: [Calls vm_create with namespace="production", name="db-server", workload="rhel", performance="c1", size="large", storage="100Gi", autostart=true] -## ✓ Virtual Machine Created and Started +## ✓ Virtual Machine Created and Starting VM Details: - Name: `db-server` - Namespace: `production` - OS: RHEL +- Performance: Compute Optimized (c1) - Size: large -- Storage: 100Gi -- Status: Running (autostarted) +- Storage: 100Gi (ceph-rbd) +- Status: Provisioning → Running (autostarted) + +[Displays access instructions] +``` + +**Example 4: VM name already exists - user prompted to choose different name** + +``` +User: "Create a VM named apache-server in namespace vms" + +Agent: [Invokes /vm-creator skill] + [Validates name: "apache-server" ✓] + [Checks if VM exists...] + +⚠️ **VM Already Exists** + +A VirtualMachine named `apache-server` already exists in namespace `vms`. + +**Current VM status**: Stopped + +**Options:** +1. Choose a different VM name +2. Delete existing VM first (use vm-inventory and manual deletion) +3. Cancel operation + +What would you like to do? + +User: "Choose a different name: apache-server-02" + +Agent: [Validates new name: "apache-server-02" ✓] + [Checks existence: Not found ✓] + [Continues with interactive menu for remaining parameters...] + +## Virtual Machine Configuration + +| Parameter | Value | Notes | +|-----------|-------|-------| +| VM Name | `apache-server-02` | from user input (updated, validated) | +| Namespace | `vms` | from user input | +... + +[Continues normally] ``` -**Example 3: User cancels after reviewing configuration** +**Example 5: User cancels after reviewing configuration** ``` User: "Create a VM named test in namespace dev" Agent: [Invokes /vm-creator skill] + [Interactive menu presented...] + [Processes responses] ## Virtual Machine Configuration | Parameter | Value | Notes | |-----------|-------|-------| -| VM Name | `test` | from user input | +| VM Name | `test` | from user input (validated) | | Namespace | `dev` | from user input | -| Operating System | `fedora` | default | -| Storage | `30Gi` | default | +| Operating System | `fedora` | from interactive menu (default) | +| Performance Profile | `u1` | from interactive menu (default) | +| Size | `medium` | from interactive menu (default) | +| Storage | `30Gi` | from interactive menu (default) | +| Storage Class | `default-sc` | from interactive menu (default) | +| Autostart | `no` | from interactive menu (default) | Confirm these settings or tell me what to change. diff --git a/rh-virt/skills/vm-delete/SKILL.md b/rh-virt/skills/vm-delete/SKILL.md new file mode 100644 index 00000000..f1d7068e --- /dev/null +++ b/rh-virt/skills/vm-delete/SKILL.md @@ -0,0 +1,1275 @@ +--- +name: vm-delete +description: | + Permanently delete virtual machines and their associated resources from OpenShift Virtualization. Use this skill when users request: + - "Delete VM [name]" + - "Remove virtual machine [name]" + - "Destroy VM [name]" + - "Clean up VM [name]" + + This skill handles permanent VM deletion with strict safety confirmations and typed verification. + + NOT for power management (use vm-lifecycle-manager to stop VMs). + +model: inherit +color: red +--- + +# /vm-delete Skill + +Permanently delete virtual machines and their associated resources (storage, DataVolumes) from OpenShift Virtualization clusters. This skill enforces strict safety protocols including typed confirmation and pre-deletion validation. + +## Critical: Human-in-the-Loop Requirements + +**CRITICAL: This skill performs IRREVERSIBLE operations.** You MUST: + +1. **Pre-Deletion Validation** (Execute FIRST, before asking for confirmation) + - Verify VM exists using `resources_get` + - Check VM running state + - Discover dependent resources (DataVolumes, PVCs) + - Check for protection labels (`protected: "true"`) + +2. **Safety Checks** + - **REFUSE deletion** if VM has label `protected: "true"` + - **REQUIRE VM to be stopped first** if currently running + - **List all resources** that will be deleted (VM, storage, DataVolumes) + +3. **Typed Confirmation Protocol** (MANDATORY) + - Display complete deletion scope + - **Require user to type the exact VM name** to confirm + - Accept only exact match (case-sensitive) + - If name doesn't match → Cancel operation, do not proceed + +4. **Deletion Options** + - Ask user what to delete: + - Option 1: VM only (preserve storage) + - Option 2: VM + storage (complete cleanup) + - Option 3: Cancel + - **NEVER assume** which option user wants + +5. **Final Confirmation Before Each Deletion** + - After typed confirmation, show exactly what will be deleted + - Ask: "Proceed with permanent deletion? (yes/cancel)" + - Wait for explicit "yes" + +6. **Never Auto-Execute** + - **NEVER delete without explicit typed confirmation** + - **NEVER proceed if user says "no", "wait", "cancel"** + - **NEVER skip the typed verification step** + +**Why This Matters:** +- **Permanent**: Deleted VMs cannot be recovered +- **Data Loss**: Storage deletion destroys all VM data +- **Service Impact**: Deleting running VMs causes immediate service outage +- **Accidental Deletion**: Typed verification prevents mistakes (typos, wrong VM name) + +## Prerequisites + +**Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) + +**Required MCP Tools**: +- `resources_get` (from openshift-virtualization) - Verify VM exists and get details +- `resources_delete` (from openshift-virtualization) - Delete Kubernetes resources +- `resources_list` (from openshift-virtualization) - List dependent resources (PVCs, DataVolumes) +- `vm_lifecycle` (from openshift-virtualization) - Stop running VMs before deletion + +**Required Environment Variables**: +- `KUBECONFIG` - Path to Kubernetes configuration file with cluster access + +**Required Cluster Setup**: +- OpenShift cluster (>= 4.19) +- OpenShift Virtualization operator installed +- ServiceAccount with RBAC permissions to delete VirtualMachine and PVC resources +- Target VM must exist + +### Prerequisite Verification + +**Before executing, verify MCP server availability:** + +1. **Check MCP Server Configuration** + - Verify `openshift-virtualization` exists in `.mcp.json` + - If missing → Report to user with setup instructions + +2. **Check Environment Variables** + - Verify `KUBECONFIG` is set (check presence only, never expose value) + - If missing → Report to user + +3. **Check RBAC Permissions** (optional verification) + - Verify ServiceAccount can delete VirtualMachine resources + - Verify ServiceAccount can delete PVC/DataVolume resources + +**Human Notification Protocol:** + +When prerequisites fail: + +``` +❌ Cannot execute vm-delete: MCP server 'openshift-virtualization' is not available + +📋 Setup Instructions: +1. Add openshift-virtualization to .mcp.json: + { + "mcpServers": { + "openshift-virtualization": { + "command": "podman", + "args": [ + "run", + "--rm", + "-i", + "--network=host", + "--userns=keep-id:uid=65532,gid=65532", + "-v", "${KUBECONFIG}:/kubeconfig:ro,Z", + "--entrypoint", "/app/kubernetes-mcp-server", + "quay.io/ecosystem-appeng/openshift-mcp-server:latest", + "--kubeconfig", "/kubeconfig", + "--toolsets", "core,kubevirt" + ], + "env": { + "KUBECONFIG": "${KUBECONFIG}" + } + } + } + } + +2. Set KUBECONFIG environment variable: + export KUBECONFIG="/path/to/your/kubeconfig" + +3. Restart Claude Code to reload MCP servers + +🔗 Documentation: https://github.com/openshift/openshift-mcp-server + +❓ How would you like to proceed? +Options: +- "setup" - Help configure the MCP server now +- "skip" - Skip this skill +- "abort" - Stop workflow + +Please respond with your choice. +``` + +⚠️ **SECURITY**: Never display actual KUBECONFIG path or credential values in output. + +## When to Use This Skill + +**Trigger this skill when:** +- User explicitly invokes `/vm-delete` command +- User requests permanent VM deletion +- User wants to clean up test/development VMs +- User needs to free cluster resources by removing VMs +- User wants to decommission VMs + +**User phrases that trigger this skill:** +- "Delete VM test-vm in namespace dev" +- "Remove the virtual machine web-server" +- "Destroy VM old-database" +- "Clean up all VMs in namespace test" +- "/vm-delete" (explicit command) + +**Do NOT use this skill when:** +- User wants to stop a VM temporarily → Use `/vm-lifecycle-manager` skill instead +- User wants to create a VM → Use `/vm-creator` skill instead +- User wants to view VMs → Use `/vm-inventory` skill instead +- User wants to pause or suspend VM → Use lifecycle management (not deletion) + +## Workflow + +### Step 1: Gather VM Information and Validate + +**CRITICAL**: This step MUST complete BEFORE asking for user confirmation. + +**Required Information from User:** +1. **VM Name** - Name of the virtual machine to delete +2. **Namespace** - OpenShift namespace where VM exists + +If user doesn't provide namespace, ask for it explicitly. + +**Pre-Deletion Validation (Execute ALL checks):** + +**1.1: Verify VM Exists** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", // User-provided + "name": "" // User-provided +} +``` + +**Expected Output**: VirtualMachine resource with metadata, spec, and status + +**Error Handling**: +- If VM not found → Report error to user, suggest checking VM name/namespace with vm-inventory skill +- If permission denied → Report RBAC error + +**1.2: Check Protection Label** + +**From the VM resource returned in step 1.1**, check `metadata.labels` for `protected: "true"`. + +**If protected label exists:** +```markdown +❌ Cannot Delete Protected VM + +**VM**: `` (namespace: ``) + +**Protection Status**: This VM has the `protected: "true"` label. + +**Reason**: Protected VMs cannot be deleted to prevent accidental removal of critical infrastructure. + +**To delete this VM:** +1. Remove the protection label first: + ``` + oc label vm -n protected- + ``` +2. Then retry deletion + +**Alternative**: Use vm-inventory skill to verify this is the correct VM. + +Operation cancelled. +``` + +**STOP workflow** - Do not proceed with deletion. + +**1.3: Check VM Running State** + +**From the VM resource returned in step 1.1**, check `status.printableStatus` or `status.ready`. + +**Running States**: "Running", "Starting", "Migrating" +**Stopped States**: "Stopped", "Halted", "Terminated" + +**If VM is running:** +```markdown +⚠️ VM is Currently Running + +**VM**: `` (namespace: ``) +**Status**: Running + +**Safety Requirement**: VMs must be stopped before deletion to ensure graceful shutdown. + +**Options:** +1. "stop-and-delete" - Stop the VM first, then proceed with deletion +2. "cancel" - Cancel deletion operation + +How would you like to proceed? +``` + +**Wait for user response.** + +- If "stop-and-delete" → Continue to Step 1.4 +- If "cancel" → Stop workflow, cancel operation + +**1.4: Stop Running VM (if applicable)** + +**ONLY execute if user chose "stop-and-delete" in step 1.3.** + +**MCP Tool**: `vm_lifecycle` (from openshift-virtualization) + +**Parameters**: +```json +{ + "namespace": "", + "name": "", + "action": "stop" +} +``` + +**Expected Output**: VM runStrategy changed to "Halted", VM stopping + +**Report to user**: +```markdown +⏸️ Stopping VM before deletion... + +VM `` is being stopped gracefully. + +Wait 10-30 seconds for VM to fully stop, then deletion will proceed. +``` + +**Wait 10 seconds**, then verify VM is stopped by checking status again. + +**1.5: Discover Dependent Resources** + +**Find DataVolumes and PVCs associated with this VM.** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters for DataVolumes**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "labelSelector": "vm.kubevirt.io/name=" +} +``` + +**Parameters for PVCs** (if DataVolumes not found): +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "labelSelector": "vm.kubevirt.io/name=" +} +``` + +**Expected Output**: List of DataVolumes or PVCs owned by this VM + +**Parse results**: +- Extract resource names +- Calculate total storage size (sum of all PVC sizes) +- Store list for deletion scope presentation + +**If no storage found**: VM uses ephemeral storage or container disks (no persistent storage to delete) + +### Step 2: Present Deletion Scope and Get Deletion Options + +**After completing ALL validation in Step 1**, present the complete deletion scope to user. + +**Display deletion scope:** + +```markdown +## ⚠️ VM Deletion - Review Scope + +**Virtual Machine**: `` +**Namespace**: `` +**Current Status**: (stopped in step 1.4 if was running) + +### Resources Found + +**VirtualMachine Resource:** +- VM: `` +- Age: +- vCPU: , Memory: + +**Persistent Storage:** (if found) +- DataVolume: `` (30Gi) +- PVC: `` (30Gi) +- **Total Storage**: 30Gi + +**OR** (if no storage found) + +**Persistent Storage:** +- None (VM uses ephemeral/container disk storage) + +--- + +### Deletion Options + +**What should be deleted?** + +**Option 1: VM Only** (preserve storage) +- Deletes: VirtualMachine resource +- Preserves: DataVolumes and PVCs for potential reuse +- Use case: Recreating VM with same storage later + +**Option 2: VM + Storage** (complete cleanup) ← Recommended for test/dev VMs +- Deletes: VirtualMachine resource +- Deletes: All DataVolumes and PVCs +- Frees: 30Gi cluster storage +- Use case: Permanently removing VM and reclaiming resources + +**Option 3: Cancel** +- No resources deleted +- VM remains in cluster + +--- + +**Select deletion option** (1, 2, or 3): +``` + +**Wait for user to select option 1, 2, or 3.** + +**Handle user response:** +- If "3" or "cancel" → Cancel operation, stop workflow +- If "1" → Proceed to Step 3 with delete_storage=false +- If "2" → Proceed to Step 3 with delete_storage=true + +### Step 3: Typed Confirmation (MANDATORY) + +**CRITICAL**: User MUST type the exact VM name to proceed. + +**Present typed confirmation prompt:** + +```markdown +## 🔴 PERMANENT DELETION - Typed Confirmation Required + +**This action CANNOT be undone.** + +### What will be deleted: + +✗ VirtualMachine: `` (namespace: ``) + +✗ DataVolume: `` (30Gi) +✗ PVC: `` (30Gi) +✗ **All VM data will be lost permanently** + + + +✓ Storage PRESERVED: DataVolumes and PVCs will remain for reuse + + +--- + +**⚠️ IMPORTANT**: This deletion is permanent. Deleted resources cannot be recovered. + +**To confirm deletion, type the VM name exactly as shown:** + +Type `` to confirm: _____ +``` + +**Wait for user to type the VM name.** + +**Validation:** +- Compare user input with VM name (case-sensitive, exact match) +- **If match**: Proceed to Step 4 +- **If mismatch**: Cancel operation + +**On mismatch:** +```markdown +❌ Confirmation Failed + +**You typed**: `` +**Expected**: `` + +**Names do not match.** Deletion cancelled for safety. + +To retry, use: +``` +"Delete VM in namespace " +``` + +Operation cancelled. No resources were deleted. +``` + +**STOP workflow** - Do not proceed with deletion. + +### Step 4: Final Confirmation Before Execution + +**After typed verification succeeds**, ask for final explicit confirmation. + +```markdown +## ✓ Typed Verification Passed + +**Confirmation received for VM**: `` + +### Ready to Delete: + + +- VirtualMachine: `` +- DataVolume: `` (30Gi) +- PVC: `` (30Gi) + +**Impact**: VM and all associated data will be permanently deleted. 30Gi storage will be freed. + + + +- VirtualMachine: `` (storage preserved) + +**Impact**: VM will be deleted. DataVolumes and PVCs will remain for potential reuse. + + +--- + +**Proceed with permanent deletion?** +- Type "yes" to execute deletion +- Type "cancel" to abort + +Your choice: _____ +``` + +**Wait for user response.** + +**Handle response:** +- If "yes" → Proceed to Step 5 (execute deletion) +- If "cancel", "no", "wait", or anything else → Cancel operation + +**On cancellation:** +```markdown +Operation cancelled by user. No resources were deleted. + +VM `` remains in namespace ``. +``` + +**STOP workflow**. + +### Step 5: Execute Deletion + +**ONLY PROCEED AFTER**: +- ✓ Step 1: Validation complete +- ✓ Step 2: User selected deletion option +- ✓ Step 3: User typed VM name correctly +- ✓ Step 4: User confirmed "yes" + +**5.1: Delete VirtualMachine Resource** + +**MCP Tool**: `resources_delete` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +**Expected Output**: VirtualMachine deleted successfully + +**Error Handling**: +- If deletion fails → Report error, do not proceed with storage deletion +- If permission denied → Report RBAC error +- If VM not found → May have been deleted externally, continue anyway + +**Report progress:** +```markdown +🗑️ Deleting VirtualMachine resource... +✓ VirtualMachine `` deleted +``` + +**5.2: Delete Storage (if delete_storage=true)** + +**ONLY execute if user selected Option 2 in Step 2.** + +**For each DataVolume found in Step 1.5:** + +**MCP Tool**: `resources_delete` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "cdi.kubevirt.io/v1beta1", + "kind": "DataVolume", + "namespace": "", + "name": "" +} +``` + +**For each PVC found in Step 1.5** (if DataVolumes not used): + +**MCP Tool**: `resources_delete` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "name": "" +} +``` + +**Expected Output**: Storage resources deleted successfully + +**Error Handling**: +- If deletion fails → Report which resources failed, continue with others +- If permission denied → Report RBAC error for specific resource + +**Report progress for each deletion:** +```markdown +🗑️ Deleting storage resources... +✓ DataVolume `` deleted (30Gi freed) +✓ PVC `` deleted +``` + +### Step 6: Report Deletion Results + +**On successful deletion:** + +**If delete_storage=true (complete cleanup):** +```markdown +## ✓ VM Deleted Successfully (Complete Cleanup) + +**Deleted Resources:** +- ✓ VirtualMachine: `` (namespace: ``) +- ✓ DataVolume: `` +- ✓ PVC: `` + +**Storage Freed**: 30Gi + +**Impact:** +- VM and all associated data permanently removed +- Cluster resources freed +- VM cannot be recovered + +**To verify deletion:** +``` +"List VMs in namespace " +``` + +The VM should no longer appear in the inventory. +``` + +**If delete_storage=false (storage preserved):** +```markdown +## ✓ VM Deleted Successfully (Storage Preserved) + +**Deleted Resources:** +- ✓ VirtualMachine: `` (namespace: ``) + +**Preserved Resources:** +- ✓ DataVolume: `` (30Gi) - Available for reuse +- ✓ PVC: `` (30Gi) - Available for reuse + +**Impact:** +- VM removed from cluster +- Storage remains available for attaching to new VMs +- Data preserved on PVCs + +**To reuse storage:** +Create a new VM and reference the existing DataVolume or PVC. + +**To delete storage later:** +``` +oc delete datavolume -n +oc delete pvc -n +``` +``` + +**On partial failure (VM deleted but storage deletion failed):** + +**OPTIONAL**: If storage deletion fails, consult documentation for storage-related deletion issues. + +**Document Consultation** (OPTIONAL - when storage deletion fails): +1. **Action**: Read [storage-errors.md](../../docs/troubleshooting/storage-errors.md) using the Read tool to understand storage deletion issues and PVC cleanup strategies +2. **Output to user**: "I consulted [storage-errors.md](../../docs/troubleshooting/storage-errors.md) to understand the storage deletion failure." + +```markdown +## ⚠️ Partial Deletion Completed + +**Successfully Deleted:** +- ✓ VirtualMachine: `` + +**Failed to Delete:** +- ✗ DataVolume: `` - Error: +- ✗ PVC: `` - Error: + +**Storage remains in cluster.** You may need to delete these resources manually: + +``` +oc delete datavolume -n +oc delete pvc -n +``` + +**Possible causes:** +- Insufficient RBAC permissions to delete PVCs +- PVC in use by another resource +- Storage class retention policy + +Would you like help troubleshooting the storage deletion failure? +``` + +**On complete failure (VM deletion failed):** + +**OPTIONAL**: If deletion operation fails, consult documentation for common deletion failure scenarios. + +**Document Consultation** (OPTIONAL - when deletion fails): +1. **Action**: Read [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) using the Read tool to understand VM deletion failure scenarios, finalizer issues, and stuck Terminating states +2. **Output to user**: "I consulted [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) to understand potential causes for the deletion failure." + +**When to consult**: +- VM deletion fails with finalizer errors +- VM stuck in Terminating state +- Storage deletion fails (PVC/DataVolume errors) +- Unexpected error messages from resources_delete tool + +**When NOT to consult**: +- Simple "VM not found" errors (clear cause) +- RBAC permission errors (clear cause) +- Namespace doesn't exist (clear cause) + +```markdown +## ❌ VM Deletion Failed + +**Error**: + +**VM**: `` (namespace: ``) + +**Common Causes:** +- **Insufficient RBAC permissions** - ServiceAccount lacks delete permission for VirtualMachines +- **VM not found** - May have been deleted externally +- **Finalizers blocking deletion** - VM has finalizers preventing deletion +- **Resource in use** - VM or associated resources locked by another process + +**Troubleshooting Steps:** + +1. **Check permissions:** + ``` + oc auth can-i delete virtualmachines -n + ``` + +2. **Verify VM exists:** + ``` + "List VMs in namespace " + ``` + +3. **Check for finalizers:** + + Consult [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) "VM Stuck in Terminating State" section for MCP-first diagnostic steps using `resources_get` to check finalizers. + +4. **Manual deletion (if needed):** + + Use `resources_delete` MCP tool or consult [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) for proper deletion procedure. + +Would you like help troubleshooting this error? +``` + +## Advanced Features + +### Dry-Run Mode (Preview Deletion) + +Allow user to preview what would be deleted without actually deleting. + +**User request:** "Show me what would be deleted if I delete VM test-vm" + +**Workflow:** +1. Execute Step 1 (validation and discovery) +2. Execute Step 2 (present deletion scope) +3. **Skip Steps 3-5** (no confirmation, no deletion) +4. Report: "This is a preview. No resources were deleted." + +**Output:** +```markdown +## 🔍 Deletion Preview (Dry-Run) + +**If you delete VM** `test-vm` **(namespace:** `dev`**):** + +**Would be deleted (Option 1 - VM only):** +- VirtualMachine: `test-vm` + +**Would be deleted (Option 2 - VM + storage):** +- VirtualMachine: `test-vm` +- DataVolume: `test-vm-disk` (30Gi) +- PVC: `test-vm-disk` (30Gi) +- **Storage freed**: 30Gi + +**Would be preserved:** +- None (complete cleanup) + +--- + +**This is a preview only. No resources were deleted.** + +To execute deletion: +``` +"Delete VM test-vm in namespace dev" +``` +``` + +### Batch Deletion (Multiple VMs) + +**User request:** "Delete VMs test-01, test-02, test-03 in namespace dev" + +**Workflow:** +1. Execute Step 1 (validation) **for each VM** +2. Present combined deletion scope for all VMs +3. Require typed confirmation: "type 'DELETE-3-VMS' to confirm" +4. Execute deletion for each VM sequentially + +**Typed confirmation for batch:** +```markdown +## 🔴 BATCH DELETION - Typed Confirmation Required + +**Deleting 3 VMs in namespace** `dev`**:** + +1. ✗ VirtualMachine: `test-01` + storage (30Gi) +2. ✗ VirtualMachine: `test-02` + storage (30Gi) +3. ✗ VirtualMachine: `test-03` + storage (30Gi) + +**Total Impact:** +- 3 VMs permanently deleted +- 90Gi storage freed +- All VM data lost + +**To confirm batch deletion, type:** `DELETE-3-VMS` + +Type confirmation: _____ +``` + +### Protected VM Label Check + +**Automatic protection enforcement:** + +If VM has label `protected: "true"`, **refuse deletion** in Step 1.2. + +**Example VM with protection:** +```yaml +metadata: + name: production-database + labels: + protected: "true" + env: production +``` + +**Skill response:** +```markdown +❌ Cannot Delete Protected VM + +**VM**: `production-database` has protection enabled. + +**To delete:** +1. Remove protection: `oc label vm production-database protected- -n ` +2. Retry deletion + +This safeguard prevents accidental deletion of critical infrastructure. +``` + +## Common Issues + +### Issue 1: VM Not Found + +**Error**: "VirtualMachine 'xyz' not found in namespace 'abc'" + +**Solution:** +1. Verify VM name spelling +2. Check namespace is correct +3. List VMs in namespace: Use `/vm-inventory` skill +4. VM may have already been deleted + +### Issue 2: Permission Denied + +**Error**: "Forbidden: User cannot delete VirtualMachines in namespace 'xyz'" + +**Solution:** +- Verify KUBECONFIG has appropriate RBAC permissions +- Required permissions: delete VirtualMachine, delete PVC, delete DataVolume +- Contact cluster admin for permission grant +- Check ServiceAccount role bindings: + ``` + oc auth can-i delete virtualmachines -n + oc auth can-i delete persistentvolumeclaims -n + ``` + +### Issue 3: VM Has Finalizers + +**Error**: "VM deletion blocked by finalizers" + +**Solution:** +- VMs with finalizers require finalizer removal before deletion +- Consult [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) "VM Stuck in Terminating State" section for MCP-first approach: + - Use `resources_get` to check finalizers + - Use `resources_create_or_update` to remove finalizers (if needed) +- Common finalizers: `kubevirt.io/virtualMachineControllerFinalize` +- Wait for controllers to remove finalizers, or manually patch VM (advanced) + +### Issue 4: Storage Deletion Fails + +**Error**: "PVC deletion failed: resource in use" + +**Solution:** +1. Verify VM was deleted first (VMs must be deleted before storage) +2. Consult [storage-errors.md](../../docs/troubleshooting/storage-errors.md) "Storage Deletion Failures" section for MCP-first diagnostics: + - Use `pods_list_in_namespace` to check if PVC is mounted by other resources + - Use `resources_get` to check PVC status +3. Wait for VM pod termination (can take 30-60 seconds) +4. Use `resources_delete` to delete PVC, or consult storage-errors.md for proper procedure + +### Issue 5: User Typed Wrong VM Name + +**Error**: User typed "test-vm-01" instead of "test-vm" + +**Solution:** +- **Skill automatically cancels** operation (Step 3 validation) +- No resources deleted +- User must retry with correct typed confirmation + +**Example:** +```markdown +❌ Confirmation Failed + +You typed: `test-vm-01` +Expected: `test-vm` + +Names do not match. Deletion cancelled for safety. +``` + +## Dependencies + +### Required MCP Servers +- `openshift-virtualization` - OpenShift MCP server with core and kubevirt toolsets + +### Required MCP Tools +- `resources_get` (from openshift-virtualization) - Get VirtualMachine details and verify existence + - Parameters: apiVersion, kind, namespace, name + - Source: https://github.com/openshift/openshift-mcp-server/blob/main/pkg/toolsets/core/resources.go + +- `resources_delete` (from openshift-virtualization) - Delete Kubernetes resources + - Parameters: apiVersion, kind, namespace, name + - Source: https://github.com/openshift/openshift-mcp-server/blob/main/pkg/toolsets/core/resources.go + +- `resources_list` (from openshift-virtualization) - List dependent resources (DataVolumes, PVCs) + - Parameters: apiVersion, kind, namespace, labelSelector + - Source: https://github.com/openshift/openshift-mcp-server/blob/main/pkg/toolsets/core/resources.go + +- `vm_lifecycle` (from openshift-virtualization) - Stop running VMs before deletion + - Parameters: namespace, name, action + - Source: https://github.com/openshift/openshift-mcp-server/blob/main/pkg/toolsets/kubevirt/vm.go + +### Related Skills +- `vm-inventory` - Verify VM exists and get details before deletion +- `vm-lifecycle-manager` - Stop running VMs before deletion (used internally by vm-delete) +- `vm-creator` - Create VMs after cleanup operations + +### Reference Documentation +- [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) - VM deletion failure scenarios, finalizer issues, and stuck Terminating states (optionally consulted when deletion operations fail) +- [storage-errors.md](../../docs/troubleshooting/storage-errors.md) - Storage deletion strategies and PVC cleanup procedures (optionally consulted when storage deletion fails) +- [Troubleshooting INDEX](../../docs/troubleshooting/INDEX.md) - Navigation hub for discovering additional error categories when encountering unexpected issues outside the categories above +- [OpenShift Virtualization Documentation](https://docs.openshift.com/container-platform/latest/virt/about_virt/about-virt.html) +- [KubeVirt VirtualMachine API](https://kubevirt.io/api-reference/) +- [Kubernetes Finalizers](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/) +- [PVC Deletion](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#delete) +- [OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server) + +## Security Considerations + +- **RBAC Enforcement**: Deletion requires explicit RBAC permissions for VirtualMachine, PVC, and DataVolume resources +- **Typed Verification**: Prevents accidental deletion through typos or wrong VM names +- **Protection Labels**: `protected: "true"` label blocks deletion entirely +- **Audit Trail**: All deletions logged in Kubernetes API audit logs +- **Graceful Shutdown**: Running VMs stopped before deletion to ensure clean shutdown +- **KUBECONFIG Security**: Credentials never exposed in output +- **Namespace Isolation**: Can only delete VMs in namespaces where ServiceAccount has permissions +- **No Force Delete**: Skill does not use `--force` or `--grace-period=0` flags + +## Example Usage + +### Example 1: Delete VM with Storage (Complete Cleanup) + +``` +User: "Delete VM test-vm in namespace dev" + +Agent: [Invokes /vm-delete skill] + [Step 1: Validation] + [Calls resources_get to verify VM exists] + [Checks protection label - none found] + [Checks VM status - Stopped] + [Calls resources_list to find DataVolumes/PVCs] + [Found: test-vm-disk DataVolume, 30Gi] + +## ⚠️ VM Deletion - Review Scope + +Virtual Machine: `test-vm` +Namespace: `dev` +Current Status: Stopped + +### Resources Found + +VirtualMachine Resource: +- VM: `test-vm` +- Age: 5 days +- vCPU: 2, Memory: 4Gi + +Persistent Storage: +- DataVolume: `test-vm-disk` (30Gi) +- PVC: `test-vm-disk` (30Gi) +- Total Storage: 30Gi + +--- + +### Deletion Options + +What should be deleted? + +Option 1: VM Only (preserve storage) +- Deletes: VirtualMachine resource +- Preserves: DataVolumes and PVCs for potential reuse + +Option 2: VM + Storage (complete cleanup) ← Recommended for test/dev VMs +- Deletes: VirtualMachine resource +- Deletes: All DataVolumes and PVCs +- Frees: 30Gi cluster storage + +Option 3: Cancel +- No resources deleted + +Select deletion option (1, 2, or 3): + +User: "2" + +Agent: [Step 2: User selected Option 2 - delete VM + storage] + [Step 3: Typed confirmation] + +## 🔴 PERMANENT DELETION - Typed Confirmation Required + +This action CANNOT be undone. + +### What will be deleted: + +✗ VirtualMachine: `test-vm` (namespace: `dev`) +✗ DataVolume: `test-vm-disk` (30Gi) +✗ PVC: `test-vm-disk` (30Gi) +✗ All VM data will be lost permanently + +--- + +⚠️ IMPORTANT: This deletion is permanent. Deleted resources cannot be recovered. + +To confirm deletion, type the VM name exactly as shown: + +Type `test-vm` to confirm: + +User: "test-vm" + +Agent: [Step 3: Typed verification passed] + [Step 4: Final confirmation] + +## ✓ Typed Verification Passed + +Confirmation received for VM: `test-vm` + +### Ready to Delete: + +- VirtualMachine: `test-vm` +- DataVolume: `test-vm-disk` (30Gi) +- PVC: `test-vm-disk` (30Gi) + +Impact: VM and all associated data will be permanently deleted. 30Gi storage will be freed. + +--- + +Proceed with permanent deletion? +- Type "yes" to execute deletion +- Type "cancel" to abort + +Your choice: + +User: "yes" + +Agent: [Step 5: Execute deletion] + [Calls resources_delete for VirtualMachine] + +🗑️ Deleting VirtualMachine resource... +✓ VirtualMachine `test-vm` deleted + + [Calls resources_delete for DataVolume] + +🗑️ Deleting storage resources... +✓ DataVolume `test-vm-disk` deleted (30Gi freed) +✓ PVC `test-vm-disk` deleted + + [Step 6: Report results] + +## ✓ VM Deleted Successfully (Complete Cleanup) + +Deleted Resources: +- ✓ VirtualMachine: `test-vm` (namespace: `dev`) +- ✓ DataVolume: `test-vm-disk` +- ✓ PVC: `test-vm-disk` + +Storage Freed: 30Gi + +Impact: +- VM and all associated data permanently removed +- Cluster resources freed +- VM cannot be recovered + +To verify deletion: +``` +"List VMs in namespace dev" +``` + +The VM should no longer appear in the inventory. +``` + +### Example 2: Delete VM Only (Preserve Storage) + +``` +User: "Delete VM web-server but keep the storage" + +Agent: "Which namespace is the VM in?" + +User: "production" + +Agent: [Invokes /vm-delete skill] + [Step 1: Validation] + [VM exists, stopped, has storage] + [Step 2: Present deletion scope] + +## ⚠️ VM Deletion - Review Scope + +Virtual Machine: `web-server` +Namespace: `production` + +[... deletion scope details ...] + +Select deletion option (1, 2, or 3): + +User: "1" + +Agent: [Step 3: Typed confirmation] + +## 🔴 PERMANENT DELETION - Typed Confirmation Required + +### What will be deleted: + +✗ VirtualMachine: `web-server` (namespace: `production`) + +✓ Storage PRESERVED: DataVolumes and PVCs will remain for reuse + +Type `web-server` to confirm: + +User: "web-server" + +Agent: [Typed verification passed] + +## ✓ Typed Verification Passed + +Ready to Delete: +- VirtualMachine: `web-server` (storage preserved) + +Impact: VM will be deleted. DataVolumes and PVCs will remain for potential reuse. + +Proceed with permanent deletion? (yes/cancel): + +User: "yes" + +Agent: [Deletes VM only] + +## ✓ VM Deleted Successfully (Storage Preserved) + +Deleted Resources: +- ✓ VirtualMachine: `web-server` (namespace: `production`) + +Preserved Resources: +- ✓ DataVolume: `web-server-disk` (50Gi) - Available for reuse +- ✓ PVC: `web-server-disk` (50Gi) - Available for reuse + +Impact: +- VM removed from cluster +- Storage remains available for attaching to new VMs +- Data preserved on PVCs +``` + +### Example 3: Protected VM (Deletion Refused) + +``` +User: "Delete VM production-database in namespace production" + +Agent: [Invokes /vm-delete skill] + [Step 1: Validation] + [Calls resources_get] + [Checks protection label - found "protected: true"] + +❌ Cannot Delete Protected VM + +VM: `production-database` (namespace: `production`) + +Protection Status: This VM has the `protected: "true"` label. + +Reason: Protected VMs cannot be deleted to prevent accidental removal of critical infrastructure. + +To delete this VM: +1. Remove the protection label first: + ``` + oc label vm production-database -n production protected- + ``` +2. Then retry deletion + +Alternative: Use vm-inventory skill to verify this is the correct VM. + +Operation cancelled. +``` + +### Example 4: Stop Running VM Before Deletion + +``` +User: "Delete VM api-server in namespace production" + +Agent: [Invokes /vm-delete skill] + [Step 1: Validation] + [VM exists, RUNNING] + [Step 1.3: VM is running] + +⚠️ VM is Currently Running + +VM: `api-server` (namespace: `production`) +Status: Running + +Safety Requirement: VMs must be stopped before deletion to ensure graceful shutdown. + +Options: +1. "stop-and-delete" - Stop the VM first, then proceed with deletion +2. "cancel" - Cancel deletion operation + +How would you like to proceed? + +User: "stop-and-delete" + +Agent: [Step 1.4: Stop VM] + [Calls vm_lifecycle with action="stop"] + +⏸️ Stopping VM before deletion... + +VM `api-server` is being stopped gracefully. + +Wait 10-30 seconds for VM to fully stop, then deletion will proceed. + + [Waits 10 seconds] + [Verifies VM stopped] + [Continues to Step 2: Present deletion scope] + [...rest of workflow...] +``` + +### Example 5: User Cancels at Typed Confirmation + +``` +User: "Delete VM test-vm in namespace dev" + +Agent: [Validation complete] + [User selects Option 2 - VM + storage] + [Step 3: Typed confirmation] + +Type `test-vm` to confirm: + +User: "wait, cancel this" + +Agent: ❌ Confirmation Failed + +You typed: `wait, cancel this` +Expected: `test-vm` + +Names do not match. Deletion cancelled for safety. + +To retry, use: +``` +"Delete VM test-vm in namespace dev" +``` + +Operation cancelled. No resources were deleted. +``` + +### Example 6: Dry-Run Preview + +``` +User: "What would happen if I delete VM test-vm in dev?" + +Agent: [Invokes /vm-delete skill in dry-run mode] + [Step 1: Validation and discovery only] + +## 🔍 Deletion Preview (Dry-Run) + +If you delete VM `test-vm` (namespace: `dev`): + +Would be deleted (Option 1 - VM only): +- VirtualMachine: `test-vm` + +Would be deleted (Option 2 - VM + storage): +- VirtualMachine: `test-vm` +- DataVolume: `test-vm-disk` (30Gi) +- PVC: `test-vm-disk` (30Gi) +- Storage freed: 30Gi + +Would be preserved: +- None (complete cleanup) + +--- + +This is a preview only. No resources were deleted. + +To execute deletion: +``` +"Delete VM test-vm in namespace dev" +``` +``` diff --git a/rh-virt/skills/vm-inventory/SKILL.md b/rh-virt/skills/vm-inventory/SKILL.md index 7efff7e2..195bc3aa 100644 --- a/rh-virt/skills/vm-inventory/SKILL.md +++ b/rh-virt/skills/vm-inventory/SKILL.md @@ -17,6 +17,23 @@ color: cyan List and inspect virtual machines in OpenShift Virtualization clusters. This skill provides read-only access to VM information without making any modifications. +## Critical: Human-in-the-Loop Requirements + +**Not applicable** - This skill performs read-only operations and does not modify any cluster resources. No user confirmation is required. + +**Read-only operations:** +- Listing VirtualMachines across namespaces or in specific namespaces +- Retrieving VM details, status, and resource configurations +- Displaying VM health conditions and resource usage +- Filtering VMs by labels or field selectors +- Viewing VM network, storage, and node placement information + +**No modifications performed:** +- ✓ Does not change VM state (start/stop/restart) +- ✓ Does not modify VM configuration +- ✓ Does not delete VMs or resources +- ✓ Does not consume cluster resources + ## Prerequisites **Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) @@ -201,27 +218,52 @@ oc get vm -A -o json **Step 2: Format and Display Results** -Present VMs organized by namespace with key information: +**CRITICAL FORMATTING RULE**: +- **If total VMs > 2**: Display results in a **table format** ordered by namespace and status +- **If total VMs ≤ 2**: Use list format organized by namespace + +**Table Format (when VMs > 2):** + +```markdown +## 📋 Virtual Machines (All Namespaces) + +| Namespace | VM Name | Status | Age | Resources | Node | +|-----------|---------|--------|-----|-----------|------| +| default | alejandro-test | ✓ Running | 1h | 2 vCPU, 4Gi | ip-10-0-15-252 | +| production | database-vm | ✗ Stopped | 30d | 8 vCPU, 16Gi | - | +| production | web-server-01 | ✓ Running | 15d | 4 vCPU, 8Gi | worker-01 | +| production | web-server-02 | ✓ Running | 15d | 4 vCPU, 8Gi | worker-02 | +| development | debug-vm | ⚠ Pending | 2d | 2 vCPU, 4Gi | - | +| development | test-vm | ✓ Running | 5d | 2 vCPU, 4Gi | worker-03 | + +**Summary:** +- **Total VMs**: 6 +- **Running**: 4 +- **Stopped**: 1 +- **Pending**: 1 +``` + +**List Format (when VMs ≤ 2):** ```markdown ## 📋 Virtual Machines (All Namespaces) ### Namespace: production - ✓ **web-server-01** - Running (4 vCPU, 8Gi RAM) -- ✓ **web-server-02** - Running (4 vCPU, 8Gi RAM) -- ✗ **database-vm** - Stopped (8 vCPU, 16Gi RAM) ### Namespace: development - ✓ **test-vm** - Running (2 vCPU, 4Gi RAM) -- ⚠ **debug-vm** - Pending (2 vCPU, 4Gi RAM) ### Summary: -- **Total VMs**: 5 -- **Running**: 3 -- **Stopped**: 1 -- **Pending**: 1 +- **Total VMs**: 2 +- **Running**: 2 ``` +**Table Ordering Rules:** +1. **Primary sort**: Namespace (alphabetical) +2. **Secondary sort**: Status (Running → Pending → Stopped → Failed/Error) +3. **Tertiary sort**: VM Name (alphabetical within same namespace and status) + **Status Indicators:** - ✓ Running/Ready - ✗ Stopped/Halted @@ -344,7 +386,24 @@ oc get virtualmachine -n -o yaml oc get vm -n -o yaml ``` -**Step 3: Display Detailed Information** +**Step 3: Interpret Status and Conditions (Optional)** + +**OPTIONAL**: If the VM has error status or complex conditions, consult documentation for interpretation. + +**Document Consultation** (OPTIONAL - when VM has error/warning status): +1. **Action**: Read [troubleshooting/INDEX.md](../../docs/troubleshooting/INDEX.md) using the Read tool to understand status meanings (ErrorUnschedulable, ErrorDataVolumeNotReady, CrashLoopBackOff, etc.) +2. **Output to user**: "I consulted [troubleshooting/INDEX.md](../../docs/troubleshooting/INDEX.md) to interpret the VM status ''." + +**When to consult**: +- VM status is ErrorUnschedulable, ErrorDataVolumeNotReady, ErrorPvcNotFound +- VM conditions show warnings or errors +- VM in unexpected state (e.g., CrashLoopBackOff, Terminating stuck) + +**When NOT to consult**: +- VM status is normal (Running, Stopped, Provisioning) +- Simple status queries without error conditions + +**Step 4: Display Detailed Information** ```markdown ## 🖥️ Virtual Machine Details @@ -688,6 +747,7 @@ No VMs were found in this namespace. - `vm-troubleshooter` (planned) - Diagnose problematic VMs from inventory ### Reference Documentation +- [Troubleshooting INDEX](../../docs/troubleshooting/INDEX.md) - VM status interpretation and navigation hub for discovering error-specific troubleshooting guides (optionally consulted when displaying VM details with error states) - [OpenShift Virtualization Documentation](https://docs.openshift.com/container-platform/latest/virt/about_virt/about-virt.html) - [KubeVirt VirtualMachine API](https://kubevirt.io/api-reference/) - [Accessing VMs](https://docs.openshift.com/container-platform/latest/virt/virtual_machines/virt-accessing-vm-consoles.html) @@ -704,7 +764,7 @@ No VMs were found in this namespace. ## Example Usage -**Example 1: List all VMs (using MCP tool)** +**Example 1: List all VMs (using MCP tool - Table format for > 2 VMs)** ``` User: "List all VMs" @@ -712,18 +772,21 @@ User: "List all VMs" Agent: [Invokes /vm-inventory skill] [Attempts MCP tool: resources_list with apiVersion="kubevirt.io/v1", kind="VirtualMachine", allNamespaces=true] [Tool succeeds] + [4 VMs found - using table format] ## 📋 Virtual Machines (All Namespaces) -Namespace: production -- ✓ web-server-01 - Running (4 vCPU, 8Gi) -- ✓ web-server-02 - Running (4 vCPU, 8Gi) -- ✗ database-vm - Stopped (8 vCPU, 16Gi) - -Namespace: development -- ✓ test-vm - Running (2 vCPU, 4Gi) +| Namespace | VM Name | Status | Age | Resources | Node | +|-----------|---------|--------|-----|-----------|------| +| development | test-vm | ✓ Running | 5d | 2 vCPU, 4Gi | worker-03 | +| production | database-vm | ✗ Stopped | 30d | 8 vCPU, 16Gi | - | +| production | web-server-01 | ✓ Running | 15d | 4 vCPU, 8Gi | worker-01 | +| production | web-server-02 | ✓ Running | 15d | 4 vCPU, 8Gi | worker-02 | -Summary: 4 VMs (3 running, 1 stopped) +**Summary:** +- **Total VMs**: 4 +- **Running**: 3 +- **Stopped**: 1 ``` **Example 1b: List all VMs (MCP tool unavailable, CLI fallback)** @@ -745,10 +808,34 @@ Would you like me to proceed with the CLI command? User: "yes" Agent: [Executes: oc get virtualmachines -A -o json] + [4 VMs found - using table format] + +## 📋 Virtual Machines (All Namespaces) + +[Same table output as Example 1 above] +``` + +**Example 1c: List all VMs (List format for ≤ 2 VMs)** + +``` +User: "List all VMs" + +Agent: [Invokes /vm-inventory skill] + [Attempts MCP tool: resources_list with apiVersion="kubevirt.io/v1", kind="VirtualMachine", allNamespaces=true] + [Tool succeeds] + [2 VMs found - using list format] ## 📋 Virtual Machines (All Namespaces) -[Same output as above] +### Namespace: default +- ✓ **web-server** - Running (4 vCPU, 8Gi RAM, worker-01) + +### Namespace: production +- ✓ **database-vm** - Running (8 vCPU, 16Gi RAM, worker-02) + +**Summary:** +- **Total VMs**: 2 +- **Running**: 2 ``` **Example 2: Get details of specific VM** diff --git a/rh-virt/skills/vm-lifecycle-manager/SKILL.md b/rh-virt/skills/vm-lifecycle-manager/SKILL.md index 89cce611..daa60e05 100644 --- a/rh-virt/skills/vm-lifecycle-manager/SKILL.md +++ b/rh-virt/skills/vm-lifecycle-manager/SKILL.md @@ -315,6 +315,24 @@ To start the VM again: **On Failure:** +**OPTIONAL**: If lifecycle operation fails, consult documentation for common failure scenarios. + +**Document Consultation** (OPTIONAL - when lifecycle operation fails): +1. **Action**: Read relevant troubleshooting guides to understand VM lifecycle failure scenarios: + - [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) - For start/stop failures, stuck transitions + - [scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) - For ErrorUnschedulable, resource constraints +2. **Output to user**: "I consulted [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) and [scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) to understand potential causes for the lifecycle operation failure." + +**When to consult**: +- VM fails to start (may be ErrorUnschedulable, resource constraints) +- VM fails to stop (may be stuck in transition, finalizers) +- Unexpected error messages from vm_lifecycle tool + +**When NOT to consult**: +- Simple "VM already in desired state" messages +- VM not found errors (clear cause) +- RBAC permission errors (clear cause) + ```markdown ## ❌ Failed to Execute Lifecycle Operation @@ -463,6 +481,9 @@ User: "Start web-server if it's not running" - `vm-troubleshooter` (planned) - Diagnose VM startup/shutdown issues ### Reference Documentation +- [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) - VM start/stop failures and stuck transitions (optionally consulted when lifecycle operations fail) +- [scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) - ErrorUnschedulable and resource constraint errors (optionally consulted when VM won't start) +- [Troubleshooting INDEX](../../docs/troubleshooting/INDEX.md) - Navigation hub for discovering additional error categories when encountering unexpected issues outside the categories above - [OpenShift Virtualization Documentation](https://docs.openshift.com/container-platform/latest/virt/about_virt/about-virt.html) - [KubeVirt VirtualMachine Lifecycle](https://kubevirt.io/user-guide/virtual_machines/lifecycle/) - [RunStrategy Documentation](https://kubevirt.io/user-guide/virtual_machines/run_strategies/)