diff --git a/CLAUDE.md b/CLAUDE.md index 518b3b4d..60f654d0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -644,6 +644,9 @@ last_updated: YYYY-MM-DD 4. Include concrete examples 5. Test with `Skill` tool invocation +**Collection-Specific Standards:** +- **rh-virt**: Follow `rh-virt/SKILL_TEMPLATE.md` for enhanced quality standards including mandatory Common Issues and Example Usage sections + ### Adding an Agent 1. Create `agents/.md` @@ -686,6 +689,19 @@ The `rh-sre` pack is the most complete implementation, demonstrating: When creating new collection, use `rh-sre` as the architectural reference. +### rh-virt (Quality-Controlled Pattern) + +The `rh-virt` pack demonstrates skill quality standardization: +- Comprehensive skill templates (`SKILL_TEMPLATE.md`) +- Risk-based color coding (cyan/green/blue/yellow/red) +- Mandatory Common Issues and Example Usage sections +- Consistent section ordering and formatting + +Use `rh-virt` as reference for collections requiring high consistency and maintainability. + +When creating new collections, follow the pattern that best matches your needs: +- **Tool-focused domains** (infrastructure, operations) → Follow rh-virt pattern + ## Key Principles ### Core Architecture diff --git a/rh-virt/README.md b/rh-virt/README.md index 5b377762..c8869697 100644 --- a/rh-virt/README.md +++ b/rh-virt/README.md @@ -98,7 +98,7 @@ claude plugin install openshift-virtualization The pack provides 5 specialized skills for complete VM lifecycle management: -### 1. **vm-creator** - Virtual Machine Provisioning +### 1. **vm-create** - Virtual Machine Provisioning Create new virtual machines in OpenShift Virtualization with automatic error diagnosis and workarounds. @@ -291,7 +291,7 @@ The server provides two toolsets enabled via `--toolsets core,kubevirt`: ``` User: "Create a VM called web-server in namespace production" -→ vm-creator skill creates the VM +→ vm-create skill creates the VM User: "Start the web-server VM" → vm-lifecycle-manager skill starts the VM @@ -383,7 +383,7 @@ Agent: "⏳ Storage cloning in progress... (45%) ``` User: "Create a Fedora VM called test-vm in namespace demo" -→ vm-creator skill creates the VM +→ vm-create skill creates the VM → Detects ErrorUnschedulable status → Consults docs/troubleshooting/scheduling-errors.md for domain knowledge → Diagnoses: Node taints prevent scheduling @@ -396,7 +396,7 @@ Agent: "⚠️ VM Scheduling Issue Detected How would you like to proceed?" User: "apply workaround" -→ vm-creator patches VM with tolerations +→ vm-create patches VM with tolerations → Verifies VM can now be scheduled → Reports success @@ -460,7 +460,7 @@ MCP server is configured in `.mcp.json`: ### Automatic Diagnosis (Recommended) -The **vm-creator** skill includes automatic error diagnosis and workaround proposals. When VMs encounter scheduling issues: +The **vm-create** skill includes automatic error diagnosis and workaround proposals. When VMs encounter scheduling issues: 1. **Detection**: Skill automatically detects ErrorUnschedulable and other error states 2. **Diagnosis**: Consults `docs/troubleshooting/INDEX.md` and category files to understand root cause @@ -492,7 +492,7 @@ The **vm-creator** skill includes automatic error diagnosis and workaround propo 2. Check namespace exists and ServiceAccount has access 3. Verify RBAC permissions for VirtualMachine resources 4. Check cluster resource availability (CPU, memory, storage) -5. Let vm-creator skill run automatic diagnosis (see Workflow 4 above) +5. Let vm-create skill run automatic diagnosis (see Workflow 4 above) ### Skills Not Triggering @@ -525,7 +525,7 @@ rh-virt/ │ └── .ai-index/ # Semantic indexing for AI discovery │ └── semantic-index.json └── skills/ - ├── vm-creator/SKILL.md # VM provisioning with auto-diagnosis + ├── vm-create/SKILL.md # VM provisioning with auto-diagnosis ├── vm-lifecycle-manager/SKILL.md # VM power management ├── vm-inventory/SKILL.md # VM discovery and status ├── vm-delete/SKILL.md # VM destruction and cleanup diff --git a/rh-virt/SKILL_TEMPLATE.md b/rh-virt/SKILL_TEMPLATE.md new file mode 100644 index 00000000..c8ecee53 --- /dev/null +++ b/rh-virt/SKILL_TEMPLATE.md @@ -0,0 +1,887 @@ +# Skill Template for rh-virt Collection + +This template provides the standardized structure for all skills in the `rh-virt` agentic collection. Use this when creating new skills to ensure consistency, maintainability, and compliance with Claude Code requirements. + +## Overview + +This template implements: +- **Repository Standards**: rh-virt collection-specific patterns and conventions +- **Claude Guidelines**: Official skill structure from `/CLAUDE.md` (Design Principles #1-7) +- **MCP Integration**: OpenShift Virtualization MCP server tool usage patterns +- **Human-in-the-Loop**: Safety confirmations for critical operations + +**Reference**: See `/CLAUDE.md` "Design Principles for Skills and Agents" for complete rationale. + +## Quick Start + +1. Copy this template to `skills//SKILL.md` +2. Replace all `` with actual content +3. Follow the validation checklist at the end of this template +4. Verify compliance with `SKILLS_CHECKLIST.md` +5. Test the skill before committing + +## Claude Code Requirements Summary + +Before using this template, understand these mandatory requirements: + +### 1. Document Consultation Transparency (Design Principle #1) +- Skills MUST actually read documentation using Read tool before invoking MCP tools +- Skills MUST declare consultation to user: "I consulted [file] to understand [topic]" +- **REQUIRED** for rh-virt when relevant troubleshooting docs exist (docs/troubleshooting/) + +### 2. Precise Parameter Specification (Design Principle #2) +- Provide exact parameter names and formats with examples +- Ensures first-attempt success when invoking MCP tools + +### 3. Skill Precedence and Conciseness (Design Principle #3) +- Description field in YAML frontmatter MUST be under 500 tokens +- Focus on "when to use" with 3-5 concrete examples + +### 4. Dependencies Declaration (Design Principle #4) +- List all MCP servers, tools, related skills, and documentation +- Follows specific format (see Dependencies section below) + +### 5. Human-in-the-Loop Requirements (Design Principle #5) +- Skills performing critical operations MUST require explicit user confirmation +- Positioned AFTER Dependencies section (not before) +- Read-only skills use "Not Applicable" pattern + +### 6. Mandatory Skill Sections (Design Principle #6) +- All sections must appear in correct order (see Template Structure below) + +### 7. MCP Server Availability Verification (Design Principle #7) +- Verify MCP server configuration before executing +- NEVER expose credential values in output + +--- + +## Template Structure + +### YAML Frontmatter + +**Purpose**: Loaded at agent initialization to help Claude decide which skill to invoke. + +**Requirements** (from CLAUDE.md Design Principle #3): +- Description field MUST be under 500 tokens total +- Focus on "when to use" with 3-5 concrete user phrases +- Include clear anti-patterns with alternatives +- Keep summary line under 100 characters for readability + +```yaml +--- +name: # REQUIRED: Lowercase with dashes, matches directory name +description: | # REQUIRED: Must be under 500 tokens total + # IMPORTANT: Keep entire description field under 500 tokens + # This includes all lines: summary, use cases, and anti-patterns + # Claude loads this at agent initialization - conciseness matters! + + (max 100 characters) + + Use when: + - "" # Actual phrases users would say + - "" # Not generic descriptions + - "" # At least 3 examples required + + (1 sentence) + + NOT for (use instead). + +model: inherit # REQUIRED: Always "inherit" unless special case + # Only use "sonnet" or "haiku" if skill needs specific model +color: # REQUIRED: red|yellow|blue|green|cyan - see Color Guide below + # Indicates operation risk level for user safety +--- + +# / Skill + + + +**Implementation Note** (OPTIONAL): + +## Prerequisites + +**Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) + +**Required MCP Tools**: +- `tool_name` (from openshift-virtualization) - Brief description of what it does +- `another_tool` (from openshift-virtualization) - Brief description + +**Required Environment Variables**: +- `KUBECONFIG` - Path to Kubernetes configuration file with cluster access +- `OTHER_VAR` (if applicable) - Description + +**Required Cluster Setup**: +- OpenShift cluster (>= 4.19) +- OpenShift Virtualization operator installed +- ServiceAccount with RBAC permissions to +- + +### Prerequisite Verification + +**Before executing, verify MCP server availability:** + +1. **Check MCP Server Configuration** + - Verify `openshift-virtualization` exists in `.mcp.json` + - If missing → Report to user with setup instructions + +2. **Check Environment Variables** + - Verify `KUBECONFIG` is set (check presence only, never expose value) + - If missing → Report to user + +3. **Check ** (optional verification) + - Verify + - If missing → Report to user + +**Human Notification Protocol:** + +When prerequisites fail: + +``` +❌ Cannot execute : MCP server 'openshift-virtualization' is not available + +📋 Setup Instructions: +1. Add openshift-virtualization to .mcp.json +2. Set KUBECONFIG environment variable: export KUBECONFIG="/path/to/kubeconfig" +3. Restart Claude Code to reload MCP servers + +🔗 Documentation: https://github.com/openshift/openshift-mcp-server + +❓ How would you like to proceed? +Options: +- "setup" - Help configure the MCP server now +- "skip" - Skip this skill +- "abort" - Stop workflow + +Please respond with your choice. +``` + +⚠️ **SECURITY**: Never display actual KUBECONFIG path or credential values in output. + +## When to Use This Skill + +**Trigger this skill when:** +- User +- User +- User +- User explicitly invokes `/` command + +**User phrases that trigger this skill:** +- "" +- "" +- "" +- `/` (explicit command) + +**Do NOT use this skill when:** +- User wants to → Use `` skill instead +- User wants to → Use `` skill instead +- User wants to → + +## Workflow + +**Purpose**: Step-by-step instructions for executing this skill. + +**Requirements** (from CLAUDE.md Design Principles #1-2): +- Document Consultation BEFORE tool invocation (REQUIRED when relevant troubleshooting docs exist) +- Precise parameter specifications with examples +- Clear error handling for each step + +### Step 1: + + + +**CRITICAL (Include if relevant troubleshooting documentation exists)**: Document consultation MUST happen BEFORE tool invocation. + +**Document Consultation** (REQUIRED when relevant troubleshooting docs exist): +1. **Action**: Read [doc.md](../../docs/troubleshooting/doc.md) using the Read tool to understand [specific topic] +2. **Output to user**: "I consulted [doc.md](../../docs/troubleshooting/doc.md) to understand [specific topic]." + +**When to consult troubleshooting docs**: +- VM creation/lifecycle skills → Read scheduling-errors.md, storage-errors.md, network-errors.md +- VM snapshot skills → Read storage-errors.md +- Any skill encountering errors → Read relevant troubleshooting doc before reporting to user + +**Available troubleshooting documentation**: +- docs/troubleshooting/INDEX.md - Master index of all troubleshooting docs +- docs/troubleshooting/scheduling-errors.md - VM scheduling failures (ErrorUnschedulable, taints, resources) +- docs/troubleshooting/storage-errors.md - Storage and PVC issues +- docs/troubleshooting/network-errors.md - Network attachment failures (Multus, NAD) +- docs/troubleshooting/lifecycle-errors.md - VM lifecycle state errors +- docs/troubleshooting/runtime-errors.md - Runtime and crash issues + +See CLAUDE.md Design Principle #1 for complete rationale. + +**MCP Tool**: `tool_name` or `category__tool_name` (from openshift-virtualization) + +**Parameters** (Design Principle #2 - Precise Parameter Specification): + +Provide exact parameter names, types, formats, and examples to ensure first-attempt success. + +```json +{ + "param1": "", // REQUIRED: Description of param1 + // Example: "database-01" (exact format shown) + "param2": "", // OPTIONAL: Description of param2 + // Example: "production" (namespace name) + "param3": true // REQUIRED: Description of what this boolean controls + // Example: true (enables feature X) +} +``` + +**Alternative Format** (bullet list - recommended by CLAUDE.md): +- `param1`: [exact specification] - REQUIRED + - Example: `"database-01"` (VM name to operate on) +- `param2`: [exact specification] - OPTIONAL + - Example: `"production"` (namespace where VM exists) +- `param3`: [exact specification] - REQUIRED + - Example: `true` (whether to enable feature X) + +**Example tool invocation:** +```json +tool_name({ + "param1": "database-01", + "param2": "production", + "param3": true +}) +``` + +**Expected Output**: + +**Error Handling**: +- If → +- If → +- If → + +**Extract/Store Information:** +- `field1` - +- `field2` - + +### Step 2: + + + +### Step N: Report Results + +**On success:** + +```markdown +## ✓ Successful + +****: `` (namespace: ``) + +###
+- **Detail 1**: +- **Detail 2**: +- **Detail 3**: + +### +- ✓ +- ✓ + +### Next Steps + +**To :** +"" + +**To :** +"" +``` + +**On failure:** + +```markdown +## ❌ Failed + +**Error**: + +****: `` (namespace: ``) + +**Common Causes:** +- **** - +- **** - +- **** - + +**Troubleshooting Steps:** + +1. **:** + + +2. **:** + + +3. **:** + + +Would you like help troubleshooting this error? +``` + +## Common Issues + +### Issue 1: + +**Error**: "" + +**Cause**: + +**Solution:** +1. +2. +3. + +**Related**: See [.md](../../docs/troubleshooting/.md) for more details + +### Issue 2: + +**Error**: "" + +**Cause**: + +**Solution:** +1. +2. + +### Issue 3: + +**Error**: "" + +**Cause**: + +**Solution:** +1. +2. + +**Minimum**: Include at least 3 common issues. Add more based on actual user pain points. + +## Dependencies + +**Purpose**: Declare all external dependencies for debugging and prerequisite verification. + +**Requirements** (from CLAUDE.md Design Principle #4): +- List MCP servers, tools, related skills, and documentation +- Makes dependencies explicit for troubleshooting +- Enables proper error handling when dependencies missing + +### Required MCP Servers +- `openshift-virtualization` - OpenShift MCP server with kubevirt toolset + - Source: https://github.com/openshift/openshift-mcp-server + +### Required MCP Tools + +List each tool with its purpose, parameters, and source. + +- `tool_name` (from openshift-virtualization) - Brief description of what it does + - **Used for**: + - **Parameters**: + - **Source**: https://github.com/openshift/openshift-mcp-server + +- `another_tool` (from openshift-virtualization) - Brief description + - **Used for**: + - **Parameters**: + - **Source**: https://github.com/openshift/openshift-mcp-server + +### Related Skills + +List skills that complement or replace this skill. + +- `skill-name-1` - When to use it instead of this skill (alternative) +- `skill-name-2` - Complementary skill (use together) +- `skill-name-3` - Follow-up skill (use after this one completes) + +### Reference Documentation + +**Internal Troubleshooting Documentation**: +- [INDEX.md](../../docs/troubleshooting/INDEX.md) - Master troubleshooting index +- [scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) - VM scheduling failures +- [storage-errors.md](../../docs/troubleshooting/storage-errors.md) - Storage and PVC issues +- [network-errors.md](../../docs/troubleshooting/network-errors.md) - Network attachment failures +- [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) - VM lifecycle errors +- [runtime-errors.md](../../docs/troubleshooting/runtime-errors.md) - Runtime and crash issues + +**Official Red Hat Documentation**: +- [Topic - OpenShift ](https://docs.redhat.com/en/documentation/openshift_container_platform//html-single/virtualization/index#section) - Main documentation +- [Blog Post Title](https://www.redhat.com/en/blog/post-slug) - Additional context + +**Upstream Documentation**: +- [KubeVirt Topic](https://kubevirt.io/user-guide/...) - Upstream project docs +- [Kubernetes Topic](https://kubernetes.io/docs/...) - K8s specification + +**IMPORTANT**: Always use the latest stable OpenShift version available for documentation links (check https://docs.redhat.com/ for current version). + +## Critical: Human-in-the-Loop Requirements + +**Purpose**: Define when and how to request user confirmation for critical operations. + +**Requirements** (from CLAUDE.md Design Principle #5): +- Skills performing critical operations MUST require explicit user confirmation +- This section appears AFTER Dependencies section (Design Principle #6) +- Read-only skills use "Not Applicable" pattern (see vm-inventory for example) + +**When to Include**: +- Playbook execution (ansible-mcp-server) +- System modifications (package updates, config changes) +- Multi-system operations (batch remediation) +- Data deletion or irreversible actions +- Resource creation that consumes cluster capacity + +**When to Omit**: +- Read-only operations (list/view operations) - Use "Not Applicable" pattern instead + +--- + +**IMPORTANT:** This skill . You MUST: + +1. **** (e.g., "Before Creating Resources") + - + - + - + - Ask: "Should I proceed with [specific action]?" + - Wait for explicit user confirmation + +2. **** (e.g., "Before Destructive Actions") + - Display preview of changes + - + - Ask: "Review the changes above. Should I execute this?" + - Wait for explicit "yes" or "proceed" + +3. **Never Auto-Execute** + - **NEVER without ** + - **NEVER when ** + - **NEVER skip ** + - **NEVER assume approval** - always wait for explicit user confirmation + +**Why This Matters:** +- ****: +- ****: +- ****: + +**Rationale**: Prevents unintended automation; maintains user control over critical operations. + +--- + +**For Read-Only Skills** (use this pattern instead): + +**Not applicable** - This skill performs read-only operations and does not modify any cluster resources. No user confirmation is required. + +**Read-only operations:** +- +- +- + +**No modifications performed:** +- ✓ Does not change cluster state +- ✓ Does not modify resources +- ✓ Does not consume cluster resources + +## Security Considerations + +- **RBAC Enforcement**: Requires for +- **Data Protection**: +- **Namespace Isolation**: +- **Storage Quotas**: +- **Audit Trail**: +- **KUBECONFIG Security**: Credentials never exposed in output +- ****: + +## Example Usage + +### Example 1: + +``` +User: "" + +Agent: [Invokes skill] + [] + + + +## + +**Field**: `value` + +| Column 1 | Column 2 | Column 3 | +|----------|----------|----------| +| data1 | data2 | data3 | + +User: "" + +Agent: [] + + + +## ✓ + +Next steps: "" +``` + +### Example 2: (OPTIONAL) + + + +**Minimum**: Include at least 1 complete example showing the full workflow. + +## Advanced Features (OPTIONAL) + +Include this section only if there are advanced use cases. + +### + + + +### + + + +**Examples:** +- Batch operations +- Special configurations +- Integration with other tools +- Performance optimizations +``` + +--- + +## Color Guide + +Use the following color codes based on operation characteristics: + +| Color | When to Use | Examples | +|-------|-------------|----------| +| **cyan** | Read-only operations (list/view) | vm-inventory, vm-snapshot-list | +| **green** | Additive operations (create new resources) | vm-create, vm-snapshot-create | +| **blue** | Reversible state changes | vm-lifecycle-manager, vm-clone | +| **yellow** | Destructive but recoverable operations | vm-snapshot-delete | +| **red** | Irreversible/critical operations (data loss risk) | vm-delete, vm-snapshot-restore | + +--- + +## Comprehensive Validation Checklist + +Before committing your skill, verify compliance with both **repository standards** and **Claude Code requirements**. + +### 1. YAML Frontmatter (Design Principle #3) + +- [ ] **name**: Lowercase with dashes, matches directory name +- [ ] **description**: Under 500 tokens total (CRITICAL) +- [ ] **description**: Includes 3-5 concrete "Use when" examples +- [ ] **description**: Includes anti-patterns with alternatives ("NOT for X, use Y instead") +- [ ] **description**: Summary line under 100 characters +- [ ] **model**: Set to "inherit" (unless special case requires specific model) +- [ ] **color**: Matches operation type (see Color Guide) + +### 2. Section Presence and Order (Design Principle #6) + +Sections MUST appear in this exact order: + +- [ ] 1. Skill title (`# /skill-name Skill`) +- [ ] 2. **Prerequisites** (with MCP server verification) +- [ ] 3. **When to Use This Skill** (with anti-patterns) +- [ ] 4. **Workflow** (with step-by-step instructions) +- [ ] 5. **Common Issues** (at least 3 issues documented) +- [ ] 6. **Dependencies** (MCP servers, tools, skills, docs) +- [ ] 7. **Critical: Human-in-the-Loop Requirements** (if applicable) +- [ ] 8. **Security Considerations** +- [ ] 9. **Example Usage** (at least 1 complete example) +- [ ] 10. **Advanced Features** (optional) + +### 3. Prerequisites Section (Design Principle #7) + +- [ ] Lists required MCP servers with setup guide links +- [ ] Lists required MCP tools with descriptions +- [ ] Lists required environment variables (if applicable) +- [ ] Includes prerequisite verification steps +- [ ] Includes Human Notification Protocol for failures +- [ ] **SECURITY**: Never exposes credential values in output + +### 4. Workflow Section (Design Principles #1-2) + +- [ ] **Document Consultation** pattern included when relevant troubleshooting docs exist + - REQUIRED when skill relates to documented error scenarios + - Consultation happens BEFORE tool invocation + - Includes "Output to user" declaration + - References specific troubleshooting docs (scheduling-errors.md, storage-errors.md, etc.) +- [ ] **Parameters**: Precise specifications with exact formats +- [ ] **Parameters**: Includes examples for each parameter +- [ ] **Expected Output**: Describes what tools return +- [ ] **Error Handling**: Documented for each step + +### 5. Common Issues Section + +- [ ] At least 3 common issues documented +- [ ] Each issue has: Error, Cause, Solution +- [ ] Solutions include 2-4 actionable steps +- [ ] Concise format (8-12 lines per issue) + +### 6. Dependencies Section (Design Principle #4) + +- [ ] **Required MCP Servers**: Listed with source links +- [ ] **Required MCP Tools**: Listed with parameters and use cases +- [ ] **Related Skills**: Listed with relationship explanation +- [ ] **Reference Documentation**: Latest stable OpenShift version +- [ ] Documentation links are valid and accessible + +### 7. Human-in-the-Loop Section (Design Principle #5) + +- [ ] Positioned AFTER Dependencies section (not before) +- [ ] Required for: resource creation, state changes, destructive operations +- [ ] Uses "Not Applicable" pattern for read-only skills +- [ ] Specifies exact confirmation points +- [ ] Includes "Why This Matters" rationale +- [ ] Uses **NEVER** statements to prevent auto-execution + +### 8. Security Considerations + +- [ ] RBAC enforcement documented +- [ ] Data protection mechanisms explained +- [ ] Namespace isolation described +- [ ] Audit trail mentioned +- [ ] KUBECONFIG security confirmed (credentials never exposed) + +### 9. Example Usage + +- [ ] At least 1 complete example included +- [ ] Shows realistic user-agent interaction +- [ ] Demonstrates full workflow from start to finish +- [ ] Uses actual markdown formatting in output + +### 10. Quality and Style + +- [ ] No emojis (unless explicitly requested by user) +- [ ] Markdown formatting correct (tables, code blocks, lists) +- [ ] No credential exposure in examples or text +- [ ] Cross-references use relative paths +- [ ] Skill name used consistently throughout + +### 11. Testing + +- [ ] Skill file loads without YAML parse errors +- [ ] All internal links are valid (skills, docs) +- [ ] All external links are accessible (Red Hat docs, GitHub) +- [ ] Skill has been tested with actual MCP server +- [ ] All workflow steps produce expected results + +### 12. Repository Standards + +- [ ] File located at `skills//SKILL.md` +- [ ] Directory name matches frontmatter `name` field +- [ ] Color code appropriate for operation risk +- [ ] Follows rh-virt collection conventions +- [ ] No conflicts with existing skill names + +--- + +**Quick Verification Commands**: + +```bash +# Check frontmatter token count (rough estimate) +wc -w skills//SKILL.md | head -n 20 + +# Verify section order +grep "^## " skills//SKILL.md + +# Check for credential exposure (should return nothing) +grep -i "password\|secret\|token.*=" skills//SKILL.md +``` + +--- + +See `SKILLS_CHECKLIST.md` for the complete validation guide with scoring criteria. + +--- + +## Tips for Writing Great Skills + +### Content Quality + +1. **Be Specific**: Use concrete examples, not generic placeholders + - ❌ "Create a VM with specified parameters" + - ✅ "Create VM database-01 with 4GB RAM in production namespace" + +2. **Show Real Output**: Include actual markdown formatting in examples + - Use real table data, actual status messages, realistic error text + - Demonstrate what users will actually see + +3. **Test First**: Run the skill before documenting to capture real behavior + - Document actual tool outputs, not assumptions + - Include real error messages you encountered + +4. **Think Like Users**: Use actual phrases users would say + - "List all VMs" not "Enumerate virtual machine resources" + - "Start the database VM" not "Initiate VM power-on sequence" + +### Claude Code Compliance + +5. **Follow Design Principles**: Reference `/CLAUDE.md` for rationale + - **Principle #1**: Document Consultation (read docs before tools) + - **Principle #2**: Precise Parameters (exact formats with examples) + - **Principle #3**: Concise Descriptions (under 500 tokens) + - **Principle #4**: Dependencies Declaration (explicit listing) + - **Principle #5**: Human-in-the-Loop (user confirmation for critical ops) + - **Principle #6**: Mandatory Sections (correct order) + - **Principle #7**: MCP Verification (check availability) + +6. **Link Everything**: Cross-reference related skills and documentation + - Use relative paths: `../../docs/file.md` or `../other-skill/SKILL.md` + - Verify links are valid before committing + +7. **Keep Current**: Use latest stable OpenShift version for all docs + - Check https://docs.redhat.com/ for current version + - Update documentation links when new versions release + +### Style and Safety + +8. **No Jargon**: Explain technical terms when first introduced + - First use: "VirtualMachineInstance (VMI) - the running pod for a VM" + - Subsequent: "VMI" is fine + +9. **Error First**: Document common failures before edge cases + - Users hit common issues 90% of the time + - Rare edge cases can go in Advanced Features + +10. **Security Always**: Never expose credentials, always check RBAC + - ✓ "KUBECONFIG is set" + - ❌ "KUBECONFIG=/path/to/kubeconfig" + - ✓ "Requires update permissions for VirtualMachines" + - ❌ Never include actual secrets, tokens, passwords + +### Maintainability + +11. **Consistency Matters**: Follow this template exactly + - Same section order across all skills + - Same formatting for MCP tools + - Same pattern for error handling + +12. **Version Control**: Document when you reference external content + - "As of OpenShift 4.21..." for version-specific behavior + - Link to specific doc versions when behavior may change + +13. **Self-Documenting**: Skill should be understandable without external context + - Don't assume user has read other skills + - Repeat critical information (RBAC requirements, prerequisites) + - Link to related skills for more detail + +### Common Pitfalls to Avoid + +- ❌ Hardcoding version numbers (use "latest stable" unless version-specific) +- ❌ Assuming prerequisites are met (always verify MCP server availability) +- ❌ Verbose Common Issues sections (keep 8-12 lines per issue) +- ❌ Missing anti-patterns in frontmatter (always include "NOT for X") +- ❌ Parameters without examples (always show exact format) +- ❌ Human-in-the-Loop before Dependencies (wrong section order) +- ❌ Description over 500 tokens (violates Claude requirement) + +### Quality Checklist Before Committing + +- [ ] Ran the skill with actual MCP server +- [ ] Captured real tool outputs in examples +- [ ] Verified all links are valid +- [ ] Checked frontmatter under 500 tokens +- [ ] Included 3-5 "Use when" examples +- [ ] Documented 3+ common issues +- [ ] Positioned Human-in-the-Loop correctly (after Dependencies) +- [ ] No credentials exposed anywhere +- [ ] Followed color guide for operation type +- [ ] Used latest stable OpenShift version + +**Remember**: Skills are loaded by Claude Code's agent system. Clear, precise, concise documentation helps Claude make correct decisions about when to invoke your skill. + +--- + +## Claude Code Design Principles Reference + +For complete details, see `/CLAUDE.md` "Design Principles for Skills and Agents" section. + +### Principle #1: Document Consultation Transparency + +**What**: Skills MUST read documentation before invoking tools, then declare consultation to user. + +**Why**: Ensures AI enriches context with domain knowledge; provides transparency to users. + +**How**: +```markdown +**Document Consultation** (REQUIRED - Execute FIRST): +1. **Action**: Read [doc.md](path) using Read tool to understand [topic] +2. **Output to user**: "I consulted [doc.md](path) to understand [topic]." +``` + +**Status for rh-virt**: REQUIRED when relevant troubleshooting documentation exists. Skills should consult docs/troubleshooting/ files before handling errors or complex operations. + +### Principle #2: Precise Parameter Specification + +**What**: Provide exact parameter names, types, and formats with examples. + +**Why**: Ensures first-attempt success when invoking MCP tools; reduces wasted cycles. + +**How**: +```markdown +**Parameters**: +- `param1`: "value" (exact format) + - Example: `"database-01"` (VM name to operate on) +``` + +### Principle #3: Skill Precedence and Conciseness + +**What**: Description field MUST be under 500 tokens; focus on "when to use". + +**Why**: Minimizes token usage at agent initialization while maintaining clarity. + +**How**: Keep frontmatter concise with 3-5 concrete examples, defer details to skill body. + +### Principle #4: Dependencies Declaration + +**What**: List all MCP servers, tools, related skills, and documentation. + +**Why**: Makes dependencies explicit for debugging and error handling. + +**How**: Follow required format in Dependencies section of template. + +### Principle #5: Human-in-the-Loop Requirements + +**What**: Skills performing critical operations MUST require explicit user confirmation. + +**Why**: Prevents unintended automation; maintains user control. + +**How**: Position AFTER Dependencies section; specify exact confirmation points. + +### Principle #6: Mandatory Skill Sections + +**What**: All sections must appear in correct order. + +**Why**: Standardizes skill structure for consistency and completeness. + +**How**: Follow exact order: Prerequisites → When to Use → Workflow → Common Issues → Dependencies → Human-in-the-Loop → Security → Examples. + +### Principle #7: MCP Server Availability Verification + +**What**: Verify MCP server configuration before executing. + +**Why**: Provides graceful degradation and clear user guidance when dependencies missing. + +**How**: Include verification steps in Prerequisites section with Human Notification Protocol. + +**CRITICAL SECURITY**: Never expose credential values in output (only report presence/absence). + +--- + +## Repository Context + +**Collection**: rh-virt (OpenShift Virtualization management) +**MCP Server**: openshift-virtualization (https://github.com/openshift/openshift-mcp-server) +**Pattern**: MCP-first approach (always use MCP tools, not CLI fallbacks) +**Documentation**: docs/troubleshooting/ contains error resolution guides (6 documents) + +**Related Collections**: +- rh-sre: Reference implementation with full docs/ directory and semantic indexing +- Use rh-sre as architectural reference for advanced patterns + +--- + +## Getting Help + +- **Template Issues**: See `SKILLS_CHECKLIST.md` for detailed validation guide +- **Claude Requirements**: Read `/CLAUDE.md` for complete design principles +- **MCP Server**: Check https://github.com/openshift/openshift-mcp-server for tool documentation +- **OpenShift Virt**: Reference https://docs.redhat.com/ for latest virtualization docs + +--- + +**Last Updated**: 2026-02-18 +**Template Version**: 2.0 (Claude Code compliant) diff --git a/rh-virt/docs/troubleshooting/INDEX.md b/rh-virt/docs/troubleshooting/INDEX.md index 0031af37..a2e76963 100644 --- a/rh-virt/docs/troubleshooting/INDEX.md +++ b/rh-virt/docs/troubleshooting/INDEX.md @@ -39,7 +39,7 @@ This guide provides comprehensive diagnostic procedures and workarounds for Virt - **Networking**: Secondary network attachment failures - **Runtime**: CrashLoopBackOff, guest OS boot failures -This guide is consulted by all rh-virt skills (vm-creator, vm-inventory, vm-lifecycle-manager, vm-delete, vm-clone) when diagnosing and remediating VM issues. +This guide is consulted by all rh-virt skills (vm-create, vm-inventory, vm-lifecycle-manager, vm-delete, vm-clone) when diagnosing and remediating VM issues. --- @@ -55,7 +55,7 @@ The troubleshooting documentation is organized by error category for easier navi - ErrorUnschedulable - Insufficient Resources - ErrorUnschedulable - Node Selector Mismatch -**Skills that use this**: vm-creator, vm-lifecycle-manager +**Skills that use this**: vm-create, vm-lifecycle-manager --- @@ -68,7 +68,7 @@ The troubleshooting documentation is organized by error category for easier navi - Storage Deletion Failures - DataVolume Cloning Failures -**Skills that use this**: vm-creator, vm-delete, vm-clone +**Skills that use this**: vm-create, vm-delete, vm-clone --- @@ -90,7 +90,7 @@ The troubleshooting documentation is organized by error category for easier navi **Errors covered**: - CrashLoopBackOff -**Skills that use this**: vm-creator, vm-lifecycle-manager +**Skills that use this**: vm-create, vm-lifecycle-manager --- @@ -100,7 +100,7 @@ The troubleshooting documentation is organized by error category for easier navi **Errors covered**: - Network Attachment Failures (Multus, SR-IOV) -**Skills that use this**: vm-creator +**Skills that use this**: vm-create --- @@ -225,7 +225,7 @@ Each error section includes: ## 🔍 Quick Navigation by Skill -**vm-creator**: +**vm-create**: - [Scheduling Errors](scheduling-errors.md) - ErrorUnschedulable diagnostics - [Storage Errors](storage-errors.md) - ErrorDataVolumeNotReady, storage provisioning - [Runtime Errors](runtime-errors.md) - CrashLoopBackOff diff --git a/rh-virt/docs/troubleshooting/lifecycle-errors.md b/rh-virt/docs/troubleshooting/lifecycle-errors.md index 084f2ce5..6167fcc8 100644 --- a/rh-virt/docs/troubleshooting/lifecycle-errors.md +++ b/rh-virt/docs/troubleshooting/lifecycle-errors.md @@ -439,7 +439,7 @@ oc describe vm -n | grep -A 20 "Events:" Extract `.spec.domain.resources.requests.memory`. - If too low, delete VM and recreate with larger instance type (e.g., change from "small" to "medium" in vm-creator). + If too low, delete VM and recreate with larger instance type (e.g., change from "small" to "medium" in vm-create). **CLI Fallback** (if MCP unavailable): ```bash diff --git a/rh-virt/docs/troubleshooting/network-errors.md b/rh-virt/docs/troubleshooting/network-errors.md index a3e920e5..82fa313c 100644 --- a/rh-virt/docs/troubleshooting/network-errors.md +++ b/rh-virt/docs/troubleshooting/network-errors.md @@ -26,7 +26,7 @@ This document covers VM secondary network attachment failures using Multus CNI a - Multus CNI failures - SR-IOV device attachment issues -**Skills that use this**: vm-creator +**Skills that use this**: vm-create --- @@ -364,7 +364,7 @@ oc describe pod virt-launcher--xxx -n 7. **Recreate VM with corrected network configuration** (if needed): - If network attachment is fundamentally broken, delete and recreate VM with correct NAD references using vm-creator skill. + If network attachment is fundamentally broken, delete and recreate VM with correct NAD references using vm-create skill. **Verification** (Use MCP Tools First): diff --git a/rh-virt/docs/troubleshooting/runtime-errors.md b/rh-virt/docs/troubleshooting/runtime-errors.md index e6f3099f..804d3cbc 100644 --- a/rh-virt/docs/troubleshooting/runtime-errors.md +++ b/rh-virt/docs/troubleshooting/runtime-errors.md @@ -25,7 +25,7 @@ This document covers VM runtime failures where the virt-launcher pod or guest OS - virt-launcher pod repeatedly restarting - Guest OS kernel panics on boot -**Skills that use this**: vm-creator, vm-lifecycle-manager +**Skills that use this**: vm-create, vm-lifecycle-manager --- @@ -278,7 +278,7 @@ virtctl console -n Extract `.spec.domain.resources.requests.memory`. - If too high for node, delete and recreate with smaller instance type using vm-creator skill (change from "large" to "medium" or "small"). + If too high for node, delete and recreate with smaller instance type using vm-create skill (change from "large" to "medium" or "small"). **CLI Fallback** (if MCP unavailable): ```bash @@ -334,7 +334,7 @@ virtctl console -n 7. **Simplify VM configuration** (eliminate variables): - Try creating minimal VM using vm-creator skill with: + Try creating minimal VM using vm-create skill with: - Small instance type - No secondary networks - Simple container disk (e.g., Fedora) @@ -344,7 +344,7 @@ virtctl console -n 8. **Recreate VM with different workload** (test disk image): - If guest OS consistently crashes, use vm-creator skill to try different OS image (e.g., switch from Ubuntu to Fedora). This tests if issue is workload-specific. + If guest OS consistently crashes, use vm-create skill to try different OS image (e.g., switch from Ubuntu to Fedora). This tests if issue is workload-specific. **Verification** (Use MCP Tools First): diff --git a/rh-virt/docs/troubleshooting/scheduling-errors.md b/rh-virt/docs/troubleshooting/scheduling-errors.md index 0c216f54..e47c8c4f 100644 --- a/rh-virt/docs/troubleshooting/scheduling-errors.md +++ b/rh-virt/docs/troubleshooting/scheduling-errors.md @@ -27,7 +27,7 @@ This document covers VM scheduling failures where the Kubernetes scheduler canno - VM shows status `ErrorUnschedulable` after creation or start attempt - VM events mention scheduling failures, taints, resources, or node selectors -**Skills that use this**: vm-creator, vm-lifecycle-manager +**Skills that use this**: vm-create, vm-lifecycle-manager --- @@ -280,7 +280,7 @@ oc describe vm -n | grep "Insufficient" **Solutions** (Use MCP Tools First): 1. **Scale cluster** - Add more worker nodes (cluster admin task, no MCP tool) -2. **Reduce VM resources** - Delete and recreate with smaller instance type using vm-creator skill +2. **Reduce VM resources** - Delete and recreate with smaller instance type using vm-create skill 3. **Delete unused VMs** - Use vm-delete skill to free up resources 4. **Check resource quotas**: diff --git a/rh-virt/docs/troubleshooting/storage-errors.md b/rh-virt/docs/troubleshooting/storage-errors.md index 02078049..16ba584d 100644 --- a/rh-virt/docs/troubleshooting/storage-errors.md +++ b/rh-virt/docs/troubleshooting/storage-errors.md @@ -29,7 +29,7 @@ This document covers VM storage-related failures including storage provisioning, - DataVolume cloning operations fail - PVC provisioning issues -**Skills that use this**: vm-creator, vm-delete, vm-clone +**Skills that use this**: vm-create, vm-delete, vm-clone --- @@ -1001,7 +1001,7 @@ oc get pvc -rootdisk -n -o jsonpath='{.status.phas # Should return: Bound ``` -**Alternative**: If cloning continuously fails, use vm-creator skill to create new VM with container disk or DataSource instead. +**Alternative**: If cloning continuously fails, use vm-create skill to create new VM with container disk or DataSource instead. --- diff --git a/rh-virt/skills/vm-clone/SKILL.md b/rh-virt/skills/vm-clone/SKILL.md index e8326139..42f2c751 100644 --- a/rh-virt/skills/vm-clone/SKILL.md +++ b/rh-virt/skills/vm-clone/SKILL.md @@ -19,35 +19,6 @@ color: blue Clone existing virtual machines in OpenShift Virtualization, creating new VMs with copied configuration and optional storage cloning. This skill is ideal for creating test environments, scaling workloads, or duplicating VM templates. -## Critical: Human-in-the-Loop Requirements - -**IMPORTANT:** This skill creates new resources that consume cluster capacity. You MUST: - -1. **Before Cloning** - - Verify source VM exists and get full configuration - - Ask user for clone configuration (name, namespace, storage strategy) - - Present clone preview with resource impact - - Wait for explicit user confirmation - -2. **Configuration Confirmation** - - Display source VM details - - Show target VM configuration - - Indicate storage cloning strategy - - Estimate resource consumption (CPU, memory, storage) - - Ask: "Proceed with VM cloning? (yes/no)" - - Wait for explicit "yes" - -3. **Never Auto-Execute** - - **NEVER clone without user confirmation** - - **NEVER assume storage strategy** - always ask user - - **NEVER proceed if user says "no", "wait", "cancel"** - -**Why This Matters:** -- **Resource Consumption**: Clones consume cluster resources (CPU, memory, storage) -- **Storage Costs**: Storage cloning can consume significant disk space -- **Naming Conflicts**: Duplicate names cause errors -- **Network Configuration**: May need adjustment for clones - ## Prerequisites **Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) @@ -149,7 +120,7 @@ Please respond with your choice. - "/vm-clone" (explicit command) **Do NOT use this skill when:** -- User wants to create a new VM from scratch → Use `/vm-creator` skill instead +- User wants to create a new VM from scratch → Use `/vm-create` skill instead - User wants a point-in-time backup → Use snapshots instead - User wants to move/migrate a VM → Use migration tools instead - User wants to resize a VM → Modify existing VM instead @@ -867,7 +838,7 @@ Modifications could include: - Source: https://github.com/openshift/openshift-mcp-server/blob/main/pkg/toolsets/core/resources.go ### Related Skills -- `vm-creator` - Create new VMs from scratch (alternative to cloning) +- `vm-create` - Create new VMs from scratch (alternative to cloning) - `vm-inventory` - List and verify source/target VMs - `vm-lifecycle-manager` - Start cloned VMs after creation - `vm-delete` - Clean up failed clones or unwanted copies @@ -875,11 +846,40 @@ Modifications could include: ### Reference Documentation - [storage-errors.md](../../docs/troubleshooting/storage-errors.md) - VM cloning failure scenarios, storage provisioning issues, and DataVolume cloning errors (optionally consulted when cloning operations fail) - [Troubleshooting INDEX](../../docs/troubleshooting/INDEX.md) - Navigation hub for discovering additional error categories when encountering unexpected issues outside the categories above -- [OpenShift Virtualization Cloning](https://docs.openshift.com/container-platform/latest/virt/virtual_machines/cloning_vms/virt-cloning-vm.html) +- [OpenShift Virtualization Cloning](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt/virtual_machines/cloning_vms/virt-cloning-vm.html) - [DataVolume Cloning](https://github.com/kubevirt/containerized-data-importer/blob/main/doc/datavolumes.md#cloning) - [KubeVirt VirtualMachine API](https://kubevirt.io/api-reference/) - [CSI Volume Cloning](https://kubernetes.io/docs/concepts/storage/volume-pvc-datasource/) +## Critical: Human-in-the-Loop Requirements + +**IMPORTANT:** This skill creates new resources that consume cluster capacity. You MUST: + +1. **Before Cloning** + - Verify source VM exists and get full configuration + - Ask user for clone configuration (name, namespace, storage strategy) + - Present clone preview with resource impact + - Wait for explicit user confirmation + +2. **Configuration Confirmation** + - Display source VM details + - Show target VM configuration + - Indicate storage cloning strategy + - Estimate resource consumption (CPU, memory, storage) + - Ask: "Proceed with VM cloning? (yes/no)" + - Wait for explicit "yes" + +3. **Never Auto-Execute** + - **NEVER clone without user confirmation** + - **NEVER assume storage strategy** - always ask user + - **NEVER proceed if user says "no", "wait", "cancel"** + +**Why This Matters:** +- **Resource Consumption**: Clones consume cluster resources (CPU, memory, storage) +- **Storage Costs**: Storage cloning can consume significant disk space +- **Naming Conflicts**: Duplicate names cause errors +- **Network Configuration**: May need adjustment for clones + ## Security Considerations - **RBAC Enforcement**: Requires create permissions in target namespace diff --git a/rh-virt/skills/vm-creator/SKILL.md b/rh-virt/skills/vm-create/SKILL.md similarity index 94% rename from rh-virt/skills/vm-creator/SKILL.md rename to rh-virt/skills/vm-create/SKILL.md index a710af9c..153b6fec 100644 --- a/rh-virt/skills/vm-creator/SKILL.md +++ b/rh-virt/skills/vm-create/SKILL.md @@ -1,5 +1,5 @@ --- -name: vm-creator +name: vm-create description: | Create new virtual machines in OpenShift Virtualization with automatic instance type resolution and OS selection. Use this skill when users request: - "Create a new VM" @@ -9,32 +9,16 @@ description: | This skill handles VM creation with intelligent defaults for OpenShift Virtualization. + NOT for managing existing VMs (use vm-lifecycle-manager or vm-delete instead). + model: inherit color: green --- -# /vm-creator Skill +# /vm-create Skill Create virtual machines in OpenShift Virtualization using the `vm_create` tool from the openshift-virtualization MCP server. -## Critical: Human-in-the-Loop Requirements - -**IMPORTANT:** This skill requires explicit user confirmation before creating VMs. You MUST: - -1. **Wait for user confirmation** on all VM configuration parameters before executing `vm_create`. -2. **Do NOT proceed** with VM creation until the user explicitly approves the configuration. -3. **Present configuration clearly** in a table format and wait for user response. -4. **Never auto-execute** VM creation without user approval - creating VMs is a additive operation that consumes cluster resources, affects quotas and may incur cost. -5. **Allow modifications** - If user wants to change parameters, update and re-confirm before proceeding. - -If the user says "no" or wants modifications, address their concerns before proceeding. - -**Why this matters:** -- VM creation consumes cluster resources (CPU, memory, storage) -- VMs persist until explicitly deleted -- Incorrect configuration can impact cluster performance -- User should verify namespace, sizing, and other parameters - ## Prerequisites **Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) @@ -72,7 +56,7 @@ If the user says "no" or wants modifications, address their concerns before proc When prerequisites fail: ``` -❌ Cannot execute vm-creator: MCP server 'openshift-virtualization' is not available +❌ Cannot execute vm-create: MCP server 'openshift-virtualization' is not available 📋 Setup Instructions: 1. Add openshift-virtualization to .mcp.json: @@ -120,7 +104,7 @@ Please respond with your choice. ## When to Use This Skill **Trigger this skill when:** -- User explicitly invokes `/vm-creator` command +- User explicitly invokes `/vm-create` command - User requests creating a new virtual machine - Deploying VMs with specific OS (Fedora, Ubuntu, RHEL, CentOS, Debian) - Setting up VMs with custom sizing (small, medium, large) @@ -130,54 +114,13 @@ Please respond with your choice. - "Create a Fedora VM in namespace vms" - "Deploy a medium Ubuntu VM with 100Gi disk" - "Set up a RHEL VM called database-01" -- "/vm-creator" (explicit command) +- "/vm-create" (explicit command) **Do NOT use this skill when:** - User wants to start/stop existing VMs → Use `/vm-lifecycle-manager` skill instead - User wants to list VMs → Use `/vm-inventory` skill instead - User only wants information about VMs (not creation) → Use `/vm-inventory` skill instead -## CRITICAL: MCP Tools First Policy - -**MANDATORY REQUIREMENT**: You MUST ALWAYS use MCP tools from the openshift-virtualization server for ALL cluster operations. - -**MCP Tools Available:** -- `namespaces_list` - List all namespaces -- `resources_list` - List resources (StorageClass, VirtualMachine, etc.) -- `resources_get` - Get specific resource details -- `resources_create_or_update` - Create or update resources -- `resources_delete` - Delete resources -- `pods_list` - List pods -- `pods_exec` - Execute commands in pods -- `events_list` - List cluster events -- And many more... - -**Policy:** -1. **ALWAYS check if an MCP tool exists** for the operation you need to perform -2. **ONLY use kubectl/oc CLI commands** when: - - No equivalent MCP tool exists for that specific operation - - The MCP tool has been tried and failed - - You have explicit confirmation that the MCP approach is not possible - -**Examples:** -- ❌ WRONG: `kubectl get namespaces` → ✅ CORRECT: Use `namespaces_list` MCP tool -- ❌ WRONG: `kubectl get storageclass -o json` → ✅ CORRECT: Use `resources_list` MCP tool with apiVersion="storage.k8s.io/v1", kind="StorageClass" -- ❌ WRONG: `kubectl get vm -n ` → ✅ CORRECT: Use `resources_get` MCP tool with apiVersion="kubevirt.io/v1", kind="VirtualMachine" -- ❌ WRONG: `kubectl config view --minify` → ⚠️ ACCEPTABLE: No MCP equivalent exists for kubeconfig context detection - -**Why this matters:** -- MCP tools provide structured, validated outputs -- Better error handling and user experience -- Consistent interface across all operations -- Reduced dependency on CLI tools -- Better integration with Claude Code environment - -**If you catch yourself about to use kubectl/oc:** -1. STOP -2. Check the available MCP tools list above -3. Use the MCP tool instead -4. Only proceed with kubectl/oc if absolutely no alternative exists - ## Workflow ### Step 1: Gather VM Requirements and Confirm Configuration @@ -997,7 +940,7 @@ Would you like me to display the default credentials for the VM? (yes/no) Would you like help troubleshooting this error? ``` -## Advanced Usage +## Advanced Features ### Custom Container Disk Images @@ -1130,11 +1073,29 @@ vm_create({ - [runtime-errors.md](../../docs/troubleshooting/runtime-errors.md) - CrashLoopBackOff and guest OS failures - [network-errors.md](../../docs/troubleshooting/network-errors.md) - Network attachment failures - [Troubleshooting INDEX](../../docs/troubleshooting/INDEX.md) - Navigation hub for discovering additional error categories when encountering unexpected issues outside the categories above -- [OpenShift Virtualization Documentation](https://docs.openshift.com/container-platform/latest/virt/about_virt/about-virt.html) -- [Troubleshooting VMs](https://docs.openshift.com/container-platform/latest/virt/virtual_machines/troubleshooting_vms.html) +- [OpenShift Virtualization Documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt/about_virt/about-virt.html) +- [Troubleshooting VMs](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt/virtual_machines/troubleshooting_vms.html) - [KubeVirt VirtualMachine API](https://kubevirt.io/api-reference/) - [OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server) -- [StorageClass Documentation](https://docs.openshift.com/container-platform/latest/storage/understanding-persistent-storage.html) +- [StorageClass Documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#storage/understanding-persistent-storage.html) + +## Critical: Human-in-the-Loop Requirements + +**IMPORTANT:** This skill requires explicit user confirmation before creating VMs. You MUST: + +1. **Wait for user confirmation** on all VM configuration parameters before executing `vm_create`. +2. **Do NOT proceed** with VM creation until the user explicitly approves the configuration. +3. **Present configuration clearly** in a table format and wait for user response. +4. **Never auto-execute** VM creation without user approval - creating VMs is a additive operation that consumes cluster resources, affects quotas and may incur cost. +5. **Allow modifications** - If user wants to change parameters, update and re-confirm before proceeding. + +If the user says "no" or wants modifications, address their concerns before proceeding. + +**Why this matters:** +- VM creation consumes cluster resources (CPU, memory, storage) +- VMs persist until explicitly deleted +- Incorrect configuration can impact cluster performance +- User should verify namespace, sizing, and other parameters ## Security Considerations @@ -1152,7 +1113,7 @@ vm_create({ ``` User: "Create a VM" -Agent: [Invokes /vm-creator skill] +Agent: [Invokes /vm-create skill] [Detects current namespace from kubeconfig: "production"] [Queries available StorageClasses, finds default: "ocs-storagecluster-ceph-rbd"] [No parameters provided, uses AskUserQuestion for full interactive menu] @@ -1215,7 +1176,7 @@ Next Steps: "Start VM my-app-server in namespace production" ``` User: "Create an Ubuntu VM called web-01 in namespace apps" -Agent: [Invokes /vm-creator skill] +Agent: [Invokes /vm-create skill] [Detects VM name="web-01", namespace="apps", OS="ubuntu"] [Queries StorageClasses for menu] [Uses AskUserQuestion only for: Performance, Size, Storage, Storage Class, Autostart] @@ -1273,7 +1234,7 @@ VM Details: ``` User: "Deploy a memory-optimized RHEL VM called db-server in production namespace" -Agent: [Invokes /vm-creator skill] +Agent: [Invokes /vm-create skill] [Detects: name="db-server", namespace="production", OS="rhel", performance="m1" (memory-optimized)] [Uses AskUserQuestion for: Size, Storage, Storage Class, Autostart] @@ -1336,7 +1297,7 @@ VM Details: ``` User: "Create a VM named apache-server in namespace vms" -Agent: [Invokes /vm-creator skill] +Agent: [Invokes /vm-create skill] [Validates name: "apache-server" ✓] [Checks if VM exists...] @@ -1375,7 +1336,7 @@ Agent: [Validates new name: "apache-server-02" ✓] ``` User: "Create a VM named test in namespace dev" -Agent: [Invokes /vm-creator skill] +Agent: [Invokes /vm-create skill] [Interactive menu presented...] [Processes responses] diff --git a/rh-virt/skills/vm-delete/SKILL.md b/rh-virt/skills/vm-delete/SKILL.md index f1d7068e..a2639d88 100644 --- a/rh-virt/skills/vm-delete/SKILL.md +++ b/rh-virt/skills/vm-delete/SKILL.md @@ -19,50 +19,6 @@ color: red Permanently delete virtual machines and their associated resources (storage, DataVolumes) from OpenShift Virtualization clusters. This skill enforces strict safety protocols including typed confirmation and pre-deletion validation. -## Critical: Human-in-the-Loop Requirements - -**CRITICAL: This skill performs IRREVERSIBLE operations.** You MUST: - -1. **Pre-Deletion Validation** (Execute FIRST, before asking for confirmation) - - Verify VM exists using `resources_get` - - Check VM running state - - Discover dependent resources (DataVolumes, PVCs) - - Check for protection labels (`protected: "true"`) - -2. **Safety Checks** - - **REFUSE deletion** if VM has label `protected: "true"` - - **REQUIRE VM to be stopped first** if currently running - - **List all resources** that will be deleted (VM, storage, DataVolumes) - -3. **Typed Confirmation Protocol** (MANDATORY) - - Display complete deletion scope - - **Require user to type the exact VM name** to confirm - - Accept only exact match (case-sensitive) - - If name doesn't match → Cancel operation, do not proceed - -4. **Deletion Options** - - Ask user what to delete: - - Option 1: VM only (preserve storage) - - Option 2: VM + storage (complete cleanup) - - Option 3: Cancel - - **NEVER assume** which option user wants - -5. **Final Confirmation Before Each Deletion** - - After typed confirmation, show exactly what will be deleted - - Ask: "Proceed with permanent deletion? (yes/cancel)" - - Wait for explicit "yes" - -6. **Never Auto-Execute** - - **NEVER delete without explicit typed confirmation** - - **NEVER proceed if user says "no", "wait", "cancel"** - - **NEVER skip the typed verification step** - -**Why This Matters:** -- **Permanent**: Deleted VMs cannot be recovered -- **Data Loss**: Storage deletion destroys all VM data -- **Service Impact**: Deleting running VMs causes immediate service outage -- **Accidental Deletion**: Typed verification prevents mistakes (typos, wrong VM name) - ## Prerequisites **Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) @@ -166,7 +122,7 @@ Please respond with your choice. **Do NOT use this skill when:** - User wants to stop a VM temporarily → Use `/vm-lifecycle-manager` skill instead -- User wants to create a VM → Use `/vm-creator` skill instead +- User wants to create a VM → Use `/vm-create` skill instead - User wants to view VMs → Use `/vm-inventory` skill instead - User wants to pause or suspend VM → Use lifecycle management (not deletion) @@ -910,18 +866,62 @@ Names do not match. Deletion cancelled for safety. ### Related Skills - `vm-inventory` - Verify VM exists and get details before deletion - `vm-lifecycle-manager` - Stop running VMs before deletion (used internally by vm-delete) -- `vm-creator` - Create VMs after cleanup operations +- `vm-create` - Create VMs after cleanup operations ### Reference Documentation - [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) - VM deletion failure scenarios, finalizer issues, and stuck Terminating states (optionally consulted when deletion operations fail) - [storage-errors.md](../../docs/troubleshooting/storage-errors.md) - Storage deletion strategies and PVC cleanup procedures (optionally consulted when storage deletion fails) - [Troubleshooting INDEX](../../docs/troubleshooting/INDEX.md) - Navigation hub for discovering additional error categories when encountering unexpected issues outside the categories above -- [OpenShift Virtualization Documentation](https://docs.openshift.com/container-platform/latest/virt/about_virt/about-virt.html) +- [OpenShift Virtualization Documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt/about_virt/about-virt.html) - [KubeVirt VirtualMachine API](https://kubevirt.io/api-reference/) - [Kubernetes Finalizers](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/) - [PVC Deletion](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#delete) - [OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server) +## Critical: Human-in-the-Loop Requirements + +**CRITICAL: This skill performs IRREVERSIBLE operations.** You MUST: + +1. **Pre-Deletion Validation** (Execute FIRST, before asking for confirmation) + - Verify VM exists using `resources_get` + - Check VM running state + - Discover dependent resources (DataVolumes, PVCs) + - Check for protection labels (`protected: "true"`) + +2. **Safety Checks** + - **REFUSE deletion** if VM has label `protected: "true"` + - **REQUIRE VM to be stopped first** if currently running + - **List all resources** that will be deleted (VM, storage, DataVolumes) + +3. **Typed Confirmation Protocol** (MANDATORY) + - Display complete deletion scope + - **Require user to type the exact VM name** to confirm + - Accept only exact match (case-sensitive) + - If name doesn't match → Cancel operation, do not proceed + +4. **Deletion Options** + - Ask user what to delete: + - Option 1: VM only (preserve storage) + - Option 2: VM + storage (complete cleanup) + - Option 3: Cancel + - **NEVER assume** which option user wants + +5. **Final Confirmation Before Each Deletion** + - After typed confirmation, show exactly what will be deleted + - Ask: "Proceed with permanent deletion? (yes/cancel)" + - Wait for explicit "yes" + +6. **Never Auto-Execute** + - **NEVER delete without explicit typed confirmation** + - **NEVER proceed if user says "no", "wait", "cancel"** + - **NEVER skip the typed verification step** + +**Why This Matters:** +- **Permanent**: Deleted VMs cannot be recovered +- **Data Loss**: Storage deletion destroys all VM data +- **Service Impact**: Deleting running VMs causes immediate service outage +- **Accidental Deletion**: Typed verification prevents mistakes (typos, wrong VM name) + ## Security Considerations - **RBAC Enforcement**: Deletion requires explicit RBAC permissions for VirtualMachine, PVC, and DataVolume resources diff --git a/rh-virt/skills/vm-inventory/SKILL.md b/rh-virt/skills/vm-inventory/SKILL.md index 195bc3aa..53dc1e4f 100644 --- a/rh-virt/skills/vm-inventory/SKILL.md +++ b/rh-virt/skills/vm-inventory/SKILL.md @@ -9,6 +9,8 @@ description: | This skill provides comprehensive VM inventory and status reporting. + NOT for creating or modifying VMs (use vm-create or vm-lifecycle-manager instead). + model: inherit color: cyan --- @@ -17,23 +19,6 @@ color: cyan List and inspect virtual machines in OpenShift Virtualization clusters. This skill provides read-only access to VM information without making any modifications. -## Critical: Human-in-the-Loop Requirements - -**Not applicable** - This skill performs read-only operations and does not modify any cluster resources. No user confirmation is required. - -**Read-only operations:** -- Listing VirtualMachines across namespaces or in specific namespaces -- Retrieving VM details, status, and resource configurations -- Displaying VM health conditions and resource usage -- Filtering VMs by labels or field selectors -- Viewing VM network, storage, and node placement information - -**No modifications performed:** -- ✓ Does not change VM state (start/stop/restart) -- ✓ Does not modify VM configuration -- ✓ Does not delete VMs or resources -- ✓ Does not consume cluster resources - ## Prerequisites **Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) @@ -145,7 +130,7 @@ Please respond with your choice. - "/vm-inventory" (explicit command) **Do NOT use this skill when:** -- User wants to create a VM → Use `/vm-creator` skill instead +- User wants to create a VM → Use `/vm-create` skill instead - User wants to start/stop VMs → Use `/vm-lifecycle-manager` skill instead - User wants to modify VM configuration → Different operation (not inventory) @@ -523,7 +508,7 @@ Found 3 VMs: - web-dev-01 (development) - Running ``` -## Common Queries and Responses +## Common Issues ### "Show me all running VMs" @@ -680,7 +665,7 @@ No VMs were found in this namespace. - Insufficient permissions to view VMs **Next steps:** -- Create a VM: Use /vm-creator skill +- Create a VM: Use /vm-create skill - List all namespaces: "Show me all namespaces" - Check permissions: `oc auth can-i list virtualmachines -n production` ``` @@ -707,7 +692,7 @@ No VMs were found in this namespace. ## Integration with Other Skills -**Before creating a VM** (vm-creator): +**Before creating a VM** (vm-create): - Use vm-inventory to check if VM name already exists - Verify namespace exists and has capacity @@ -742,17 +727,34 @@ No VMs were found in this namespace. **Important**: Always attempt MCP tools first. Only use CLI commands after MCP tool failure and with user confirmation. ### Related Skills -- `vm-creator` - Create VMs after checking inventory +- `vm-create` - Create VMs after checking inventory - `vm-lifecycle-manager` - Manage VMs discovered in inventory - `vm-troubleshooter` (planned) - Diagnose problematic VMs from inventory ### Reference Documentation - [Troubleshooting INDEX](../../docs/troubleshooting/INDEX.md) - VM status interpretation and navigation hub for discovering error-specific troubleshooting guides (optionally consulted when displaying VM details with error states) -- [OpenShift Virtualization Documentation](https://docs.openshift.com/container-platform/latest/virt/about_virt/about-virt.html) +- [OpenShift Virtualization Documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt/about_virt/about-virt.html) - [KubeVirt VirtualMachine API](https://kubevirt.io/api-reference/) -- [Accessing VMs](https://docs.openshift.com/container-platform/latest/virt/virtual_machines/virt-accessing-vm-consoles.html) +- [Accessing VMs](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt/virtual_machines/virt-accessing-vm-consoles.html) - [VM Status Conditions](https://kubevirt.io/user-guide/virtual_machines/vm_status_conditions/) +## Critical: Human-in-the-Loop Requirements + +**Not applicable** - This skill performs read-only operations and does not modify any cluster resources. No user confirmation is required. + +**Read-only operations:** +- Listing VirtualMachines across namespaces or in specific namespaces +- Retrieving VM details, status, and resource configurations +- Displaying VM health conditions and resource usage +- Filtering VMs by labels or field selectors +- Viewing VM network, storage, and node placement information + +**No modifications performed:** +- ✓ Does not change VM state (start/stop/restart) +- ✓ Does not modify VM configuration +- ✓ Does not delete VMs or resources +- ✓ Does not consume cluster resources + ## Security Considerations - Read-only operations - no modifications to VMs @@ -906,5 +908,5 @@ To create a VM: "Create a VM in namespace test" ``` -The /vm-creator skill will help you set up a new virtual machine. +The /vm-create skill will help you set up a new virtual machine. ``` diff --git a/rh-virt/skills/vm-lifecycle-manager/SKILL.md b/rh-virt/skills/vm-lifecycle-manager/SKILL.md index daa60e05..6051b3d4 100644 --- a/rh-virt/skills/vm-lifecycle-manager/SKILL.md +++ b/rh-virt/skills/vm-lifecycle-manager/SKILL.md @@ -9,6 +9,8 @@ description: | This skill handles VM state transitions safely with user confirmation for each action. + NOT for creating VMs (use vm-create) or deleting VMs (use vm-delete). + model: inherit color: blue --- @@ -17,24 +19,6 @@ color: blue Control virtual machine power state and lifecycle operations in OpenShift Virtualization using the `vm_lifecycle` tool from the openshift-virtualization MCP server. -## Critical: Human-in-the-Loop Requirements - -**IMPORTANT:** This skill requires explicit user confirmation before executing lifecycle actions. You MUST: - -1. **Wait for user confirmation** on the specific action (start/stop/restart) before executing `vm_lifecycle` -2. **Do NOT proceed** with the lifecycle operation until the user explicitly approves -3. **Present the action clearly** with VM name, namespace, and intended state change -4. **Never auto-execute** VM lifecycle changes without user approval - these operations impact running services -5. **Warn about consequences** - stopping VMs interrupts services, starting consumes resources, restarting causes brief downtime - -If the user says "no" or wants to reconsider, do not proceed with the operation. - -**Why this matters:** -- **Start**: Consumes cluster resources (CPU, memory), affects resource availability for other VMs -- **Stop**: Interrupts running services and applications, may cause service downtime -- **Restart**: Causes brief service interruption, running processes will be terminated -- User should verify they're targeting the correct VM and understand the impact - ## Prerequisites **Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) @@ -134,7 +118,7 @@ Please respond with your choice. - "/vm-lifecycle-manager" (explicit command) **Do NOT use this skill when:** -- User wants to create a new VM → Use `/vm-creator` skill instead +- User wants to create a new VM → Use `/vm-create` skill instead - User wants to list VMs → Use `/vm-inventory` skill instead - User wants to delete a VM → Different operation (not lifecycle management) @@ -405,7 +389,7 @@ Would you like help troubleshooting this error? 1. Check cluster resource availability 2. Stop other VMs first to free resources 3. Consider scaling cluster nodes -4. Resize VM to smaller instance type (using `/vm-creator` to recreate) +4. Resize VM to smaller instance type (using `/vm-create` to recreate) ## Understanding RunStrategy @@ -424,7 +408,7 @@ When you execute lifecycle actions, the VM's `runStrategy` changes: **Note**: This skill sets `Always` for start/restart and `Halted` for stop. -## Advanced Usage +## Advanced Features ### Batch Operations @@ -476,7 +460,7 @@ User: "Start web-server if it's not running" - `vm_lifecycle` (from openshift-virtualization) - Manage VM power state (start/stop/restart) ### Related Skills -- `vm-creator` - Create VMs before managing their lifecycle +- `vm-create` - Create VMs before managing their lifecycle - `vm-inventory` - Check current VM status before lifecycle operations - `vm-troubleshooter` (planned) - Diagnose VM startup/shutdown issues @@ -484,11 +468,29 @@ User: "Start web-server if it's not running" - [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) - VM start/stop failures and stuck transitions (optionally consulted when lifecycle operations fail) - [scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) - ErrorUnschedulable and resource constraint errors (optionally consulted when VM won't start) - [Troubleshooting INDEX](../../docs/troubleshooting/INDEX.md) - Navigation hub for discovering additional error categories when encountering unexpected issues outside the categories above -- [OpenShift Virtualization Documentation](https://docs.openshift.com/container-platform/latest/virt/about_virt/about-virt.html) +- [OpenShift Virtualization Documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt/about_virt/about-virt.html) - [KubeVirt VirtualMachine Lifecycle](https://kubevirt.io/user-guide/virtual_machines/lifecycle/) - [RunStrategy Documentation](https://kubevirt.io/user-guide/virtual_machines/run_strategies/) - [OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server) +## Critical: Human-in-the-Loop Requirements + +**IMPORTANT:** This skill requires explicit user confirmation before executing lifecycle actions. You MUST: + +1. **Wait for user confirmation** on the specific action (start/stop/restart) before executing `vm_lifecycle` +2. **Do NOT proceed** with the lifecycle operation until the user explicitly approves +3. **Present the action clearly** with VM name, namespace, and intended state change +4. **Never auto-execute** VM lifecycle changes without user approval - these operations impact running services +5. **Warn about consequences** - stopping VMs interrupts services, starting consumes resources, restarting causes brief downtime + +If the user says "no" or wants to reconsider, do not proceed with the operation. + +**Why this matters:** +- **Start**: Consumes cluster resources (CPU, memory), affects resource availability for other VMs +- **Stop**: Interrupts running services and applications, may cause service downtime +- **Restart**: Causes brief service interruption, running processes will be terminated +- User should verify they're targeting the correct VM and understand the impact + ## Security Considerations - VM lifecycle changes require RBAC permissions (update VirtualMachine resources) diff --git a/rh-virt/skills/vm-rebalance/REBALANCE_AUTOMATIC.md b/rh-virt/skills/vm-rebalance/REBALANCE_AUTOMATIC.md new file mode 100644 index 00000000..936a4f3b --- /dev/null +++ b/rh-virt/skills/vm-rebalance/REBALANCE_AUTOMATIC.md @@ -0,0 +1,760 @@ +# Automatic Rebalancing Strategy + +**Status**: ✅ PRODUCTION READY + +**Purpose**: AI-driven rebalancing where user explains high-level goals (CPU balance, memory optimization, drain node, etc.) and AI generates optimal rebalance plan. User can modify or approve plan before execution. + +--- + +## When to Use Automatic Mode + +Use this mode when the user wants: +- AI to analyze cluster and propose optimal rebalancing +- High-level goal specification (balance CPU, optimize memory, drain node) +- Expert recommendations with ability to customize +- Multi-objective optimization (CPU AND memory simultaneously) +- Intelligent rebalance planning without manual VM-by-node decisions + +**User Request Patterns:** +- "Rebalance VMs based on CPU load" +- "Optimize cluster for CPU and memory" +- "Drain worker-02 for maintenance" +- "Automatically balance the cluster" +- "Help me redistribute VMs to improve performance" +- "Optimize VM placement" + +**Do NOT use Automatic mode when:** +- User specifies exact VM→node mappings → Use Manual mode +- User only wants to see available VMs → Use `/vm-inventory` skill + +--- + +## Workflow + +### Step 1: Gather Cluster State and Determine Optimization Goal + +**1.1 Collect Cluster Information** + +**List all VMs across namespaces:** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine" +} +``` + +Extract for each VM: +- Name, namespace +- Current node placement (from VirtualMachineInstance if running) +- Resource requests (CPU, memory) +- Storage type (RWX vs RWO) - determines live vs cold migration capability + +**List all nodes:** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "Node" +} +``` + +Extract for each node: +- Name, status (Ready/NotReady) +- Capacity and allocatable resources +- Current utilization +- Taints and labels +- Schedulable status (not cordoned) + +**Gather resource usage metrics:** + +**MCP Tool**: `nodes_top` (from openshift-virtualization) + +**Parameters**: None (lists all nodes) + +Extract current CPU and memory utilization for each node. + +**MCP Tool**: `pods_top` (from openshift-virtualization) + +**Parameters**: +```json +{ + "all_namespaces": true, + "label_selector": "kubevirt.io=virt-launcher" +} +``` + +Extract current CPU and memory usage for each VM. + +--- + +**1.2 Determine Optimization Goal from User Request** + +**Analyze user's language to infer goal:** + +| User Phrase | Optimization Goal | Metrics to Optimize | +|-------------|-------------------|---------------------| +| "balance CPU", "CPU load" | Balance CPU utilization | Minimize CPU variance across nodes | +| "optimize memory", "memory pressure" | Balance memory utilization | Minimize memory variance across nodes | +| "balance cluster", "rebalance", "optimize" | Multi-objective (CPU + memory) | Minimize both CPU and memory variance | +| "drain worker-02", "evacuate node-X" | Drain specific node | Migrate all VMs off target node | +| "optimize performance" | Performance optimization | Balance resources + avoid hotspots | +| "distribute VMs evenly" | VM count distribution | Equal number of VMs per node | + +**If goal is ambiguous**, ask user to clarify: + +``` +I can optimize the cluster for several goals: +1. **CPU load balancing** - Distribute CPU usage evenly across nodes +2. **Memory load balancing** - Distribute memory usage evenly across nodes +3. **Both CPU and memory** - Multi-objective optimization +4. **Drain specific node** - Move all VMs off a node for maintenance +5. **VM count distribution** - Equal number of VMs per node + +Which optimization goal would you like me to pursue? +``` + +**WAIT for user response** before proceeding. + +--- + +**1.3 Support Multi-Objective Optimization** + +When user requests multiple goals (e.g., "balance CPU and memory"): + +**Approach:** +1. **Calculate scoring function** combining all objectives +2. **Weight objectives** (can ask user for priorities or use defaults) +3. **Find rebalance plan** that optimizes combined score + +**Example Scoring:** +``` +Score = (0.5 × CPU_variance_reduction) + (0.5 × Memory_variance_reduction) +``` + +**User can adjust weights** if AI proposes alternative approach: + +``` +I can optimize for: +- Equal priority: CPU 50%, Memory 50% +- CPU-focused: CPU 70%, Memory 30% +- Memory-focused: CPU 30%, Memory 70% + +Would you like to adjust priorities, or proceed with equal weighting? +``` + +--- + +### Step 2: Analyze and Generate Optimal Migration Plan + +**2.1 Identify Migration Candidates** + +For each optimization goal: + +**CPU Balancing:** +- Identify overloaded nodes (>80% CPU) +- Identify underloaded nodes (<50% CPU) +- Select VMs to migrate from overloaded to underloaded nodes + +**Memory Balancing:** +- Identify nodes with high memory pressure (>85%) +- Identify nodes with low memory usage (<50%) +- Select VMs to migrate for better distribution + +**Node Drain:** +- Select ALL VMs currently on target node +- Find suitable destination nodes with capacity + +**Performance Optimization:** +- Identify VMs with high resource variance (bursty workloads) +- Distribute high-performance VMs across different nodes +- Avoid co-locating resource-intensive VMs + +**2.2 Apply Constraints and Validation** + +For each candidate migration, check: + +**Storage Compatibility** (see SKILL.md - Common Validation Logic): +- RWX storage → Live migration possible +- RWO storage → Cold migration required +- Stopped VM → Cold migration required + +**Target Node Capacity:** +- Verify target has sufficient CPU and memory +- Account for VM resource requests +- Ensure node is Ready and schedulable + +**Taints and Tolerations:** +- Check target node for taints +- Verify VM has matching tolerations +- If mismatch, skip that target or propose adding tolerations + +**Concurrency Limits** (see [references/performance-tuning.md](./references/performance-tuning.md)): +- Cluster limit: 5 concurrent migrations (default) +- Per-node limit: 2 outbound migrations (default) +- Plan migration batches respecting limits + +**Network Bandwidth:** +- Avoid saturating network with too many concurrent large VM migrations +- Consider VM memory size when scheduling concurrent migrations + +**2.3 Optimize Migration Plan** + +**Migration Ordering Strategy:** + +1. **Smallest VMs first** - Faster migrations, higher success rate +2. **Live migrations before cold** - Minimize total downtime +3. **Group by source node** - Efficient for node draining +4. **Respect dependencies** - Avoid migrating related VMs simultaneously (e.g., database + app tier) + +**Expected Improvement Calculation:** + +**Before migration:** +``` +CPU variance = StdDev([worker-01: 85%, worker-02: 78%, worker-03: 42%, worker-04: 38%]) + = 22.1% +``` + +**After migration:** +``` +CPU variance = StdDev([worker-01: 65%, worker-02: 58%, worker-03: 62%, worker-04: 55%]) + = 4.2% +``` + +**Improvement:** +``` +Variance reduction: 22.1% → 4.2% = 81% improvement +``` + +--- + +### Step 3: Present Plan to User with Modification Options + +**CRITICAL**: Use the **Common Plan Visualization** format defined in SKILL.md. + +**Reference**: [SKILL.md - Common Plan Visualization](./SKILL.md#common-plan-visualization) + +**Present the plan with:** +1. **Optimization Goal** statement +2. **VM Rebalance Plan** table (from SKILL.md common format) +3. **Node State: Before → After** table (from SKILL.md common format) +4. **Key Improvement** summary line +5. **Migration Summary** (for batch operations) +6. **Risks & Considerations** (optional) +7. **User options** (approve / modify / explain / cancel) + +**Additional context for automatic mode:** +- State the optimization goal clearly (CPU balance, memory balance, multi-objective, node drain, etc.) +- Include expected improvements with metrics (variance reduction, capacity headroom, etc.) +- Offer modification options (skip, change target, reorder) +- Allow user to explain reasoning for specific migrations + +**CRITICAL: WAIT for user response before executing.** + +--- + +**3.1 Handle User Modifications** + +If user selects **"modify"**, enter interactive modification mode: + +**Example Interaction:** + +``` +User: "modify" + +Agent: "Which migrations would you like to modify?" + +Options: +- "skip <#>" - Remove migration from plan (e.g., "skip 5") +- "change target <#> to " - Change destination (e.g., "change target 1 to worker-04") +- "reorder" - Change execution order +- "done" - Finish modifications and execute + +User: "skip 5" + +Agent: "Migration #5 (app-vm-05) removed from plan. Updated plan:" + +## Updated Rebalancing Plan + +**Proposed Migrations:** (Now 4 migrations) + +| # | VM | Current Node | Target Node | Type | Impact | +|---|----|--------------|-------------|------|--------| +| 1 | database-01 | worker-01 | worker-03 | live | <1s | +| 2 | web-server-02 | worker-01 | worker-04 | live | <1s | +| 3 | cache-vm | worker-02 | worker-04 | live | <1s | +| 4 | api-gateway | worker-02 | worker-03 | live | <1s | + +**Updated Expected Improvements:** +- CPU variance: 22.1% → 5.8% (74% improvement) +- Memory variance: 11.8% → 8.1% (31% improvement) + +**Ready to execute?** (approve / modify more / cancel) + +User: "approve" + +Agent: [Proceeds to Step 4 - Execution] +``` + +--- + +**3.2 AI Proposes Better Approach (If Detected)** + +If AI detects a potentially better strategy, propose alternative: + +``` +💡 **Alternative Approach Detected** + +I notice that migrating app-vm-05 via cold migration will cause 40s downtime. However, I found: +- app-vm-05 is currently stopped (not running) +- Moving it now via cold migration has **zero additional downtime** since it's already offline + +**Alternative proposal:** +Include app-vm-05 in the plan (no additional impact vs current state) + +Would you like to: +- **accept alternative** - Include app-vm-05 in plan +- **keep original** - Proceed with current plan +- **explain more** - Tell me more about this alternative +``` + +**User has final word** - If user prefers original plan, execute original plan. + +--- + +### Step 4: Validate All VMs Before Execution + +**BEFORE executing any migration**, validate ALL VMs in the plan: + +**For each VM**, perform **Common Validation Logic from SKILL.md**: + +1. **Verify VM exists** (see SKILL.md - Validation 1) +2. **Check current location** (see SKILL.md - Validation 2) +3. **Validate storage compatibility** (see SKILL.md - Validation 3) +4. **Verify target node exists** (see SKILL.md - Validation 4) + +**If any VM fails validation:** +- Remove from rebalance plan +- Warn user: "Migration #X (vm-name) failed validation: [reason]. Proceeding with remaining migrations." +- Continue with other migrations + +**Reference**: [SKILL.md - Common Validation Logic](./SKILL.md#common-validation-logic) + +--- + +### Step 5: Execute Migrations with Progress Reporting + +**5.1 Group Migrations by Type** + +**Live Migrations** (execute first): +- Can run concurrently (up to cluster limits) +- Lower risk, zero downtime +- Follow live migration workflow from REBALANCE_MANUAL.md + +**Cold Migrations** (execute after live migrations): +- Run sequentially (to prevent cascading failures) +- Higher risk, has downtime +- Follow cold migration workflow from REBALANCE_MANUAL.md + +**5.2 Respect Concurrency Limits** + +**Cluster-wide limit**: 5 concurrent migrations (default) +**Per-node limit**: 2 outbound migrations per source node (default) + +**Monitor current migrations:** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstanceMigration" +} +``` + +Count migrations where `.status.phase` is NOT "Succeeded" or "Failed". + +**Wait if at limit** before starting new migrations. + +**Reference**: [references/performance-tuning.md](./references/performance-tuning.md#concurrency-limits-tuning) + +--- + +**5.3 Execute Each Migration** + +**For Live Migrations:** + +**Create VirtualMachineInstanceMigration:** + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Parameters**: +```json +{ + "resource": "apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstanceMigration\nmetadata:\n name: migrate--\n namespace: \nspec:\n vmiName: " +} +``` + +**Monitor migration progress:** + +Poll using `resources_get` for VirtualMachineInstanceMigration, checking `.status.phase`: +- Pending → Scheduling → PreparingTarget → Running → Succeeded + +**For Cold Migrations:** + +Follow cold migration workflow from REBALANCE_MANUAL.md: +1. Stop VM using `vm_lifecycle` (action: stop) and wait for completion +2. Re-read VM using `resources_get` for fresh resourceVersion +3. Update VM nodeAffinity to target node +4. Start VM using `vm_lifecycle` (action: start) +5. Verify VM reached target node + +**Reference**: [REBALANCE_MANUAL.md - Cold Migration Workflow](./REBALANCE_MANUAL.md) + +--- + +**5.4 Report Progress Incrementally** + +After each migration completes: + +```markdown +## Automatic Rebalancing in Progress + +**Status:** 2/4 migrations complete + +✓ **database-01**: Migrated to worker-03 (live, 42s) +✓ **web-server-02**: Migrated to worker-04 (live, 38s) +⏳ **cache-vm**: Migrating to worker-04 (live, in progress - 15s elapsed) +⏸️ **api-gateway**: Pending (waiting for cache-vm to complete) + +**Estimated time remaining:** 2-3 minutes +``` + +Update after each completion/start. + +--- + +**5.5 Error Handling During Execution** + +**On first failure:** + +1. **Stop remaining migrations** (do not continue blindly) +2. **Report detailed status**: + +```markdown +## ⚠️ Rebalancing Paused - Migration Failed + +**Status:** 2/4 successful, 1 failed, 1 not attempted + +**Successful:** +- ✓ database-01: Migrated to worker-03 (live, 42s) +- ✓ web-server-02: Migrated to worker-04 (live, 38s) + +**Failed:** +- ❌ cache-vm: Migration timeout (VM memory write rate too high) + +**Not Attempted:** +- ⏸️ api-gateway: Skipped due to previous failure + +**Troubleshooting:** + +Migration timeout typically occurs when: +- VM has high memory write rate (database, caching workload) +- Network bandwidth insufficient for transfer speed +- VM memory size very large (>32GB) + +**Recommendations:** +1. Reduce workload on cache-vm and retry +2. Use cold migration for cache-vm (will have ~30-60s downtime) +3. Check network bandwidth availability + +**How would you like to proceed?** +- **retry** - Retry failed migration with same settings +- **retry cold** - Retry using cold migration instead +- **skip** - Skip cache-vm and continue with api-gateway +- **abort** - Stop rebalancing, leave cluster in current state +``` + +3. **Wait for user decision** before proceeding + +**Reference**: [SKILL.md - Common Error Handling](./SKILL.md#common-error-handling) + +--- + +### Step 6: Report Final Results + +**On Complete Success:** + +```markdown +## ✓ Automatic Rebalancing Complete + +**All migrations successful!** + +**Executed Migrations:** + +| VM | From | To | Type | Duration | Status | +|----|------|----|----|----------|--------| +| database-01 | worker-01 | worker-03 | live | 42s | ✓ Success | +| web-server-02 | worker-01 | worker-04 | live | 38s | ✓ Success | +| cache-vm | worker-02 | worker-04 | live | 35s | ✓ Success | +| api-gateway | worker-02 | worker-03 | live | 41s | ✓ Success | + +**Cluster State: Before → After** + +| Node | CPU Before | CPU After | Change | Memory Before | Memory After | Change | +|------|------------|-----------|--------|---------------|--------------|--------| +| worker-01 | 85% | 68% | -17% ✓ | 72% | 59% | -13% ✓ | +| worker-02 | 78% | 58% | -20% ✓ | 65% | 52% | -13% ✓ | +| worker-03 | 42% | 62% | +20% | 48% | 61% | +13% | +| worker-04 | 38% | 55% | +17% | 51% | 63% | +12% | + +**Improvements Achieved:** +- ✓ **CPU load balanced**: All nodes within 10% variance (was 22.1%) +- ✓ **Memory balanced**: All nodes within 8% variance (was 11.8%) +- ✓ **No node exceeding 70% utilization** (was 85% max) +- ✓ **Cluster capacity headroom**: 41% average (was 28%) +- ✓ **Total execution time**: 2 minutes 36 seconds + +**Next Steps:** +- Monitor cluster for 24-48 hours to ensure sustained improvement +- Consider removing nodeAffinity constraints (if added) for long-term flexibility +- Use `/vm-inventory` to verify all VMs are healthy + +Cluster is now optimally balanced. No further action needed. +``` + +**On Partial Success:** + +Display similar format but include: +- Which migrations succeeded +- Which failed (with error details and troubleshooting) +- Which were not attempted (and why) +- Current cluster state vs target +- Recommendations for completing rebalancing + +--- + +## Advanced Features + +### Intelligent Workload Analysis + +**Categorize VMs by workload type** (see [references/production-considerations.md](./references/production-considerations.md)): + +- **Database** (high dirty page rate) → Schedule during low-activity window, consider cold migration +- **Web servers** (low dirty page rate) → Safe for concurrent live migration +- **Caching** (very high dirty page rate) → Migrate during idle or use cold migration +- **Batch processing** → Migrate during job idle periods + +**Use workload characteristics** to optimize migration scheduling. + +### Network Bandwidth Awareness + +**Monitor network saturation:** + +**MCP Tool**: `nodes_stats_summary` (from openshift-virtualization) + +**Parameters**: +```json +{ + "name": "" +} +``` + +Review `.network.interfaces[].rxBytes` and `.network.interfaces[].txBytes`. + +**If saturation detected** (>80% utilization): +- Reduce concurrent migrations +- Set bandwidth limits per migration +- Suggest dedicated migration network + +**Reference**: [references/live-migration-best-practices.md#dedicated-migration-network](./references/live-migration-best-practices.md#dedicated-migration-network-production-best-practice) + +### Multi-Constraint Optimization + +**Consider additional constraints:** +- **Anti-affinity rules**: Don't co-locate VMs with same label +- **Topology spread**: Distribute VMs across zones/racks +- **Resource quotas**: Respect namespace limits +- **Custom scheduling**: Honor existing nodeSelector/tolerations + +**If conflicts detected**, explain to user and suggest resolution. + +--- + +## Human-in-the-Loop Requirements + +**CRITICAL: This mode requires user approval at multiple points.** + +### 1. Goal Clarification (if ambiguous) +- Present optimization options +- Wait for user to select goal +- Do NOT proceed with assumptions + +### 2. Plan Approval (MANDATORY) +- Display complete rebalance plan +- Show expected impact and improvements +- Offer modification options +- **REQUIRE explicit approval** ("approve", "yes", "execute") +- **NEVER execute without approval** + +### 3. Plan Modification (if requested) +- Allow user to skip migrations +- Allow changing target nodes +- Allow reordering +- Recalculate expected improvements +- Present updated plan for approval + +### 4. Error Handling (on failure) +- Stop execution immediately +- Report failure details +- Ask user how to proceed +- **Do NOT continue without user decision** + +### 5. Alternative Proposals (if AI detects better approach) +- Present alternative with rationale +- Show comparison vs original plan +- **User chooses** - execute user's preference + +**Rationale**: User maintains control over cluster changes while benefiting from AI's analytical capabilities. + +--- + +## Safety Considerations + +**Automatic mode involves batch operations with higher complexity:** + +**Risks:** +- ⚠️ Multiple concurrent migrations can saturate network +- ⚠️ Batch failures can compound +- ⚠️ Wrong optimization goal can worsen cluster state +- ⚠️ User must understand multi-VM impact + +**Mitigations:** +- ✓ Require explicit plan approval before execution +- ✓ Allow plan modification before execution +- ✓ Stop on first failure (no blind continuation) +- ✓ Respect cluster concurrency limits +- ✓ Validate all VMs before starting any migration +- ✓ Detailed progress reporting with errors +- ✓ Clear troubleshooting guidance + +**Best Practices:** +- Start with dev/test clusters to learn behavior +- Use during maintenance windows for production +- Monitor network bandwidth during migrations +- Have rollback plan for critical workloads +- Consider snapshots before risky migrations (use `/vm-snapshot-create`) + +**Reference**: [references/anti-patterns.md](./references/anti-patterns.md) + +--- + +## Example Usage + +### Example 1: Multi-Objective Optimization + +``` +User: "Rebalance VMs to optimize both CPU and memory" + +Agent: [Loads REBALANCE_AUTOMATIC.md] + [Gathers cluster state: VMs, nodes, metrics] + [Determines goal: Multi-objective (CPU + memory)] + [Analyzes and generates plan] + +## Automatic Rebalancing Plan + +**Optimization Goal:** Balance CPU and memory load across cluster + +[Displays plan with 5 migrations] +[Shows current state, proposed migrations, expected improvements] + +**How would you like to proceed?** +Options: approve | modify | explain <#> | cancel + +User: "approve" + +Agent: [Validates all VMs] + [Executes migrations with progress reporting] + +## ✓ Automatic Rebalancing Complete + +All 5 migrations successful. +- CPU variance: 22.1% → 4.2% (81% improvement) +- Memory variance: 11.8% → 7.3% (38% improvement) + +Cluster is now balanced. +``` + +### Example 2: With User Modification + +``` +User: "Balance cluster CPU load" + +Agent: [Generates plan with 4 migrations] + +## Automatic Rebalancing Plan + +[Shows plan including cold migration of app-vm-05] + +User: "modify - skip 4" + +Agent: "Migration #4 (app-vm-05) removed from plan." + +[Shows updated plan with 3 migrations] +[Recalculates expected improvements] + +**Ready to execute?** (approve / modify more / cancel) + +User: "approve" + +Agent: [Executes 3 migrations] + +## ✓ Rebalancing Complete + +3/3 migrations successful. +CPU variance: 22.1% → 6.8% (69% improvement) +``` + +### Example 3: AI Proposes Alternative + +``` +User: "Drain worker-02 for maintenance" + +Agent: [Generates plan to migrate all 5 VMs from worker-02] + +💡 **Alternative Approach Detected** + +I notice 2 VMs on worker-02 use RWO storage (cold migration required). +Current plan has ~80s total downtime (2 VMs × 40s each). + +**Alternative:** +Migrate VMs sequentially instead of concurrently to reduce risk: +- Same total time +- Lower network impact +- Easier to abort if issues + +Would you like to: accept alternative | keep original | explain more + +User: "accept alternative" + +Agent: [Updates plan to sequential execution] + [Presents updated plan for approval] +``` + +--- + +**Sources:** +- [Live Migration - KubeVirt User Guide](https://kubevirt.io/user-guide/compute/live_migration/) +- [Node Assignment - KubeVirt User Guide](https://kubevirt.io/user-guide/compute/node_assignment/) +- [Kubernetes Descheduler](https://github.com/kubernetes-sigs/descheduler) +- [Best Practices for Virtual Machine Deployments](https://learn.microsoft.com/en-us/azure/openshift/best-practices-openshift-virtualization) + +**Last Updated**: 2026-02-24 +**Status**: Production Ready diff --git a/rh-virt/skills/vm-rebalance/REBALANCE_MANUAL.md b/rh-virt/skills/vm-rebalance/REBALANCE_MANUAL.md new file mode 100644 index 00000000..f1e9d2c6 --- /dev/null +++ b/rh-virt/skills/vm-rebalance/REBALANCE_MANUAL.md @@ -0,0 +1,848 @@ +# Manual Rebalancing Strategy + +**Status**: ✅ FULLY IMPLEMENTED + +**Purpose**: Execute VM migrations where the user specifies which VM(s) to move and the target node(s). + +## When to Use Manual Mode + +Use this mode when the user specifies: +- Exact VM name(s) to migrate +- Target node(s) for placement +- Migration type (live or cold) + +**User Request Patterns:** +- "Move VM database-01 to worker-03" +- "Live migrate web-server from worker-01 to worker-05" +- "Cold migrate app-vm to worker-02" +- "Migrate VMs web-01, web-02, web-03 to worker-05" + +## Manual Rebalancing Workflow + +### Step 1: Gather Migration Parameters + +**Collect the following information from the user:** + +**Required Parameters:** +1. **VM Name(s)** - Name of virtual machine(s) to migrate + - Example: "database-01" or ["web-01", "web-02", "web-03"] + +2. **Source Namespace** - Namespace where VM(s) exist + - Example: "production", "vms", "dev-environment" + +3. **Target Node** (optional for live migration, required for cold migration) + - Example: "worker-03", "worker-node-1.example.com" + - For live migration: Can be omitted (scheduler chooses) + - For cold migration: Required (set via nodeAffinity) + +4. **Migration Type** - Choose migration strategy + - `live` - Zero downtime, VM stays running (requires RWX storage) + - `cold` - Brief downtime, stop → move → start (always works) + +**If user doesn't specify migration type, determine automatically:** +- Check VM's PVC access mode using `resources_get` +- If PVC has ReadWriteMany (RWX) → Suggest live migration +- If PVC has ReadWriteOnce (RWO) → Require cold migration +- If uncertain → Ask user to choose + +### Step 2: Validate Migration Feasibility + +**CRITICAL**: Before proceeding with migration execution, perform the **Common Validation Logic** defined in SKILL.md. + +**Reference**: See [SKILL.md - Common Validation Logic](./SKILL.md#common-validation-logic) for complete validation steps. + +**The common validation performs these checks** (defined in SKILL.md): +1. **Validation 1**: Verify VM Exists +2. **Validation 2**: Check Current VM Location +3. **Validation 3**: Validate Storage Compatibility (RWX vs RWO) +4. **Validation 4**: Verify Target Node Exists + +**Only proceed to Step 3 after all validations pass.** + +### Step 3: Present Rebalance Plan for Confirmation + +**After validation, present the rebalance plan to the user:** + +**For Live Migration:** + +```markdown +## VM Rebalance Plan + +**Please review and confirm the migration:** + +| Parameter | Value | Details | +|-----------|-------|---------| +| VM Name | `` | [from user input] | +| Namespace | `` | [from user input] | +| Current Node | `` | [detected from VMI status] | +| Target Node | `` | [from user input or "Scheduler decides"] | +| Migration Type | `live` | Zero downtime, VM stays running | +| Storage Type | `RWX (ReadWriteMany)` | Live migration supported | +| Current Status | `Running` | [from VMI phase] | + +**Migration Strategy: Live Migration** + +**What will happen:** +1. Create VirtualMachineInstanceMigration resource +2. KubeVirt will: + - Create new virt-launcher pod on target node + - Transfer VM memory and state (live) + - Switch network traffic to new pod + - Terminate old virt-launcher pod +3. VM continues running throughout (brief network pause <1s) + +**Impact:** +- ✓ Zero downtime (VM stays running) +- ✓ Applications remain accessible +- ⚠️ Brief network pause during cutover (<1 second) +- ⚠️ Requires network bandwidth for memory transfer +- ⚠️ Migration duration depends on VM memory size + +**Estimated Duration:** ~30-60 seconds for typical VM (depends on memory size) + +**Reference**: See [references/live-migration-best-practices.md](./references/live-migration-best-practices.md) for configuration parameters, storage requirements, and network optimization + +**Rollback:** If migration fails, VM continues running on current node + +Confirm this migration or tell me what to change: +- yes - Proceed with migration +- no - Cancel migration +- modify - Change parameters +``` + +**For Cold Migration:** + +```markdown +## VM Rebalance Plan + +| Parameter | Value | Details | +|-----------|-------|---------| +| VM Name | `` | [from user input] | +| Namespace | `` | [from user input] | +| Current Node | `` | [detected] | +| Target Node | `` | [from user input] | +| Migration Type | `cold` | VM will stop and restart | +| Storage Type | `RWO (ReadWriteOnce)` | Live migration not supported | +| Current Status | `Running` | Will be stopped then restarted | + +**Migration Strategy: Cold Migration** + +**What will happen:** +1. Update VM spec with **required nodeAffinity** (forces placement on target node) +2. Stop VM using vm_lifecycle +3. VM pod terminates on current node +4. Start VM using vm_lifecycle +5. Kubernetes scheduler places VM on target node (due to nodeAffinity) +6. **After migration:** You'll choose whether to remove the affinity for long-term flexibility + +**Impact:** +- ⚠️ **Downtime**: ~30-60 seconds while VM restarts +- ⚠️ Applications will be interrupted +- ⚠️ Network connections will be dropped +- ✓ Works with any storage type (RWO or RWX) +- ✓ Less resource-intensive than live migration +- ℹ️ **Long-term**: After migration, you can remove node affinity to allow flexible rescheduling if target node fails + +**Estimated Downtime:** ~30-60 seconds + +**Rollback:** Can start VM again if migration fails + +**Note on Node Affinity:** +We use nodeAffinity (not nodeSelector) for better long-term resilience. After migration, you'll have the option to remove it, allowing the VM to reschedule to other nodes if fails in the future. + +**Reference**: See [references/production-considerations.md](./references/production-considerations.md) for workload categorization and downtime planning strategies + +Confirm this migration or tell me what to change: +- yes - Proceed with cold migration +- no - Cancel +- modify - Change parameters +``` + +**WAIT for user confirmation before proceeding to Step 4.** Do NOT continue until user explicitly confirms with "yes". + +### Step 4a: Execute Live Migration (if migration type = live) + +**ONLY PROCEED AFTER USER CONFIRMATION IN STEP 3.** + +**Create a VirtualMachineInstanceMigration resource to trigger live migration:** + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Parameters**: +- `resource`: (JSON object as string) - REQUIRED + - Complete VirtualMachineInstanceMigration resource definition as a JSON-formatted string + - Must follow KubeVirt API specification + +**Resource Structure** (for live migration): + +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstanceMigration", + "metadata": { + "name": "migration--", + "namespace": "" + }, + "spec": { + "vmiName": "" + } +} +``` + +**Example tool invocation:** +```json +{ + "resource": "{\"apiVersion\":\"kubevirt.io/v1\",\"kind\":\"VirtualMachineInstanceMigration\",\"metadata\":{\"name\":\"migration-database-01-20260223\",\"namespace\":\"production\"},\"spec\":{\"vmiName\":\"database-01\"}}" +} +``` + +**Note**: The `resource` parameter accepts the resource definition as a JSON-formatted string. The MCP tool will parse and apply this resource to the cluster. + +**Note on Target Node Selection:** +- KubeVirt scheduler automatically selects target node +- To influence target node, update VM's nodeAffinity BEFORE creating migration +- For manual target node selection, combine with nodeAffinity update first + +**Expected Output**: VirtualMachineInstanceMigration resource created successfully + +**Error Handling:** +- If creation fails → Check RBAC permissions, report error to user +- If VMI not found → Verify VM is running, report error +- If VMI not migratable → Check storage access mode, suggest cold migration + +#### Monitor Migration Progress + +**After creating migration, monitor progress:** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters** (to monitor migration status): +- `apiVersion`: "kubevirt.io/v1" - REQUIRED +- `kind`: "VirtualMachineInstanceMigration" - REQUIRED +- `name`: "migration--" - REQUIRED +- `namespace`: "" - REQUIRED + +**Poll every 5-10 seconds until migration completes or fails.** + +**Timeout**: Stop polling after 10 minutes (600 seconds) and report timeout error to user. Most migrations complete within 1-5 minutes depending on VM memory size. + +**Reference**: See [references/performance-tuning.md](./references/performance-tuning.md) for timeout configuration and migration optimization strategies + +**Expected Output**: VMIM resource with status field + +**Extract Information:** +- `status.phase` - Migration phase: + - `Scheduling` - Finding target node + - `PreparingTarget` - Setting up target pod + - `TargetReady` - Target pod ready + - `Running` - Transferring memory + - `Succeeded` - Migration completed + - `Failed` - Migration failed +- `status.migrationState.completed` - Boolean, true when done +- `status.migrationState.targetNode` - Destination node +- `status.migrationState.sourceNode` - Origin node +- `status.migrationState.startTimestamp` - When migration began +- `status.migrationState.endTimestamp` - When migration completed + +**When status.phase = "Succeeded":** +- Migration completed successfully +- Proceed to Step 5 (Report Results) + +**When status.phase = "Failed":** +- Extract failure reason from status +- Consult troubleshooting documentation (see Step 5 failure handling) +- Report detailed error to user + +### Step 4b: Execute Cold Migration (if migration type = cold) + +**ONLY PROCEED AFTER USER CONFIRMATION IN STEP 3.** + +**Cold migration workflow: Stop VM → Re-read VM → Update node placement → Start VM** + +#### Sub-step 4b.1: Stop the VM + +**MCP Tool**: `vm_lifecycle` (from openshift-virtualization) + +**Parameters**: +- `namespace`: "" - REQUIRED +- `name`: "" - REQUIRED +- `action`: "stop" - REQUIRED + +**Expected Output**: VM stopped successfully, VMI terminates + +**Wait for VM to fully stop:** +1. Wait 10 seconds +2. Check VM status using `resources_get` (VirtualMachine) +3. If `status.printableStatus` is not "Stopped", wait another 10 seconds and check again +4. Repeat until VM is fully stopped + +**Error Handling:** +- If stop fails → Report error, check if VM is already stopped +- If VM stuck in Terminating after 60 seconds → Report to user + +#### Sub-step 4b.2: Re-read VM for Fresh ResourceVersion + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +- `apiVersion`: "kubevirt.io/v1" - REQUIRED +- `kind`: "VirtualMachine" - REQUIRED +- `name`: "" - REQUIRED +- `namespace`: "" - REQUIRED + +**Why**: `vm_lifecycle` modified the VM. Re-reading gets fresh resourceVersion to prevent conflicts when updating nodeAffinity. + +**Use this fresh VM spec for nodeAffinity update in next step.** + +#### Sub-step 4b.3: Update VM nodeAffinity + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Parameters**: +- `resource`: (JSON object as string) - REQUIRED + - Complete VirtualMachine resource from Sub-step 4b.2 with updated nodeAffinity + +**Resource Modification** (add required nodeAffinity to VM spec): + +Preserve all existing VM fields and only modify the affinity section. + +**Critical Affinity Structure:** +```json +{ + "spec": { + "template": { + "spec": { + "affinity": { + "nodeAffinity": { + "requiredDuringSchedulingIgnoredDuringExecution": { + "nodeSelectorTerms": [ + { + "matchExpressions": [ + { + "key": "kubernetes.io/hostname", + "operator": "In", + "values": [""] + } + ] + } + ] + } + } + } + } + } + } +} +``` + +**Why nodeAffinity instead of nodeSelector?** +- Provides flexibility for long-term resilience +- If target node fails later, user can remove affinity to allow rescheduling +- More powerful than nodeSelector (supports multiple nodes, preferences) + +**Expected Output**: VirtualMachine resource updated successfully + +**Error Handling:** +- If update fails → Check RBAC permissions, explain to user with exact error +- If VM not found → Report error with VM name and namespace +- If conflicts with existing affinity → Ask user: "VM already has node affinity rules. Should I replace them with new affinity to ?" + +#### Sub-step 4b.4: Start the VM + +**MCP Tool**: `vm_lifecycle` (from openshift-virtualization) + +**Parameters**: +- `namespace`: "" - REQUIRED +- `name`: "" - REQUIRED +- `action`: "start" - REQUIRED + - Example: `"start"` (power on the VM) + +**Expected Output**: VM starts successfully + +**After starting, verify VM scheduled on target node:** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters** (get VMI to verify node placement): +- `apiVersion`: "kubevirt.io/v1" - REQUIRED +- `kind`: "VirtualMachineInstance" - REQUIRED +- `name`: "" - REQUIRED +- `namespace`: "" - REQUIRED + +**Expected Output**: VMI resource with node placement + +**Extract Information:** +- `status.nodeName` - Should match target node +- `status.phase` - Should be "Running" + +**Verification:** +- If `status.nodeName` matches target node → Success, proceed to Step 4b.5 +- If `status.nodeName` does NOT match target node → Migration failed, report error with explanation + +**Error Handling with Clear Explanations:** +- If VM fails to start → Explain: "VM failed to start on . This usually means the node doesn't have enough resources (CPU/memory) or has scheduling constraints preventing this VM." + - Consult [scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) + - Suggest: Check node capacity, verify node is Ready, check for taints +- If VM scheduled on wrong node → Explain: "VM started but Kubernetes placed it on instead of . This means couldn't accommodate the VM due to resource constraints, taints, or other scheduling rules." + - Report actual vs expected node + - Ask user if they want to retry with different target node +- If VM stuck in Scheduling phase → Explain: "VM cannot be scheduled. The target node likely lacks sufficient CPU/memory resources, or has taints that prevent scheduling." + - Show node constraints + - Suggest checking node details using `resources_get` with kind="Node" + +#### Sub-step 4b.5: Post-Migration Cleanup (Optional - Long-term Resilience) + +**After successful cold migration, ask user about removing node affinity for long-term flexibility.** + +**Present cleanup option to user:** + +```markdown +## ✓ Cold Migration Successful - VM Running on Target Node + +**VM Details:** +- **Name**: `` +- **Namespace**: `` +- **Current Node**: `` ✓ + +**Important: Long-term Resilience** + +The VM now has a **required node affinity** to ``. This means: +- ✓ VM will stay on `` (as you requested) +- ⚠️ If `` fails or needs maintenance later, the VM **cannot reschedule** to other nodes + +**Recommendation:** Remove the node affinity to allow flexible rescheduling in the future. + +Would you like to remove the node affinity now? +- **yes** - Remove affinity (VM can reschedule to any healthy node if fails) +- **no** - Keep affinity (VM stays pinned to permanently) +- **later** - Keep for now, I'll remove it manually when needed +``` + +**WAIT for user decision.** + +**If user says "yes" (remove affinity):** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters** (get current VM spec): +- `apiVersion`: "kubevirt.io/v1" - REQUIRED +- `kind`: "VirtualMachine" - REQUIRED +- `name`: "" - REQUIRED +- `namespace`: "" - REQUIRED + +**Expected Output**: Full VirtualMachine resource with current affinity + +**Modify the VM spec to remove nodeAffinity:** + +Extract the full VM resource, then remove `spec.template.spec.affinity.nodeAffinity` (or set to null/empty). + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Parameters**: +- `resource`: (JSON object as string) - REQUIRED + - Complete VirtualMachine resource with affinity removed as a JSON-formatted string + +**Important**: +- Preserve all other VM spec fields +- Only remove the affinity section +- VM continues running on current node (doesn't trigger immediate reschedule) +- Future restarts will allow flexible scheduling + +**Expected Output**: VirtualMachine resource updated successfully + +**Error Handling:** +- If update fails → Explain: "Failed to remove node affinity. RBAC permissions issue or API error. VM will remain pinned to . You can remove manually using `resources_create_or_update` with the VM spec after removing the affinity field." + +**If user says "no" or "later" (keep affinity):** +- Report: "Node affinity kept. VM will remain on . To remove manually later, use `resources_create_or_update` with the VM spec after removing the affinity field." +- Proceed to Step 5 + +### Step 5: Report Migration Results + +**On Success (Live Migration):** + +```markdown +## ✓ Live Migration Successful + +**VM Details:** +- **Name**: `` +- **Namespace**: `` +- **Migration Type**: Live (zero downtime) + +**Migration Summary:** +- **Source Node**: `` +- **Target Node**: `` +- **Duration**: seconds +- **Status**: Succeeded + +**Impact:** +- ✓ VM remained running throughout migration +- ✓ Applications stayed accessible +- ✓ Brief network pause during cutover (<1 second) + +**Current VM Status:** +- **Running on**: `` +- **Phase**: Running +- **Ready**: True + +### Next Steps + +**Verify application health:** +Use `/vm-inventory` to check VM status + +**View migration details:** +Use `resources_get` with kind="VirtualMachineInstanceMigration" and name="migration--" + +**Cleanup:** +The VirtualMachineInstanceMigration resource can be deleted if no longer needed. +``` + +**On Success (Cold Migration) - If affinity was REMOVED:** + +```markdown +## ✓ Cold Migration Successful + +**VM Details:** +- **Name**: `` +- **Namespace**: `` +- **Migration Type**: Cold (with downtime) + +**Migration Summary:** +- **Source Node**: `` +- **Target Node**: `` +- **Downtime**: ~ seconds +- **Status**: Succeeded + +**Steps Completed:** +1. ✓ Updated VM with required nodeAffinity to `` +2. ✓ Stopped VM on `` +3. ✓ Started VM on `` +4. ✓ Verified VM running on target node +5. ✓ Removed node affinity for flexible rescheduling + +**Current VM Status:** +- **Running on**: `` +- **Phase**: Running +- **Ready**: True +- **Node Affinity**: None (can reschedule to any node if needed) + +**Long-term Resilience:** +✓ VM can now reschedule to other nodes if `` fails or needs maintenance. + +### Next Steps + +**Verify application health:** +Use `/vm-inventory` to check VM status + +**Test application connectivity:** +VM has restarted, verify services are healthy. +``` + +**On Success (Cold Migration) - If affinity was KEPT:** + +```markdown +## ✓ Cold Migration Successful + +**VM Details:** +- **Name**: `` +- **Namespace**: `` +- **Migration Type**: Cold (with downtime) + +**Migration Summary:** +- **Source Node**: `` +- **Target Node**: `` +- **Downtime**: ~ seconds +- **Status**: Succeeded + +**Steps Completed:** +1. ✓ Updated VM with required nodeAffinity to `` +2. ✓ Stopped VM on `` +3. ✓ Started VM on `` +4. ✓ Verified VM running on target node +5. ℹ️ Kept node affinity (as requested) + +**Current VM Status:** +- **Running on**: `` +- **Phase**: Running +- **Ready**: True +- **Node Affinity**: Required on `` (VM will stay on this node) + +**Important:** +⚠️ VM is pinned to ``. If this node fails, the VM cannot reschedule to other nodes. + +**To remove affinity later:** +Use `resources_create_or_update` with the VM spec after removing the `spec.template.spec.affinity` field + +### Next Steps + +**Verify application health:** +Use `/vm-inventory` to check VM status + +**Test application connectivity:** +VM has restarted, verify services are healthy. +``` + +**On Failure (with Troubleshooting):** + +**OPTIONAL**: If migration fails, consult documentation for common failure scenarios. + +**Document Consultation** (OPTIONAL - when migration fails): +1. **Action**: Read relevant troubleshooting guides to understand VM migration failure scenarios: + - [scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) - For ErrorUnschedulable, node taints, resource constraints + - [lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) - For VM start/stop failures during cold migration + - [storage-errors.md](../../docs/troubleshooting/storage-errors.md) - For PVC access mode issues affecting live migration +2. **Output to user**: "I consulted [scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) to understand potential causes for the migration failure." + +**When to consult**: +- Live migration fails (check storage, network, resource constraints) +- Cold migration fails during start (likely scheduling or resource issues) +- VM stuck in ErrorUnschedulable state +- Unexpected error messages from migration tools + +**When NOT to consult**: +- RBAC permission errors (clear cause) +- VM not found errors (clear cause) +- PVC access mode incompatibility (already validated in Step 2) + +```markdown +## ❌ Migration Failed + +**Error**: + +**VM Details:** +- **Name**: `` +- **Namespace**: `` +- **Migration Type**: +- **Source Node**: `` +- **Target Node**: `` + +**Common Causes:** + +**For Live Migration Failures:** +- **Insufficient resources** - Target node lacks CPU/memory for VM +- **Network bandwidth** - Slow network prevents migration convergence +- **PVC access mode** - Storage is not ReadWriteMany (RWX) +- **Timeout exceeded** - Migration took longer than allowed (default 150s/GiB) +- **High memory write rate** - VM writes to memory faster than transfer rate + +**For Cold Migration Failures:** +- **Node not schedulable** - Target node is cordoned, has taints, or lacks resources +- **NodeSelector conflict** - VM has other scheduling constraints preventing placement +- **VM failed to start** - Check scheduling errors on target node +- **Storage issues** - PVC not accessible from target node + +**Troubleshooting Steps:** + +1. **Check node availability:** + Use `resources_list` with kind="Node" to verify target node is Ready and schedulable + +2. **Check VM events:** + Use `resources_get` with kind="VirtualMachine" to see VM status and conditions + +3. **Check migration status** (for live migration): + Use `resources_list` with kind="VirtualMachineInstanceMigration" to see migration resource status + +4. **Verify storage access:** + Use `resources_get` with kind="PersistentVolumeClaim" to check PVC access modes + +5. **Check resource capacity:** + Use `nodes_top` or `resources_get` with kind="Node" to verify available CPU/memory + +**Recommended Actions:** + +- **If timeout**: Retry with slower workload or enable auto-converge +- **If resources**: Choose different target node with more capacity +- **If storage**: Use cold migration instead (works with RWO) +- **If node unavailable**: Verify node is Ready and untainted + +**Rollback (if migration succeeded but VM has issues on target node):** + +To move VM back to original node, perform reverse migration: +- **For live migration**: Create new migration from current node → original node +- **For cold migration**: Update nodeAffinity to original node, restart VM +- Use same migration type as original migration + +Example: "Migrate from back to " + +Would you like help troubleshooting this error? +``` + +## Batch VM Migration + +**For migrating multiple VMs in a single operation:** + +**User Request:** "Migrate VMs web-01, web-02, web-03 to worker-05" + +**Safety Limits:** +- ⚠️ **Batches >5 VMs**: Warn user about potential network saturation and suggest staged execution +- ⚠️ **Batches >10 VMs**: Strongly recommend splitting into multiple batches (e.g., 5 VMs at a time) +- ⚠️ **Multiple cold migrations**: Warn about cumulative downtime impact + +**Reference**: See [references/anti-patterns.md](./references/anti-patterns.md) for common mistakes when performing batch migrations and network saturation issues + +**Workflow:** +1. Validate all VMs exist and are migratable +2. Determine migration type for each (based on storage) +3. **Check batch size** - If >5 VMs, warn user and offer to stage migrations +4. Present batch rebalance plan: + ```markdown + ## Batch Rebalance Plan + + Migrating 3 VMs to worker-05: + + | VM Name | Current Node | Migration Type | Impact | + |---------|--------------|----------------|--------| + | web-01 | worker-01 | live | <1s pause | + | web-02 | worker-02 | live | <1s pause | + | web-03 | worker-03 | cold | ~40s downtime | + + Total: 2 live migrations + 1 cold migration + + Proceed with batch migration? + ``` +4. Execute migrations sequentially (avoid saturating network) +5. Report results for each VM + +## Example Usage + +### Example 1: Successful Live Migration + +``` +User: "Live migrate VM database-01 from production namespace to worker-03" + +Agent: [Validates VM exists, checks storage is RWX, verifies target node] + +## VM Rebalance Plan + +Please review and confirm the migration: + +| Parameter | Value | Details | +|-----------|-------|---------| +| VM Name | `database-01` | from user input | +| Namespace | `production` | from user input | +| Current Node | `worker-01` | detected from VMI status | +| Target Node | `worker-03` | from user input | +| Migration Type | `live` | Zero downtime, VM stays running | +| Storage Type | `RWX (ReadWriteMany)` | Live migration supported | + +**What will happen:** +1. Create VirtualMachineInstanceMigration resource +2. VM memory and state transferred live to worker-03 +3. Brief network pause during cutover (<1 second) + +**Impact:** Zero downtime, brief network pause + +Confirm this migration? + +User: "yes" + +Agent: [Creates VirtualMachineInstanceMigration resource] + [Monitors migration status] + +## ✓ Live Migration Successful + +**VM Details:** +- Name: `database-01` +- Namespace: `production` +- Migration Type: Live (zero downtime) + +**Migration Summary:** +- Source Node: `worker-01` +- Target Node: `worker-03` +- Duration: 45 seconds +- Status: Succeeded + +VM is now running on `worker-03`. +``` + +### Example 2: Successful Cold Migration + +``` +User: "Move VM web-server to worker-05 using cold migration" + +Agent: [Validates VM exists, checks current node, verifies target node] + +## VM Rebalance Plan + +| Parameter | Value | Details | +|-----------|-------|---------| +| VM Name | `web-server` | from user input | +| Namespace | `vms` | detected/default | +| Current Node | `worker-02` | detected | +| Target Node | `worker-05` | from user input | +| Migration Type | `cold` | VM will stop and restart | +| Storage Type | `RWO (ReadWriteOnce)` | Live migration not supported | + +**What will happen:** +1. Update VM nodeAffinity to target worker-05 +2. Stop VM (applications will be interrupted) +3. Start VM (Kubernetes places on worker-05) + +**Impact:** Downtime ~30-60 seconds + +Confirm this migration? + +User: "yes" + +Agent: [Updates VM nodeAffinity] + [Calls vm_lifecycle stop] + [Calls vm_lifecycle start] + [Verifies VM on target node] + +## ✓ Cold Migration Successful + +**VM Details:** +- Name: `web-server` +- Namespace: `vms` +- Migration Type: Cold (with downtime) + +**Migration Summary:** +- Source Node: `worker-02` +- Target Node: `worker-05` +- Downtime: ~40 seconds +- Status: Succeeded + +VM is now running on `worker-05`. Verify application health. +``` + +### Example 3: Migration Failure - Storage Incompatible + +``` +User: "Live migrate app-vm to worker-03" + +Agent: [Validates VM] + [Checks PVC access mode - discovers RWO] + +## ❌ Live Migration Not Supported + +**VM Details:** +- Name: `app-vm` +- Namespace: `default` +- Storage Type: `RWO (ReadWriteOnce)` + +**Issue:** Live migration requires ReadWriteMany (RWX) storage. This VM uses ReadWriteOnce (RWO), which only allows attachment to one node at a time. + +**Recommended Solution:** Use cold migration instead. + +Would you like to proceed with cold migration? (Brief downtime ~30-60 seconds) + +User: "yes, do cold migration" + +Agent: [Switches to cold migration workflow] + [Continues with cold migration process...] +``` + +--- + +**Reference Documentation:** + +**Internal Skill Documentation**: +- [references/live-migration-best-practices.md](./references/live-migration-best-practices.md) - Configuration, requirements, dedicated networks +- [references/performance-tuning.md](./references/performance-tuning.md) - Right-sizing, overcommit, bandwidth tuning +- [references/anti-patterns.md](./references/anti-patterns.md) - Common mistakes to avoid +- [references/production-considerations.md](./references/production-considerations.md) - HA strategies, capacity planning + +**Official KubeVirt Documentation**: +- [Live Migration - KubeVirt User Guide](https://kubevirt.io/user-guide/compute/live_migration/) +- [Node Assignment - KubeVirt User Guide](https://kubevirt.io/user-guide/compute/node_assignment/) +- [VirtualMachineInstanceMigration API](https://kubevirt.io/api-reference/main/definitions.html#_v1_virtualmachineinstancemigration) diff --git a/rh-virt/skills/vm-rebalance/SKILL.md b/rh-virt/skills/vm-rebalance/SKILL.md new file mode 100644 index 00000000..ed8393e6 --- /dev/null +++ b/rh-virt/skills/vm-rebalance/SKILL.md @@ -0,0 +1,580 @@ +--- +name: vm-rebalance +description: | + Orchestrate VM migrations across cluster nodes for load balancing, maintenance, and resource optimization. + + Use when: + - "Move VM database-01 to worker-03" + - "Rebalance VMs to optimize CPU load" + - "Drain worker-02 for maintenance" + - "Automatically rebalance the cluster" + + Supports Manual (user-driven) and Automatic (AI-driven) modes. + + NOT for creating VMs (use vm-create) or lifecycle only (use vm-lifecycle-manager). + +model: inherit +color: yellow +--- + +# /vm-rebalance Skill + +Orchestrate virtual machine migrations across OpenShift cluster nodes for load balancing, maintenance, and resource optimization. Supports manual and automatic rebalancing modes with both live migration (zero downtime) and cold migration (brief downtime) strategies. + +**Implementation**: Uses KubeVirt's VirtualMachineInstanceMigration API for live migrations and node affinity for cold migrations, following official KubeVirt patterns. + +## Prerequisites + +**Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) + +**Required MCP Tools**: +- `resources_list` - List VMs and nodes +- `resources_get` - Get VM and node details +- `resources_create_or_update` - Create migrations and update VM specs +- `vm_lifecycle` - Start/stop VMs for cold migration +- `nodes_top` - Monitor node resource usage +- `pods_top` - Monitor VM resource consumption + +**Required Environment Variables**: +- `KUBECONFIG` - Path to Kubernetes configuration file + +**Required Cluster Setup**: +- OpenShift cluster (>= 4.17) +- OpenShift Virtualization operator installed +- ServiceAccount with permissions: get/list/update for VMs, create for VirtualMachineInstanceMigration +- For live migration: RWX storage and sufficient network bandwidth + +### Prerequisite Verification + +**Before executing:** + +1. **Check MCP Server** - Verify `openshift-virtualization` exists in `.mcp.json` +2. **Check Environment** - Verify `KUBECONFIG` is set (check presence only, never expose value) +3. **Check Capability** (for live migration): + - Use `resources_get` to check PVC access mode is ReadWriteMany (RWX) + - Verify VM has running VirtualMachineInstance + +**If Prerequisites Fail:** + +``` +❌ Cannot execute vm-rebalance: MCP server 'openshift-virtualization' is not available + +📋 Setup Instructions: +1. Add openshift-virtualization to .mcp.json +2. Set KUBECONFIG environment variable +3. Ensure ServiceAccount has required permissions +4. Restart Claude Code to reload MCP servers + +🔗 Documentation: https://github.com/openshift/openshift-mcp-server + +Options: "setup" | "skip" | "abort" +``` + +⚠️ **SECURITY**: Never display KUBECONFIG path or credential values in output. + +## When to Use This Skill + +**Trigger when:** +- User explicitly invokes `/vm-rebalance` command +- User requests moving specific VM(s) to specific node(s) +- User wants to drain a node for maintenance +- User requests load balancing or resource optimization +- User wants to redistribute VMs across cluster + +**User phrases:** +- "Move VM database-01 to worker-03" +- "Live migrate web-server to worker-05" +- "Drain worker-02 for maintenance" +- "Balance CPU load across nodes" +- "Automatically rebalance the cluster" + +**Do NOT use when:** +- Creating VMs → Use `/vm-create` +- Start/stop only → Use `/vm-lifecycle-manager` +- Cloning VMs → Use `/vm-clone` +- Deleting VMs → Use `/vm-delete` + +## Workflow + +### Step 1: Determine Rebalancing Mode + +**Analyze user request:** + +**Manual Mode Indicators:** +- User specifies VM name(s) and target node(s) +- Examples: "Move VM database-01 to worker-03", "Migrate web-server to worker-05" + +**Automatic Mode Indicators:** +- User requests AI-driven rebalancing with high-level goal +- Examples: "Rebalance VMs based on CPU", "Automatically optimize cluster load" + +### Step 2: Load Strategy File and Execute + +**For Manual Mode:** + +**Document Consultation** (REQUIRED - Execute FIRST): +1. Read [REBALANCE_MANUAL.md](./REBALANCE_MANUAL.md) using Read tool +2. Output: "I consulted [REBALANCE_MANUAL.md](rh-virt/skills/vm-rebalance/REBALANCE_MANUAL.md) to understand the manual migration workflow." + +**Then execute**: Follow workflow in REBALANCE_MANUAL.md + +--- + +**For Automatic Mode:** + +**Document Consultation** (REQUIRED - Execute FIRST): +1. Read [REBALANCE_AUTOMATIC.md](./REBALANCE_AUTOMATIC.md) using Read tool +2. Output: "I consulted [REBALANCE_AUTOMATIC.md](rh-virt/skills/vm-rebalance/REBALANCE_AUTOMATIC.md) to understand the automatic rebalancing workflow." + +**Then execute**: Follow workflow in REBALANCE_AUTOMATIC.md + +## Common Validation Logic + +**This validation is shared by ALL migration strategies.** + +Before executing any VM migration, perform these validations for each VM: + +### Validation 1: Verify VM Exists + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +- `apiVersion`: "kubevirt.io/v1" +- `kind`: "VirtualMachine" +- `name`: "" +- `namespace`: "" + +**Extract**: +- `spec.template.spec.volumes[].persistentVolumeClaim.claimName` - PVC names +- `status.ready` - VM ready state + +**Error Handling**: +- VM not found → "VM '' does not exist in namespace ''. Use '/vm-inventory' to see available VMs." +- Namespace not found → "Namespace '' does not exist. Verify namespace name." +- Permission denied → "Insufficient permissions. ServiceAccount needs 'get' permission for VirtualMachine resources." + +### Validation 2: Check Current VM Location + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters** (only if VM is running): +- `apiVersion`: "kubevirt.io/v1" +- `kind`: "VirtualMachineInstance" +- `name`: "" +- `namespace`: "" + +**Extract**: +- `status.nodeName` - Current node +- `status.phase` - Current phase + +**Validation**: +- If already on target node → "VM '' is already on ''. No migration needed." + +### Validation 3: Validate Storage Compatibility + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +- `apiVersion`: "v1" +- `kind`: "PersistentVolumeClaim" +- `name`: "" +- `namespace`: "" + +**Extract**: +- `spec.accessModes` - Contains "ReadWriteMany" (RWX) → Live migration supported +- Only "ReadWriteOnce" (RWO) → Live migration NOT supported + +**Error Handling for live migration**: +- If RWO → "Cannot perform live migration. VM uses ReadWriteOnce (RWO) storage. Use cold migration instead (brief downtime ~30-60s)." +- If VM not running → "Cannot perform live migration. VM is not running. Use cold migration or start VM first." + +**Reference**: See [references/live-migration-best-practices.md](./references/live-migration-best-practices.md) for complete storage requirements. + +### Validation 4: Verify Target Node Exists + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +- `apiVersion`: "v1" +- `kind`: "Node" + +**Validation**: +- Verify target node exists in list +- Check `status.conditions[]` shows Ready=True +- Check node is schedulable (not cordoned) + +**Error Handling**: +- Not found → "Target node '' does not exist." +- Not Ready → "Target node '' is not Ready. Choose different target." +- Cordoned → "Target node '' is cordoned (SchedulingDisabled). Choose different target or uncordon node." + +**Reference**: See [../../docs/troubleshooting/scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) for scheduling issues. + +## Node Selection for Automatic Rebalancing + +**Applies to Automatic Mode only. Manual Mode uses user-specified nodes.** + +### Suitable Node Criteria + +**Use `resources_list` with apiVersion="v1", kind="Node"** + +Filter nodes where ALL conditions are true: +1. `metadata.labels["kubevirt.io/schedulable"] == "true"` (official KubeVirt marker) +2. `status.capacity["devices.kubevirt.io/kvm"]` > "0" (KVM devices available) +3. No `node-role.kubernetes.io/control-plane` or `node-role.kubernetes.io/master` label (worker nodes only) + +**If no nodes found**: "No suitable nodes found. Check OpenShift Virtualization operator status and node hardware virtualization support." + +**Note**: Ignore cluster-specific custom taints (e.g., `node-purpose=virtualization`). Always use official KubeVirt labels. + +## Common Migration Types + +### Live Migration +- **Zero downtime** - VM stays running +- **Brief network pause** (<1 second) during cutover +- **Requires**: ReadWriteMany (RWX) storage +- **Process**: Memory transferred while VM runs +- **Use when**: Minimal disruption required + +**Reference**: [references/live-migration-best-practices.md](./references/live-migration-best-practices.md) + +### Cold Migration +- **Brief downtime** (~30-60 seconds) +- **Works with any storage** (RWO or RWX) +- **Process**: Stop VM → Update placement → Start on target +- **Use when**: Live migration not supported or acceptable downtime + +**Reference**: See REBALANCE_MANUAL.md for cold migration workflow + +## Common Plan Visualization + +**All rebalancing strategies MUST use this standardized format when presenting plans to users.** + +This ensures consistent, clear communication across Manual and Automatic modes. + +### Information Relevance Principle + +**Show only what matters to the user's decision:** + +- **Include**: Deviations from defaults, user-specified criteria, or non-obvious context +- **Exclude**: Standard procedures, default settings, or information already visible in tables + +**Examples**: +- ✅ Show node criteria IF user specified constraints ("only nodes with SSD") +- ❌ Don't show default criteria when using standard behavior +- ✅ Show analysis IF it reveals critical context (e.g., all VMs on one node) +- ❌ Don't repeat what's visible in the tables + +**Apply this principle throughout the plan**: Only explain what's non-obvious, non-standard, or specifically requested. + +### Standard Plan Format + +**Present rebalancing plans using these two tables:** + +#### Table 1: VM Rebalance Plan + +Shows what will happen to each VM: + +```markdown +## 📋 VM Rebalance Plan + +| VM | Instance Type | Current Node | → | New Node | Type | Downtime | Notes | +|----|---------------|--------------|---|----------|------|----------|-------| +| vm-name-1 | u1.xlarge | worker-01 | → | worker-03 | Live | <1s pause | ContainerDisk - easy migration | +| vm-name-2 | u1.2xmedium | worker-01 | → | worker-02 | Cold | ~40s | RWO storage | +| vm-name-3 | u1.medium | worker-02 | - | *stays* | - | - | Already balanced | +``` + +**Column Definitions:** +- **VM**: VM name +- **Instance Type**: Instance type (e.g., u1.xlarge, u1.2xmedium) +- **Current Node**: Node where VM is currently running +- **→**: Visual indicator of movement +- **New Node**: Target node (or "*stays*" if no migration needed) +- **Type**: Migration type (Live, Cold, or "-" for no migration) +- **Downtime**: Expected downtime (<1s pause for live, ~30-60s for cold, "-" for no migration) +- **Notes**: Brief explanation (storage type, constraints, reason for staying, etc.) + +#### Table 2: Node State Before → After + +Shows cluster-wide impact: + +```markdown +## 📊 Node State: Before → After + +| Node | VMs Now | CPU Now | Memory Now | → | VMs After | CPU After | Memory After | Change | +|------|---------|---------|------------|---|-----------|-----------|--------------|--------| +| worker-01 | 5 | 85% | 72% | → | 3 | 68% | 59% | ✓ Reduced load | +| worker-02 | 2 | 42% | 48% | → | 3 | 58% | 61% | ← Receiving VMs | +| worker-03 | 3 | 38% | 51% | → | 4 | 55% | 63% | ← Receiving VMs | +``` + +**Column Definitions:** +- **Node**: Node name +- **VMs Now**: Current number of VMs on this node +- **CPU Now**: Current CPU utilization percentage +- **Memory Now**: Current memory utilization percentage +- **→**: Visual indicator of change +- **VMs After**: Number of VMs after rebalancing +- **CPU After**: Estimated CPU utilization after rebalancing +- **Memory After**: Estimated memory utilization after rebalancing +- **Change**: Brief description of what's happening (e.g., "✓ Reduced load", "← Receiving VMs", "✓ Balanced") + +### Additional Plan Context + +After the two tables, include: + +**Key Improvement:** +```markdown +**Key Improvement:** [Brief description of main benefit] +Example: "Distribution from 1 node to 4 nodes hosting VMs" +Example: "CPU variance reduced from 22% to 4% (81% improvement)" +``` + +**Rebalance Summary (for batch operations):** +```markdown +**Rebalance Summary:** +- **Total VMs to Rebalance**: 5 +- **Live**: 4 (zero downtime) +- **Cold**: 1 (brief downtime) +- **VMs Staying**: 2 (nodeAffinity constraints) +- **Total Downtime**: ~40s cumulative (cold only) +- **Estimated Duration**: 1-2 minutes (parallel execution) +``` + +**Execution Mode:** +```markdown +**Execution Mode**: **Parallel** (all VMs rebalance simultaneously) +- Default to parallel execution for time efficiency +- Only use sequential if user explicitly requests it or cluster constraints require it +``` + +**CRITICAL GUIDELINES:** +1. **Execution Default**: Parallel execution UNLESS user specifies sequential +2. **Risk Assessment**: ONLY include if there's a genuine risk users must understand (e.g., network saturation, resource exhaustion). Skip this section for normal rebalancing operations. + +### Terminology Standards + +**CRITICAL**: Use consistent terminology across all modes: + +- ✅ **"VM Rebalance Plan"** - Correct term for this skill +- ❌ **"VM Migration Plan"** - Do NOT use (reserved for future migration skill) +- ✅ **"Rebalancing"** - Correct term for the overall operation +- ✅ **"Live/Cold migration"** - Correct for describing the method +- ✅ **"Current Node" / "New Node"** - Clear directional language +- ✅ **"VMs Now" / "VMs After"** - Clear temporal language + +### When to Use This Format + +**Manual Mode**: Use when presenting plan for user confirmation (Step 3) +**Automatic Mode**: Use when presenting generated plan for user approval (Step 3) + +**Reference**: Both REBALANCE_MANUAL.md and REBALANCE_AUTOMATIC.md must follow this format. + +## Common Error Handling + +**All migration strategies use this error handling logic.** + +### Error 1: Live Migration Fails - Storage Not RWX + +**Symptom**: "Cannot live migrate VM: PVC access mode is ReadWriteOnce" + +**Cause**: Live migration requires RWX storage. VMs with RWO cannot be live migrated. + +**Solution**: +1. Use cold migration (works with RWO) +2. Or convert PVC to RWX storage class (requires data migration) + +**Reference**: [../../docs/troubleshooting/storage-errors.md](../../docs/troubleshooting/storage-errors.md) + +### Error 2: VM Stuck in ErrorUnschedulable After Cold Migration + +**Symptom**: "VM cannot be scheduled on target node: ErrorUnschedulable" + +**Cause**: Target node has insufficient resources, taints, or scheduling constraints. + +**Solution**: +1. Check node capacity using `nodes_top` MCP tool +2. Verify node has no blocking taints using `resources_get` for Node +3. Add tolerations to VM if node has taints +4. Choose different target node +5. Remove nodeSelector to allow scheduler flexibility + +**Reference**: [../../docs/troubleshooting/scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) + +### Error 3: Live Migration Times Out + +**Symptom**: "Migration exceeded timeout: 150 seconds per GiB" + +**Cause**: Large VM memory or high dirty page rate preventing convergence. + +**Solution**: +1. Retry migration (may succeed on second attempt) +2. Reduce VM workload during migration +3. Use cold migration (guaranteed to complete) +4. Increase timeout in HyperConverged CR (see performance-tuning reference) + +**Reference**: [references/performance-tuning.md](./references/performance-tuning.md) + +### Error 4: Migration Rejected - Cluster Limit Reached + +**Symptom**: "Migration rejected: cluster migration limit reached (5 concurrent)" + +**Cause**: KubeVirt limits concurrent migrations (default: 5 cluster-wide, 2 per node). + +**Solution**: +1. Wait for ongoing migrations to complete (monitor with `resources_list` for VirtualMachineInstanceMigration) +2. Retry migration +3. For batch operations, migrate sequentially +4. Increase cluster limit in HyperConverged CR (if network allows) + +**Reference**: [references/performance-tuning.md](./references/performance-tuning.md) + +### Error 5: RBAC Permission Denied + +**Symptom**: "Forbidden: User cannot create VirtualMachineInstanceMigration" + +**Cause**: ServiceAccount lacks RBAC permissions. + +**Solution**: +1. Verify KUBECONFIG has appropriate permissions +2. Required: `create` on VirtualMachineInstanceMigration, `update` on VirtualMachine +3. Contact cluster admin to grant permissions + +### Error 6: Network Saturation During Concurrent Migrations + +**Symptom**: Multiple migrations slow or fail; high network utilization + +**Cause**: Too many concurrent migrations saturating network bandwidth. + +**Solution**: +1. Reduce concurrent migrations in HyperConverged CR +2. Set bandwidth limit per migration +3. Use dedicated migration network + +**Reference**: [references/performance-tuning.md](./references/performance-tuning.md) + +### Error 7: Resource Version Conflict During Cold Migration + +**Symptom**: "Apply failed with 1 conflict: conflict with 'kubernetes-mcp-server' using kubevirt.io/v1: .spec.runStrategy" + +**Cause**: Using stale VM resourceVersion when updating nodeAffinity after `vm_lifecycle` operation. + +**Solution**: +After `vm_lifecycle` stop, re-read VM using `resources_get` before updating nodeAffinity. This gets fresh resourceVersion. + +**Workflow**: Stop VM → Wait for completion → Re-read VM → Update nodeAffinity → Start VM + +**Reference**: [REBALANCE_MANUAL.md - Sub-step 4b.2.5](./REBALANCE_MANUAL.md) + +## Dependencies + +### Required MCP Servers +- `openshift-virtualization` - OpenShift MCP server with KubeVirt toolset + - Source: https://github.com/openshift/openshift-mcp-server + +### Required MCP Tools + +- `resources_list` - List cluster resources (VMs, nodes, PVCs, migrations) +- `resources_get` - Get resource details (VM specs, VMI status, PVC access modes) +- `resources_create_or_update` - Create migrations, update VM nodeAffinity +- `vm_lifecycle` - Start/stop VMs for cold migration +- `nodes_top` - Monitor node resource usage +- `pods_top` - Monitor VM resource consumption +- `nodes_stats_summary` - Detailed node statistics + +**Source**: https://github.com/openshift/openshift-mcp-server + +### Related Skills + +- `vm-inventory` - List VMs and check placement before migration +- `vm-lifecycle-manager` - Simple start/stop without migration +- `vm-create` - Create VMs with initial placement preferences +- `vm-snapshot-create` - Backup VMs before risky migrations + +### Reference Documentation + +**Skill Strategy Files**: +- [REBALANCE_MANUAL.md](./REBALANCE_MANUAL.md) - User-driven migration workflow +- [REBALANCE_AUTOMATIC.md](./REBALANCE_AUTOMATIC.md) - AI-driven rebalancing workflow + +**Performance and Best Practices**: +- [references/live-migration-best-practices.md](./references/live-migration-best-practices.md) - Configuration, requirements, dedicated networks +- [references/performance-tuning.md](./references/performance-tuning.md) - Right-sizing, overcommit, bandwidth tuning +- [references/anti-patterns.md](./references/anti-patterns.md) - Common mistakes to avoid +- [references/production-considerations.md](./references/production-considerations.md) - HA strategies, capacity planning, security + +**Troubleshooting**: +- [../../docs/troubleshooting/INDEX.md](../../docs/troubleshooting/INDEX.md) - Master troubleshooting index +- [../../docs/troubleshooting/scheduling-errors.md](../../docs/troubleshooting/scheduling-errors.md) - ErrorUnschedulable, taints, resources +- [../../docs/troubleshooting/storage-errors.md](../../docs/troubleshooting/storage-errors.md) - PVC access mode issues +- [../../docs/troubleshooting/lifecycle-errors.md](../../docs/troubleshooting/lifecycle-errors.md) - VM start/stop failures + +**Official Red Hat Documentation**: +- [OpenShift Virtualization - Live Migration](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt-live-migration) +- [OpenShift Virtualization - Node Placement](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt-node-placement) + +**Upstream KubeVirt Documentation**: +- [Live Migration - KubeVirt User Guide](https://kubevirt.io/user-guide/compute/live_migration/) +- [Node Assignment - KubeVirt User Guide](https://kubevirt.io/user-guide/compute/node_assignment/) +- [VirtualMachineInstanceMigration API](https://kubevirt.io/api-reference/main/definitions.html#_v1_virtualmachineinstancemigration) + +## Critical: Human-in-the-Loop Requirements + +**IMPORTANT**: This skill performs VM migrations affecting workload placement and availability. You MUST: + +1. **Before Initiating Migration** + - Present complete rebalance plan (VM, nodes, type, impact) + - Clearly explain downtime (live = <1s pause, cold = 30-60s) + - Show current vs target placement + - Ask: "Confirm this migration?" + - Wait for explicit user confirmation + +2. **Never Auto-Execute** + - **NEVER migrate without confirmation** + - **NEVER assume live vs cold** - ask or infer from storage + - **NEVER skip impact explanation** + - **NEVER proceed if validation fails** - report issues first + +3. **For Batch Operations** + - Present all VMs to be migrated + - Show total impact (e.g., "3 VMs, 2 live + 1 cold") + - Confirm entire batch before starting + - Report progress for each + - Stop on first failure + +**Why This Matters**: +- **Live Migration**: Brief pause, bandwidth usage, performance impact during transfer +- **Cold Migration**: Service downtime, dropped connections, application interruption +- **Wrong Node**: Performance degradation or scheduling conflicts +- **Batch**: Can saturate network or exhaust resources + +**Rationale**: Prevents unintended disruption; maintains user control over placement and availability. + +## Security Considerations + +- **RBAC Enforcement**: Requires specific permissions (create/update/list) +- **Node Access**: Respects node taints and RBAC policies +- **Storage Security**: Data remains encrypted if using encrypted storage classes +- **Network Isolation**: Migrations respect NetworkPolicies +- **Audit Trail**: All operations logged in Kubernetes API audit logs +- **KUBECONFIG Security**: Credentials never exposed in output +- **Resource Quotas**: Respects namespace quotas and limits +- **Tenant Isolation**: Cannot migrate across namespaces without RBAC + +--- + +**Strategy Implementation Status**: +- ✅ **REBALANCE_MANUAL.md** - Fully implemented +- ✅ **REBALANCE_AUTOMATIC.md** - Fully implemented + +**Reference Documentation Status**: +- ✅ **live-migration-best-practices.md** - Complete +- ✅ **performance-tuning.md** - Complete +- ✅ **anti-patterns.md** - Complete +- ✅ **production-considerations.md** - Complete + +--- + +**Last Updated**: 2026-02-24 +**OpenShift Virtualization Versions**: 4.17, 4.18, 4.19, 4.20 diff --git a/rh-virt/skills/vm-rebalance/references/anti-patterns.md b/rh-virt/skills/vm-rebalance/references/anti-patterns.md new file mode 100644 index 00000000..f1321fdc --- /dev/null +++ b/rh-virt/skills/vm-rebalance/references/anti-patterns.md @@ -0,0 +1,869 @@ +# Anti-Patterns: What NOT to Do + +**Purpose**: Common mistakes, anti-patterns, and pitfalls to avoid when rebalancing VMs in OpenShift Virtualization. + +**When to consult this document**: Before planning rebalancing operations, when troubleshooting failures, or when designing cluster architecture. + +--- + +## Official Sources + +This document is compiled from official Red Hat documentation and community best practices: + +- [Best Practices for Virtual Machine Deployments on OpenShift Virtualization](https://learn.microsoft.com/en-us/azure/openshift/best-practices-openshift-virtualization) - Microsoft Azure Red Hat OpenShift (2026-02-16) +- [Best Practices to Deploy VMs in Red Hat OpenShift Virtualization](https://docs.netapp.com/us-en/netapp-solutions-virtualization/openshift/os-osv-bpg.html) - NetApp Solutions +- [OpenShift Virtualization Best Practices](https://www.tigera.io/learn/guides/kubernetes-networking/openshift-virtualization/) - Tigera +- [Troubleshooting OpenShift Virtualization](https://access.redhat.com/articles/6256861) - Red Hat Customer Portal + +--- + +## Storage Anti-Patterns + +### ❌ Anti-Pattern 1: Using RWO Storage for Live Migration + +**What NOT to Do:** +```yaml +# BAD: VM using ReadWriteOnce storage +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: vm-disk +spec: + accessModes: + - ReadWriteOnce # Cannot live migrate! + storageClassName: gp3 +``` + +**Why It Fails:** + +From Red Hat documentation: +> "Live migration requires the use of a shared storage solution that provides ReadWriteMany (RWX) access mode. The VM disks should be backed by storage option that provides RWX access mode." + +**Error Message:** +``` +cannot migrate VMI: PVC vm-disk is not shared, live migration requires +that all PVCs must be shared (using ReadWriteMany access mode) +``` + +**Correct Approach:** + +**Before Planning Live Migration**, verify storage using MCP tools: + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "name": "" +} +``` + +Check `.spec.accessModes` includes `"ReadWriteMany"`. + +**If RWO storage**, use **cold migration** instead (see REBALANCE_MANUAL.md). + +**Storage Types Supporting RWX:** +- ✅ NFS (ontap-nas driver) +- ✅ SMB/CIFS (ontap-nas driver) +- ✅ iSCSI/FC (ontap-san driver, **raw block mode only**) +- ❌ AWS EBS gp3 (RWO only) +- ❌ Local storage (RWO only) + +--- + +### ❌ Anti-Pattern 2: Not Setting SVM Volume Limits + +**What NOT to Do:** + +Deploy Trident without configuring SVM (Storage Virtual Machine) volume limits, allowing unchecked resource consumption. + +**Why It's Dangerous:** + +From NetApp documentation: +> "Set volume limits to prevent Trident from consuming all storage" + +**Impact:** +- Trident creates unlimited volumes +- Storage backend exhausted +- Other workloads starved of storage +- Production outages + +**Correct Approach:** + +Configure limits at multiple levels: + +**1. SVM-level volume limit:** +```bash +vserver modify -vserver -max-volumes +``` + +**2. Storage limits on SVM:** +```bash +vserver create -vserver vserver_name -aggregate aggregate_name -storage-limit value +``` + +**3. Trident backend parameters:** +- `limitVolumeSize`: Maximum volume size created by Trident (e.g., "100Gi") +- `limitVolumePoolSize`: Maximum FlexVol size for economy drivers (e.g., "500Gi") + +--- + +### ❌ Anti-Pattern 3: Enabling showmount Without Justification + +**What NOT to Do:** + +Leave `showmount` enabled on NFS SVMs, exposing volume information to unauthorized clients. + +**Why It's a Security Risk:** + +From NetApp documentation: +> "Disable showmount to prevent unauthorized volume discovery" + +**Correct Approach:** + +Disable showmount unless specifically required: + +```bash +vserver nfs modify -vserver -showmount disabled +``` + +Implement separate export policies for infrastructure vs application nodes for granular access control. + +--- + +## Scheduling and Node Placement Anti-Patterns + +### ❌ Anti-Pattern 4: Excessive Affinity Rules + +**What NOT to Do:** + +Apply too many complex affinity, anti-affinity, node selector, and toleration rules to VMs. + +**Why It's Problematic:** + +From community best practices: +> "Too many rules make scheduling slow and hard to reason about." + +From Red Hat documentation: +> "Affinity rules only apply during scheduling. OpenShift Container Platform does not reschedule running workloads if the constraints are no longer met." + +**Impact:** +- Slow VM scheduling (scheduler overhead) +- Impossible-to-satisfy constraints (VM stuck in Pending) +- Difficult troubleshooting (complex rule interactions) +- No automatic rebalancing when constraints violated + +**Correct Approach:** + +**Keep rules simple and minimal:** + +```yaml +# GOOD: Simple, clear node selector +spec: + template: + spec: + nodeSelector: + workload-type: virtualization +``` + +```yaml +# BAD: Too many overlapping constraints +spec: + template: + spec: + nodeSelector: + node-role.kubernetes.io/worker: "" + workload-type: virtualization + zone: us-east-1a + instance-type: m5.4xlarge + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app + operator: In + values: [web, database, cache] + topologyKey: kubernetes.io/hostname + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: tier + operator: In + values: [frontend] + topologyKey: failure-domain.beta.kubernetes.io/zone + tolerations: + - key: dedicated + operator: Equal + value: virtualization + effect: NoSchedule + - key: high-performance + operator: Exists + effect: NoSchedule +``` + +**Recommendation:** +- Use **one** primary constraint (nodeSelector OR affinity) +- Add tolerations only when nodes have taints +- Avoid mixing required and preferred affinity rules +- Document the intent of each rule + +--- + +### ❌ Anti-Pattern 5: Not Planning for Node Failures + +**What NOT to Do:** + +Rely on automatic VM failover without configuring machine health checks. + +**Why It Fails:** + +From Red Hat documentation: +> "If a node fails and machine health checks are not deployed on your cluster, virtual machines (VMs) with RunStrategy: Always configured are not automatically relocated to healthy nodes. To trigger VM failover, you must manually delete the Node object." + +**Impact:** +- VMs remain assigned to failed node +- Manual intervention required for recovery +- Extended downtime during node failures + +**Correct Approach:** + +**1. Deploy Machine Health Checks:** + +Configure cluster-level machine health checks to detect and remediate node failures automatically. + +**2. Use RunStrategy: Always for HA VMs:** + +```yaml +spec: + runStrategy: Always # Ensures VM restarts after node recovery +``` + +**3. Implement VM Replication:** + +For critical VMs, create replicas with anti-affinity rules to ensure distribution across different nodes/zones. + +**4. Monitor Node Health:** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "Node" +} +``` + +Filter for nodes where `.status.conditions[]` shows `Ready=False` or other unhealthy states. + +--- + +## Resource Management Anti-Patterns + +### ❌ Anti-Pattern 6: Exceeding CPU Overcommit Limits + +**What NOT to Do:** + +Configure CPU overcommit ratio >1.8x physical cores. + +**Why It's Dangerous:** + +From Red Hat documentation: +> "CPU over-commitment ratio must not exceed 1.8x of the number of physical cores while memory usage may not exceed 0.9x of the physical memory available in a cluster. CPU over-commitment leads to throttling, causing slowness of all workloads on the impacted node." + +**Impact:** +- CPU throttling across ALL VMs on node +- Unpredictable performance degradation +- Cascading slowness affecting entire cluster +- User-facing application latency + +**Correct Approach:** + +**Check Current Overcommit:** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "hco.kubevirt.io/v1beta1", + "kind": "HyperConverged", + "namespace": "openshift-cnv", + "name": "kubevirt-hyperconverged" +} +``` + +Review `.spec.resourceRequirements.vmiCPUAllocationRatio`. + +**Safe Limits:** +- **Production**: 1.0-1.2x (no/minimal overcommit) +- **Dev/Test**: 1.2-1.5x (moderate overcommit) +- **Absolute Maximum**: 1.8x (with careful monitoring) + +**Never Exceed**: 1.8x CPU or 0.9x memory limits. + +--- + +### ❌ Anti-Pattern 7: Applying Strict Resource Limits to VMs + +**What NOT to Do:** + +Set both resource requests **and** limits on VMs without specific governance requirements. + +**Why It's Problematic:** + +From Microsoft Azure Red Hat OpenShift guidance: +> "Avoid strict resource limits: Set only guest memory for VMs; avoid strict resource limits unless required for governance." + +**Impact:** +- CPU throttling even when node has spare capacity +- Reduced VM performance +- Wasted cluster resources +- Difficult troubleshooting (invisible throttling) + +**Correct Approach:** + +**Set requests only:** + +```yaml +# GOOD: Requests only (allows bursting) +spec: + template: + spec: + domain: + resources: + requests: + memory: 16Gi + cpu: 4 +``` + +```yaml +# BAD: Requests + limits (strict throttling) +spec: + template: + spec: + domain: + resources: + requests: + memory: 16Gi + cpu: 4 + limits: # Avoid unless required + memory: 16Gi + cpu: 4 +``` + +**Only set limits when:** +- Governance policies mandate strict resource boundaries +- Multi-tenant environments require isolation +- Preventing one VM from starving others + +--- + +### ❌ Anti-Pattern 8: Relying on On-Premises Sizing References + +**What NOT to Do:** + +Size VMs in OpenShift Virtualization based on on-premises VM sizes without testing. + +**Why It Fails:** + +From Microsoft Azure Red Hat OpenShift guidance: +> "Avoid relying solely on on-premises sizing references; benchmark your own workloads to inform right sizing." + +**Impact:** +- Overprovisioned VMs (wasted resources) +- Underprovisioned VMs (performance issues) +- Unexpected architectural overhead (VMs != native pods) +- Incorrect migration time estimates + +**Correct Approach:** + +**1. Benchmark workloads in OpenShift Virtualization:** +- Deploy test VMs with various sizes +- Run representative workload tests +- Measure actual performance vs requirements + +**2. Account for architectural overhead:** + +Expect 4-56% performance overhead vs bare metal (see performance-tuning.md for details). + +**3. Monitor and adjust:** + +**MCP Tool**: `pods_top` (from openshift-virtualization) + +**Parameters**: +```json +{ + "all_namespaces": true, + "label_selector": "kubevirt.io=virt-launcher" +} +``` + +Track actual resource usage and resize VMs accordingly. + +--- + +## Network Anti-Patterns + +### ❌ Anti-Pattern 9: Using OVN-Kubernetes with Linux Bridge on Default Interface + +**What NOT to Do:** + +Attempt to attach a Linux bridge or bonding device to the host's default interface when using OVN-Kubernetes CNI. + +**Why It Fails:** + +From Red Hat documentation: +> "If your OpenShift Container Platform cluster uses OVN-Kubernetes as the default CNI provider, you cannot attach a Linux bridge or bonding device to a host's default interface." + +**Impact:** +- Network configuration failures +- VM networking broken +- Migration network setup fails + +**Correct Approach:** + +**Option 1: Use secondary network interface** + +Attach Linux bridge to a different physical interface (not the default). + +**Option 2: Switch to OpenShift SDN CNI** + +If Linux bridge on default interface is required, reconfigure cluster to use OpenShift SDN instead of OVN-Kubernetes. + +**Option 3: Use OVS bridge instead** + +For migration networks, use Open vSwitch bridge (compatible with OVN-Kubernetes). + +--- + +### ❌ Anti-Pattern 10: Ignoring MTU Mismatches + +**What NOT to Do:** + +Mix network types with different default MTUs without explicit configuration. + +**Why It's Problematic:** + +From Red Hat documentation: +> "When a virtual machine interface is connected to an OVS bridge, the default MTU is 1400, but when connected to a Linux bridge, the default MTU is 1500." + +**Impact:** +- Packet fragmentation +- Reduced network performance +- Subtle communication failures +- Migration slowdowns + +**Correct Approach:** + +**Explicitly set MTU in NetworkAttachmentDefinition:** + +```json +{ + "cniVersion": "0.3.1", + "name": "migration-bridge", + "type": "macvlan", + "master": "eth1", + "mode": "bridge", + "mtu": 9000, # Explicit MTU setting + "ipam": {...} +} +``` + +**Validate MTU consistency across all interfaces involved in migration.** + +--- + +## Architecture and Platform Anti-Patterns + +### ❌ Anti-Pattern 11: Using RHEL Compute Nodes + +**What NOT to Do:** + +Deploy OpenShift Virtualization on Red Hat Enterprise Linux (RHEL) compute nodes. + +**Why It Fails:** + +From Red Hat documentation: +> "OpenShift Virtualization requires Red Hat Enterprise Linux CoreOS (RHCOS) compute nodes. Even though it is possible to deploy Red Hat Enterprise Linux (RHEL) compute nodes, they are incompatible with OpenShift Virtualization." + +**Impact:** +- VM scheduling failures +- Unsupported configuration +- Migration failures +- No Red Hat support + +**Correct Approach:** + +**Verify all nodes are RHCOS:** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "Node" +} +``` + +For each node, check `.status.nodeInfo.osImage` contains "CoreOS". + +**If RHEL nodes detected:** +- Replace with RHCOS nodes +- Do NOT schedule VMs on RHEL nodes +- Remove RHEL nodes from cluster before deploying virtualization workloads + +--- + +### ❌ Anti-Pattern 12: Placing Master Nodes on Same VMware Host + +**What NOT to Do:** + +In VMware-based deployments, place multiple OpenShift master nodes on the same VMware ESXi host. + +**Why It's Dangerous:** + +From VMware best practices: +> "Critical best practices include: distributing the 3 virtual master nodes across different VMware hosts, placing each master node on a separate datastore, and avoiding hosting master nodes on datastores with high I/O workloads." + +**Impact:** +- Single point of failure (host failure kills multiple masters) +- etcd performance degradation (etcd is latency-sensitive) +- Cluster control plane outage +- Violates high-availability principles + +**Correct Approach:** + +**1. Distribute master nodes across different VMware hosts** + +Use VM anti-affinity rules to enforce separation. + +**2. Use separate datastores for each master node** + +Prevents storage failure from affecting multiple masters. + +**3. Avoid high I/O datastores for master nodes** + +etcd is sensitive to disk latency; use low-latency storage. + +--- + +### ❌ Anti-Pattern 13: Ignoring etcd Latency Sensitivity + +**What NOT to Do:** + +Place etcd (control plane) on high-latency storage or overloaded nodes. + +**Why It's Critical:** + +From best practices: +> "The etcd component hosted on control-plane nodes is usually the component most sensitive to latency issues." + +**Impact:** +- Cluster control plane slowness +- API server timeouts +- Failed VM operations +- Cluster instability + +**Correct Approach:** + +**1. Use low-latency storage for control plane nodes:** +- SSD-backed storage (not HDD) +- Local NVMe if available +- Avoid shared storage with high I/O contention + +**2. Monitor etcd latency:** + +Prometheus metrics: `etcd_disk_wal_fsync_duration_seconds` + +**Target**: <10ms for WAL fsync + +**3. Isolate control plane from VM workloads:** + +Use taints on master nodes to prevent VM scheduling. + +--- + +## Migration Operation Anti-Patterns + +### ❌ Anti-Pattern 14: Not Reducing VM Workload During Migration + +**What NOT to Do:** + +Attempt live migration of write-intensive VMs (databases, caches) under full load. + +**Why It's Problematic:** + +High memory write rate (dirty page rate) can exceed network transfer rate, preventing migration convergence. + +**Impact:** +- Migration timeouts +- Failed migrations +- Extended migration duration +- Network saturation + +**Correct Approach:** + +**Before migrating write-intensive VMs:** + +1. **Schedule migration during low-activity window** (off-hours, maintenance window) + +2. **Temporarily reduce workload:** + - Stop non-critical background processes + - Scale down application traffic + - Pause batch jobs + +3. **Consider cold migration instead** for extremely write-heavy workloads (guaranteed completion) + +4. **Increase timeouts if load cannot be reduced:** + +Modify `.spec.liveMigrationConfig.completionTimeoutPerGiB` in HyperConverged CR (see performance-tuning.md). + +--- + +### ❌ Anti-Pattern 15: Parallel Migrations Without Dedicated Network + +**What NOT to Do:** + +Run many concurrent migrations on shared application network without bandwidth limits. + +**Why It's Dangerous:** + +- Saturates network bandwidth +- Degrades application performance +- Migration failures due to slow transfers +- Cascading performance impact + +**Impact Observed:** + +From search results: +> "Network saturation risk with concurrent migrations" + +**Correct Approach:** + +**Option 1: Use dedicated migration network** (see live-migration-best-practices.md) + +**Option 2: Limit concurrent migrations:** + +Modify `.spec.liveMigrationConfig.parallelMigrationsPerCluster` in HyperConverged CR: + +```yaml +spec: + liveMigrationConfig: + parallelMigrationsPerCluster: 3 # Conservative limit + bandwidthPerMigration: 64Mi # Bandwidth cap per migration +``` + +**Option 3: Migrate sequentially** + +Migrate VMs one at a time instead of batch operations. + +--- + +### ❌ Anti-Pattern 16: Not Validating Migration Prerequisites + +**What NOT to Do:** + +Attempt migration without verifying storage, network, and capacity prerequisites. + +**Why It Fails:** + +Common failures: +- RWO storage → "PVC is not shared" error +- VM not running → "cannot migrate stopped VM" +- Node at capacity → ErrorUnschedulable +- Network issues → Migration timeout + +**Correct Approach:** + +**Always run pre-migration validation** (see live-migration-best-practices.md for complete checklist): + +1. ✅ Verify PVC access modes (RWX required) +2. ✅ Check VM is running (VMI exists) +3. ✅ Validate target node capacity +4. ✅ Confirm virt-handler pods healthy +5. ✅ Check cluster migration limits + +**Use Common Validation Logic from SKILL.md before every migration.** + +--- + +## Production Deployment Anti-Patterns + +### ❌ Anti-Pattern 17: Deploying to Production Without Testing + +**What NOT to Do:** + +Deploy VMs directly to production without dev/test validation. + +**Why It's Risky:** + +From best practices: +> "Begin with non-critical or dev/test workloads before moving production systems - this phased approach allows teams to gain hands-on experience while minimizing risk." + +**Impact:** +- Unexpected performance issues +- Migration failures affecting production +- Learning curve impacts critical systems +- Difficult rollback + +**Correct Approach:** + +**Phased Rollout:** + +**Phase 1: Development/Test** +- Deploy test VMs +- Validate performance and functionality +- Test migration workflows +- Gain operational experience + +**Phase 2: Non-Critical Production** +- Migrate non-critical workloads +- Monitor performance and stability +- Refine sizing and configurations +- Build confidence + +**Phase 3: Critical Production** +- Migrate critical workloads +- Ensure HA and DR configured +- 24/7 monitoring in place +- Rollback plan ready + +--- + +### ❌ Anti-Pattern 18: Starting Big Instead of Small + +**What NOT to Do:** + +Provision large VM fleet from day one without iterative growth. + +**Why It's Problematic:** + +From best practices: +> "When starting with OpenShift Virtualization, it's essential to start small and scale up as needed to avoid over-provisioning and wasting resources." + +**Impact:** +- Overprovisioned cluster (wasted costs) +- Underutilized resources +- Difficult rightsizing later +- Commitment to suboptimal architecture + +**Correct Approach:** + +**Start small:** +1. Deploy 5-10 VMs initially +2. Monitor resource usage patterns +3. Adjust sizing based on actual metrics +4. Gradually add VMs as needs grow + +**Validate assumptions:** +- Test architectural overhead +- Measure actual performance +- Refine resource allocation +- Iterate on configuration + +--- + +### ❌ Anti-Pattern 19: Not Monitoring After Rebalancing + +**What NOT to Do:** + +Execute rebalancing operations and assume everything is optimal without validation. + +**Why It's Risky:** + +- May not achieve intended load distribution +- Hidden performance degradation +- VMs scheduled suboptimally +- Resource contention not detected + +**Correct Approach:** + +**Post-Rebalancing Validation:** + +**1. Verify VM placement:** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +For each migrated VM, confirm `.status.nodeName` matches expected target node. + +**2. Monitor node resource usage:** + +**MCP Tool**: `nodes_top` (from openshift-virtualization) + +**Before vs After Comparison:** + +| Node | CPU Before | CPU After | Improvement | +|------|------------|-----------|-------------| +| worker-01 | 85% | 65% | -20% ✓ | +| worker-02 | 78% | 64% | -14% ✓ | +| worker-03 | 42% | 58% | +16% | +| worker-04 | 38% | 53% | +15% | + +**3. Validate application performance:** + +Check application-specific metrics (response time, throughput, error rates). + +**4. Monitor for 24-48 hours:** + +Ensure sustained improvement without unexpected side effects. + +--- + +## Summary: Anti-Pattern Checklist + +Before rebalancing VMs, avoid these critical mistakes: + +**Storage:** +- ❌ Using RWO storage for live migration +- ❌ Not setting SVM volume limits +- ❌ Leaving showmount enabled + +**Scheduling:** +- ❌ Too many complex affinity rules +- ❌ Not configuring machine health checks + +**Resources:** +- ❌ CPU overcommit >1.8x +- ❌ Strict resource limits without justification +- ❌ Using on-premises sizing without testing + +**Network:** +- ❌ Linux bridge on default interface with OVN-Kubernetes +- ❌ Ignoring MTU mismatches +- ❌ Parallel migrations without dedicated network + +**Platform:** +- ❌ Using RHEL compute nodes +- ❌ Master nodes on same VMware host +- ❌ Ignoring etcd latency sensitivity + +**Operations:** +- ❌ Not reducing VM workload during migration +- ❌ Skipping pre-migration validation +- ❌ No post-rebalancing monitoring + +**Production:** +- ❌ Deploying to production without testing +- ❌ Starting big instead of small + +--- + +## Related Documentation + +- [Live Migration Best Practices](./live-migration-best-practices.md) - What TO do for successful migrations +- [Performance Tuning](./performance-tuning.md) - Optimization strategies +- [Production Considerations](./production-considerations.md) - Right-sizing, workload planning, HA strategies + +--- + +**Last Updated**: 2026-02-24 +**OpenShift Virtualization Versions**: 4.17, 4.18, 4.19, 4.20 +**Status**: Curated from official Red Hat sources and production experience diff --git a/rh-virt/skills/vm-rebalance/references/live-migration-best-practices.md b/rh-virt/skills/vm-rebalance/references/live-migration-best-practices.md new file mode 100644 index 00000000..7df97527 --- /dev/null +++ b/rh-virt/skills/vm-rebalance/references/live-migration-best-practices.md @@ -0,0 +1,794 @@ +# Live Migration Best Practices + +**Purpose**: Configuration parameters, requirements, and best practices for VM live migration in OpenShift Virtualization. + +**When to consult this document**: Before executing live migrations, when configuring cluster-wide migration settings, or when troubleshooting migration performance issues. + +--- + +## Official Sources + +This document is compiled from official Red Hat documentation: + +- [Live Migrating VMs with OpenShift Virtualization](https://developers.redhat.com/articles/2025/07/14/live-migrating-vms-openshift-virtualization) - Red Hat Developer (2025-07-14) +- [How OpenShift Virtualization Supports VM Live Migration](https://developers.redhat.com/articles/2025/06/05/how-openshift-virtualization-supports-vm-live-migration) - Red Hat Developer (2025-06-05) +- [Chapter 12. Live Migration - OpenShift Container Platform 4.18](https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/virtualization/live-migration) - Red Hat Documentation +- [Best Practices for Virtual Machine Deployments on OpenShift Virtualization](https://learn.microsoft.com/en-us/azure/openshift/best-practices-openshift-virtualization) - Microsoft Azure Red Hat OpenShift (2026-02-16) +- [Best Practices to Deploy VMs in Red Hat OpenShift Virtualization](https://docs.netapp.com/us-en/netapp-solutions-virtualization/openshift/os-osv-bpg.html) - NetApp Solutions + +--- + +## Configuration Parameters + +### HyperConverged CR Live Migration Settings + +All live migration settings are configured in the `HyperConverged` custom resource located in the `openshift-cnv` namespace. + +**Default Configuration:** +```yaml +apiVersion: hco.kubevirt.io/v1beta1 +kind: HyperConverged +metadata: + name: kubevirt-hyperconverged + namespace: openshift-cnv +spec: + liveMigrationConfig: + completionTimeoutPerGiB: 800 # Seconds per GiB for migration completion + parallelMigrationsPerCluster: 5 # Max concurrent migrations cluster-wide + parallelOutboundMigrationsPerNode: 2 # Max concurrent migrations per source node + progressTimeout: 150 # Max seconds without progress before cancellation + bandwidthPerMigration: 64Mi # (Optional) Bandwidth limit per migration + network: "" # (Optional) Dedicated secondary network for migration +``` + +**Parameter Explanations:** + +| Parameter | Default | Description | Tuning Guidance | +|-----------|---------|-------------|-----------------| +| `completionTimeoutPerGiB` | 800s | Migration completion duration per gigabyte of VM memory | Increase for high memory write rate (dirty page) workloads | +| `progressTimeout` | 150s | Maximum seconds without migration progress before cancellation | Increase for large VMs (>100GB) or slow networks | +| `parallelMigrationsPerCluster` | 5 | Cluster-wide concurrent migration limit | Increase if network bandwidth allows; decrease if saturation occurs | +| `parallelOutboundMigrationsPerNode` | 2 | Per-node concurrent outbound migration limit | Keep at 2 to prevent single-node overload | +| `bandwidthPerMigration` | 64Mi | (Optional) Bandwidth limit per migration | Set to prevent network saturation; omit for unlimited | +| `network` | "" | (Optional) NetworkAttachmentDefinition for dedicated migration network | Highly recommended for production; see Dedicated Networks section | + +**How to Update Configuration Using MCP Tools:** + +**Step 1: Get current HyperConverged resource** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "hco.kubevirt.io/v1beta1", + "kind": "HyperConverged", + "namespace": "openshift-cnv", + "name": "kubevirt-hyperconverged" +} +``` + +**Step 2: Modify the returned JSON to update liveMigrationConfig** + +Add or update the `.spec.liveMigrationConfig` section: +```json +{ + "spec": { + "liveMigrationConfig": { + "completionTimeoutPerGiB": 1200, + "parallelMigrationsPerCluster": 10, + "progressTimeout": 300, + "bandwidthPerMigration": "32Mi" + } + } +} +``` + +**Step 3: Apply the updated configuration** + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Parameters**: +```json +{ + "resource": "" +} +``` + +Pass the complete modified HyperConverged resource as YAML or JSON string. + +--- + +## Prerequisites and Requirements + +### Storage Requirements + +**CRITICAL**: Live migration requires **ReadWriteMany (RWX)** access mode storage. + +**Supported Storage Types for Live Migration:** + +| Storage Type | Access Mode | Live Migration Support | Notes | +|--------------|-------------|------------------------|-------| +| NFS (ontap-nas driver) | RWX | ✅ Supported | Recommended for general use | +| SMB/CIFS (ontap-nas driver) | RWX | ✅ Supported | Windows-compatible | +| iSCSI/FC (ontap-san driver) | RWX (raw block mode only) | ✅ Supported | High performance; requires raw block volumes | +| Local storage / AWS EBS (gp3) | RWO | ❌ NOT Supported | Use cold migration instead | + +**Validation Using MCP Tools:** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "name": "" +} +``` + +**Expected Output**: Check `.spec.accessModes` in the returned PVC resource. + +For live migration, access modes must include `"ReadWriteMany"`. + +**What Happens with RWO Storage:** + +When attempting live migration with ReadWriteOnce (RWO) storage: +``` +Error: cannot migrate VMI: PVC is not shared, live migration requires +that all PVCs must be shared (using ReadWriteMany access mode) +``` + +**Solution**: Use cold migration workflow for VMs with RWO storage (see REBALANCE_MANUAL.md). + +--- + +### Hardware and Network Requirements + +**Minimum Requirements:** + +- **Nodes**: Red Hat Enterprise Linux CoreOS (RHCOS) compute nodes (RHEL nodes are incompatible) +- **Network**: All nodes must be on the same L2 network or have routable connectivity +- **CPU**: Sufficient CPU headroom on target node for incoming VM workload +- **Memory**: Sufficient free memory on target node (>= VM memory allocation) + +**Recommended for Production:** + +- **Network Cards**: 100Gbps NICs for large VM migrations (>500GB memory) +- **Dedicated Migration Network**: Secondary physical network or VLAN for isolation +- **Storage Backend**: SSD-backed shared storage (NFS-CSI, OpenShift Data Foundation, Azure NetApp Files) +- **MTU Configuration**: Set to 9000 for migration networks to improve efficiency + +--- + +## Dedicated Migration Network (Production Best Practice) + +### Why Use a Dedicated Network? + +**Benefits:** +- Isolates migration traffic from application workloads +- Prevents network contention and performance degradation +- Enables higher bandwidth allocation (e.g., 100Gbps dedicated) +- Improves security and manageability +- Reduces migration time for large VMs + +**When to Use:** +- Production environments with large VMs (>100GB memory) +- Clusters with high application network traffic +- Environments requiring strict network isolation +- High-availability requirements with frequent migrations + +### Configuration Example + +**Step 1: Create NodeNetworkConfigurationPolicy (NNCP)** + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Parameters**: +```json +{ + "resource": "apiVersion: nmstate.io/v1\nkind: NodeNetworkConfigurationPolicy\nmetadata:\n name: migration-network-policy\nspec:\n desiredState:\n interfaces:\n - name: br-lm\n description: OVS bridge for live migration\n type: ovs-bridge\n state: up\n bridge:\n allow-extra-patch-ports: true\n port:\n - name: enp4s0\n vlan:\n mode: access\n tag: 3030\n options:\n stp: false" +} +``` + +**Step 2: Create NetworkAttachmentDefinition (NAD)** + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Parameters**: +```json +{ + "resource": "apiVersion: k8s.cni.cncf.io/v1\nkind: NetworkAttachmentDefinition\nmetadata:\n name: migration-network\n namespace: openshift-cnv\nspec:\n config: '{\n \"cniVersion\": \"0.3.1\",\n \"name\": \"migration-bridge\",\n \"type\": \"macvlan\",\n \"master\": \"eth1\",\n \"mode\": \"bridge\",\n \"ipam\": {\n \"type\": \"whereabouts\",\n \"range\": \"10.200.5.0/24\",\n \"excludeSubnets\": \"10.200.5.0/30\"\n }\n }'" +} +``` + +**Step 3: Configure HyperConverged CR to Use Network** + +**MCP Tool**: Get current HyperConverged, modify, and update using `resources_create_or_update` + +Add to `.spec.liveMigrationConfig`: +```json +{ + "network": "migration-network" +} +``` + +**Step 4: Verify virt-handler Pods Restarted** + +**MCP Tool**: `pods_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "labelSelector": "kubevirt.io=virt-handler" +} +``` + +**Expected**: All pods show READY status and recent start time (AGE). + +Filter results where `status.containerStatuses[0].ready == true` and `status.containerStatuses[0].restartCount` is recent. + +**Verification After Migration:** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstanceMigration", + "namespace": "", + "name": "" +} +``` + +Check `.status.migrationState.targetNodeAddress` - should be an IP from the dedicated subnet (e.g., 10.200.5.15). + +--- + +## Migration Process and Technologies + +### Pre-copy Migration + +Live migration uses **pre-copy** strategy: + +1. **Initial Copy**: VM continues running on source node while memory is copied to target +2. **Iterative Copy**: Pages modified during copy (dirty pages) are re-copied +3. **Cutover**: Brief pause (<1 second) to copy final dirty pages and switch execution +4. **Cleanup**: Source VM instance is terminated + +**Multi-fd Technology** (for high-load scenarios): + +- Sends data over multiple network streams in parallel +- Maximizes bandwidth utilization during migration +- Handles high dirty page rates (e.g., SAP HANA, databases with high write rates) +- Automatically enabled by KubeVirt when beneficial + +**Migration Phases:** + +``` +Pending → Scheduling → PreparingTarget → Running → Succeeded +``` + +**Monitor with MCP Tools:** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstanceMigration" +} +``` + +Filter results by `.status.phase` to see current migration status. + +--- + +## Best Practices + +### 1. VM Resource Optimization + +**Enable Dedicated Resources:** + +Configure VMs with dedicated CPU and memory isolation for performance-sensitive workloads: + +```yaml +spec: + template: + spec: + domain: + cpu: + dedicatedCpuPlacement: true + resources: + requests: + memory: 16Gi +``` + +**Benefits:** +- Improves VM performance and latency predictability +- Reduces migration time (less CPU contention) +- Better accuracy for latency predictions + +**When to Use:** +- Database workloads (PostgreSQL, MySQL, SAP HANA) +- Real-time analytics applications +- Low-latency requirements + +### 2. Hugepage Configuration + +For large VMs (>100GB memory), configure hugepages to reduce memory page overhead: + +**Node Configuration Using MCP Tools:** + +**MCP Tool**: `resources_get` then `resources_create_or_update` (from openshift-virtualization) + +**Step 1: Get Node resource** + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "Node", + "name": "" +} +``` + +**Step 2: Add label to node** + +Modify returned JSON to add `.metadata.labels.cpumanager = "true"`, then update with `resources_create_or_update`. + +**VM Configuration:** +```yaml +spec: + template: + spec: + domain: + memory: + hugepages: + pageSize: 1Gi +``` + +**Benefits:** +- Reduces page-dirtying overhead during migration +- Improves memory access performance +- Faster migration completion for very large VMs (tested with 1TB VMs) + +### 3. Network Optimization + +**Set Network MTU to 9000** (jumbo frames): + +Configure in the NetworkAttachmentDefinition used for migration: + +```yaml +spec: + liveMigrationConfig: + network: migration-network # NetworkAttachmentDefinition with MTU 9000 +``` + +**Benefits:** +- Significantly improves network efficiency +- Reduces packet overhead +- Faster data transfer for large VM migrations + +**Validate MTU Setting Using MCP Tools:** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "k8s.cni.cncf.io/v1", + "kind": "NetworkAttachmentDefinition", + "namespace": "openshift-cnv", + "name": "migration-network" +} +``` + +Check `.spec.config` for MTU setting in the JSON configuration. + +### 4. Storage Configuration + +**For Testing/Development:** +- NFS-CSI with SSD backend storage +- Shared storage accessible from all nodes + +**For Production:** +- OpenShift Data Foundation (ODF) with SSD-backed storage +- Azure NetApp Files with appropriate performance tier +- NetApp ONTAP with dedicated SVM for virtualization workloads + +**Storage Validation Before Migration Using MCP Tools:** + +For each VM in rebalance plan: + +**Step 1: Get VM resource** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +**Step 2: Extract PVC names from `.spec.template.spec.volumes[].persistentVolumeClaim.claimName`** + +**Step 3: For each PVC, verify access mode** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "name": "" +} +``` + +**Expected**: `.spec.accessModes` must include `"ReadWriteMany"`. + +### 5. Concurrency Management + +**Default Limits:** +- **Cluster-wide**: 5 concurrent migrations +- **Per-node outbound**: 2 concurrent migrations + +**When to Increase:** +- Cluster has sufficient network bandwidth (100Gbps+ NICs) +- Dedicated migration network is configured +- Routine maintenance window with many VMs to migrate + +**When to Decrease:** +- Network saturation detected (monitor with Prometheus) +- Migration failures due to timeouts +- Shared application network (no dedicated migration network) + +**Monitoring Network Saturation Using MCP Tools:** + +**MCP Tool**: `nodes_stats_summary` (from openshift-virtualization) + +**Parameters**: +```json +{ + "name": "" +} +``` + +Review `.network.interfaces[].rxBytes` and `.network.interfaces[].txBytes` for throughput metrics. + +Alternatively, use `nodes_top` for current resource usage: + +**MCP Tool**: `nodes_top` (from openshift-virtualization) + +**Parameters**: +```json +{ + "name": "" +} +``` + +### 6. Pre-Migration Validation Checklist + +Before initiating migration: + +1. ✅ **Storage**: Verify all PVCs use ReadWriteMany (RWX) access mode +2. ✅ **Network**: Confirm all nodes are network-accessible +3. ✅ **Capacity**: Verify target node has sufficient CPU and memory +4. ✅ **Health**: Check `virt-handler` pods are Running (1/1) on all nodes +5. ✅ **Workload**: Consider VM workload intensity (reduce load if possible) +6. ✅ **Limits**: Check current cluster migration count < `parallelMigrationsPerCluster` + +**Validation Using MCP Tools:** + +**1. Check PVC Access Modes:** + +For each VM, use `resources_get` to get VirtualMachine, extract PVC names, then `resources_get` for each PVC and verify `.spec.accessModes` includes `"ReadWriteMany"`. + +**2. Check virt-handler Health:** + +**MCP Tool**: `pods_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "labelSelector": "kubevirt.io=virt-handler" +} +``` + +Filter results where `status.containerStatuses[0].ready == true`. All pods must show ready status. + +**3. Check Current Migration Count:** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstanceMigration" +} +``` + +Count results where `.status.phase` is NOT "Succeeded" or "Failed". Compare to `parallelMigrationsPerCluster` limit from HyperConverged CR. + +**4. Check Target Node Capacity:** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "Node", + "name": "" +} +``` + +Review `.status.allocatable` and `.status.capacity` for available CPU and memory. + +Alternatively use `nodes_top`: + +**MCP Tool**: `nodes_top` (from openshift-virtualization) + +**Parameters**: +```json +{ + "name": "" +} +``` + +--- + +## Test Results and Validation + +### SAP HANA 1TB VM Live Migration (Red Hat Developer Article 2025-07-14) + +**Test Environment:** +- **Hardware**: 8-socket Intel Xeon Platinum, 12TB memory, 100Gbps NICs +- **OpenShift**: 4.17.15 +- **VM Size**: 1TB memory (SAP HANA 2.00.081.00.1733303410) +- **Storage**: NFS-CSI with SSD backend +- **Network**: Dedicated 100Gbps secondary network, MTU 9000 + +**Results:** +- ✅ **Idle/Cooled-off**: Live migration completed successfully with **zero data loss or corruption** +- ✅ **High-load**: Migrations progressed as expected even with large volumes of dirty pages +- ✅ **Integrity**: Full VM and data integrity maintained; failed migrations safely canceled +- ⏱️ **Duration**: ~30-60 seconds for typical VMs; longer for 1TB VM under load + +**Key Findings:** +- Multi-fd technology enabled migrations to continue transferring data quickly while dirty pages were being generated +- Dedicated 100Gbps network critical for large VM migrations +- 1GB hugepages reduced page-dirtying overhead + +--- + +## Limitations and Constraints + +### Migration Requirements + +**MUST HAVE for Live Migration:** +- ReadWriteMany (RWX) storage on all VM volumes +- VM currently running (VirtualMachineInstance exists) +- Target node has sufficient capacity (CPU, memory) +- All nodes have RHCOS (not RHEL) + +**CANNOT Live Migrate When:** +- VM uses ReadWriteOnce (RWO) storage → Use cold migration +- VM is stopped (no VirtualMachineInstance) → Use cold migration or start VM first +- Target node is cordoned or NotReady → Choose different target +- Cluster at `parallelMigrationsPerCluster` limit → Wait for completion + +### Known Issues + +**Single Node OpenShift (SNO):** +- VMs created from common templates with `evictionStrategy: LiveMigrate` trigger `VMCannotBeEvicted` alert +- **Workaround**: Use `evictionStrategy: None` for SNO clusters + +**OVN-Kubernetes CNI:** +- Cannot attach Linux bridge or bonding device to host's default interface +- **Workaround**: Use secondary network interface or switch to OpenShift SDN CNI + +**MTU Differences:** +- OVS bridge default MTU: 1400 +- Linux bridge default MTU: 1500 +- **Impact**: May cause fragmentation; configure MTU explicitly + +--- + +## Troubleshooting Common Issues + +### Issue 1: Migration Timeout + +**Symptom:** +``` +Migration exceeded timeout: 150 seconds per GiB +``` + +**Causes:** +- High memory write rate (dirty page rate exceeds transfer rate) +- Insufficient network bandwidth +- Large VM memory size + +**Solutions:** + +**1. Increase timeout (temporary):** + +Use `resources_get` to fetch HyperConverged CR, modify `.spec.liveMigrationConfig`, then update with `resources_create_or_update`: + +```json +{ + "spec": { + "liveMigrationConfig": { + "completionTimeoutPerGiB": 1200, + "progressTimeout": 300 + } + } +} +``` + +**2. Reduce VM workload** during migration: +- Stop write-intensive processes temporarily +- Schedule migration during low-activity window + +**3. Use cold migration** instead (guaranteed completion - see REBALANCE_MANUAL.md) + +**4. Configure auto-converge** (cluster-level KubeVirt setting): +- Throttles vCPU to reduce dirty page rate +- Enables migration convergence for high write-rate VMs + +### Issue 2: Network Saturation + +**Symptom:** +- Multiple concurrent migrations slow or fail +- High network utilization on migration network + +**Solutions:** + +**1. Reduce concurrent migrations:** + +Use `resources_get` to fetch HyperConverged CR, modify `.spec.liveMigrationConfig.parallelMigrationsPerCluster`, then update with `resources_create_or_update`: + +```json +{ + "spec": { + "liveMigrationConfig": { + "parallelMigrationsPerCluster": 3 + } + } +} +``` + +**2. Set bandwidth limit per migration:** + +Modify `.spec.liveMigrationConfig.bandwidthPerMigration`: + +```json +{ + "spec": { + "liveMigrationConfig": { + "bandwidthPerMigration": "32Mi" + } + } +} +``` + +**3. Use dedicated migration network** (see Dedicated Migration Network section) + +### Issue 3: virt-handler Pods Not Ready + +**Symptom:** + +Using `pods_list` with `labelSelector: "kubevirt.io=virt-handler"`, some pods show `status.containerStatuses[0].ready == false`. + +**Causes:** +- Recent HyperConverged configuration change +- Network configuration error +- Node connectivity issue + +**Solutions:** + +**1. Wait for pod restart** (after config change): + +Pods restart automatically after HyperConverged update. Monitor using `pods_list` until all show ready status. + +**2. Check pod logs:** + +**MCP Tool**: `pods_log` (from openshift-virtualization) + +**Parameters**: +```json +{ + "name": "", + "namespace": "openshift-cnv", + "tail": 100 +} +``` + +**3. Verify node network configuration** (if using dedicated network): + +Use `resources_list` to check NodeNetworkConfigurationPolicy: + +**Parameters**: +```json +{ + "apiVersion": "nmstate.io/v1", + "kind": "NodeNetworkConfigurationPolicy" +} +``` + +And NetworkAttachmentDefinition: + +**Parameters**: +```json +{ + "apiVersion": "k8s.cni.cncf.io/v1", + "kind": "NetworkAttachmentDefinition", + "namespace": "openshift-cnv" +} +``` + +### Issue 4: Migration Rejected - Cluster Limit Reached + +**Symptom:** +``` +Migration rejected: cluster migration limit reached (5 concurrent) +``` + +**Solutions:** + +**1. Wait for ongoing migrations** to complete: + +Monitor using `resources_list`: + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstanceMigration" +} +``` + +Filter for migrations where `.status.phase` is NOT "Succeeded" or "Failed". + +**2. Increase cluster limit** (if network allows): + +Use `resources_get` to fetch HyperConverged CR, modify `.spec.liveMigrationConfig.parallelMigrationsPerCluster`, then update with `resources_create_or_update`: + +```json +{ + "spec": { + "liveMigrationConfig": { + "parallelMigrationsPerCluster": 10 + } + } +} +``` + +**3. Migrate VMs sequentially** instead of batch operation + +--- + +## Related Documentation + +- [Performance Tuning Guide](./performance-tuning.md) - Advanced tuning for migration performance +- [Anti-Patterns](./anti-patterns.md) - Common mistakes to avoid +- [Production Considerations](./production-considerations.md) - Right-sizing, workload planning, HA strategies +- [Troubleshooting: Scheduling Errors](../../../docs/troubleshooting/scheduling-errors.md) - ErrorUnschedulable after cold migration + +--- + +**Last Updated**: 2026-02-24 +**OpenShift Virtualization Versions**: 4.17, 4.18, 4.19, 4.20 +**Status**: Production-ready guidance from official Red Hat sources diff --git a/rh-virt/skills/vm-rebalance/references/performance-tuning.md b/rh-virt/skills/vm-rebalance/references/performance-tuning.md new file mode 100644 index 00000000..f1af24e9 --- /dev/null +++ b/rh-virt/skills/vm-rebalance/references/performance-tuning.md @@ -0,0 +1,719 @@ +# Performance Tuning for VM Rebalancing + +**Purpose**: Advanced performance tuning parameters, optimization strategies, and monitoring guidance for VM live migration and rebalancing operations. + +**When to consult this document**: When migrations are slow, when planning large-scale rebalancing, or when optimizing cluster performance for frequent migrations. + +--- + +## Official Sources + +This document is compiled from official Red Hat documentation: + +- [Live Migrating VMs with OpenShift Virtualization](https://developers.redhat.com/articles/2025/07/14/live-migrating-vms-openshift-virtualization) - Red Hat Developer (2025-07-14) +- [Best Practices for Virtual Machine Deployments on OpenShift Virtualization](https://learn.microsoft.com/en-us/azure/openshift/best-practices-openshift-virtualization) - Microsoft Azure Red Hat OpenShift (2026-02-16) +- [Announcing Right-Sizing for OpenShift Virtualization](https://developers.redhat.com/articles/2025/04/28/announcing-right-sizing-openshift-virtualization) - Red Hat Developer (2025-04-28) +- [Best Practices to Deploy VMs in Red Hat OpenShift Virtualization](https://docs.netapp.com/us-en/netapp-solutions-virtualization/openshift/os-osv-bpg.html) - NetApp Solutions + +--- + +## Right-Sizing Virtual Machines + +### Why Right-Sizing Matters for Rebalancing + +Properly sized VMs: +- Migrate faster (smaller memory footprint) +- Reduce network bandwidth requirements +- Improve cluster resource utilization +- Enable more efficient load balancing +- Prevent resource contention during migrations + +### Right-Sizing Methodology + +**Step 1: Define Health Metrics** + +Target healthy resource utilization ranges: + +| Resource | Target Range | Warning Threshold | Critical Threshold | +|----------|--------------|-------------------|-------------------| +| CPU Utilization | 60-70% average | >80% | >90% | +| Memory Pressure | <80% | >85% | >95% | +| Disk I/O Latency | <10ms | >50ms | >100ms | +| Network Throughput | <70% capacity | >80% | >90% | + +**Step 2: Monitor VM Resource Usage** + +**Using MCP Tools:** + +**MCP Tool**: `pods_top` (from openshift-virtualization) + +**Parameters**: +```json +{ + "all_namespaces": true, + "label_selector": "kubevirt.io=virt-launcher" +} +``` + +This returns CPU and memory usage for all VM launcher pods. Filter by specific namespace or VM name as needed. + +**For detailed metrics:** + +**MCP Tool**: `nodes_stats_summary` (from openshift-virtualization) + +**Parameters**: +```json +{ + "name": "" +} +``` + +Review `.pods[].containers[]` metrics for specific VM resource consumption including: +- `cpu.usageNanoCores` - Current CPU usage +- `memory.workingSetBytes` - Active memory usage +- `rootfs.usedBytes` - Disk usage + +**Step 3: Analyze Historical Data** + +Collect metrics over time (minimum 7 days for meaningful patterns): + +- Peak usage periods +- Resource saturation events +- Correlation between workload and resource consumption +- Trending (growing vs stable resource needs) + +**Step 4: Adjust VM Specifications** + +Based on observed metrics, resize VMs using `resources_get` and `resources_create_or_update`: + +**Example: Resize VM Memory** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +Modify `.spec.template.spec.domain.resources.requests.memory` based on usage analysis, then update with `resources_create_or_update`. + +**Avoid Overprovisioning:** +- Don't rely on on-premises sizing references +- Benchmark your actual workloads +- Consider OpenShift Virtualization architectural overhead (see Architectural Overhead section) + +--- + +## Architectural Overhead + +### OpenShift Virtualization Performance Characteristics + +Running VMs in OpenShift Virtualization introduces architectural overhead compared to bare metal or native pods: + +**Observed Performance (Azure Red Hat OpenShift with Standard_D96ds_v5 nodes, OpenShift 4.20, Virtualization 4.20):** + +| Workload Type | VM Performance | Pod Performance | Overhead | +|---------------|----------------|-----------------|----------| +| **Compute** (events/sec) | 525,022 | 546,997 | ~4% slower | +| **Compute** (latency ms) | 0.70 | 0.65 | ~8% higher latency | +| **Storage** (1 thread TPM) | 4,332 | 6,303 | ~31% slower | +| **Storage** (32 threads TPM) | 64,294 | 103,359 | ~38% slower | +| **Network** (64B, 1 thread Gbps) | 0.4 | 0.9 | ~56% slower | +| **Network** (1024B, 8 threads Gbps) | 24.7 | 28.9 | ~15% slower | + +**Key Takeaways:** +- Compute overhead is minimal (~4-8%) +- Storage and network have higher overhead (15-56% depending on workload) +- Multi-threaded workloads show better relative performance + +**Implications for Rebalancing:** +- VMs require more time to migrate than equivalent containerized workloads +- Plan capacity with overhead in mind (don't fill nodes to 100%) +- Network-intensive VMs benefit most from dedicated migration networks +- Consider workload characteristics when planning concurrent migrations + +--- + +## Tuned Configuration for High-Performance VMs + +### SAP HANA Tuning Example + +For database and high-performance workloads, apply tuned profiles to guest OS: + +**Tuned Profile (RHEL Guest):** + +```ini +[main] +summary=Optimize for SAP HANA and high-performance VMs + +[cpu] +force_latency=cstate.id:3|70 +governor=performance +energy_perf_bias=performance +min_perf_pct=100 + +[vm] +transparent_hugepages=never + +[sysctl] +# Semaphore limits +kernel.sem = 32000 1024000000 500 32000 + +# Disable NUMA balancing for predictable performance +kernel.numa_balancing = 0 + +# Scheduler tuning for low latency +kernel.sched_min_granularity_ns = 3000000 +kernel.sched_wakeup_granularity_ns = 4000000 + +# Memory management +vm.dirty_ratio = 40 +vm.dirty_background_ratio = 10 +vm.swappiness = 10 +``` + +**When to Apply:** +- Database VMs (PostgreSQL, MySQL, Oracle, SAP HANA) +- Real-time analytics workloads +- Low-latency trading platforms +- High-performance computing (HPC) VMs + +**Impact on Migration:** +- Reduces dirty page rate (faster convergence) +- More predictable migration times +- Better performance during and after migration + +--- + +## CPU and Memory Overcommit + +### Understanding Overcommit Ratios + +OpenShift Virtualization allows overcommit of CPU and memory resources, enabling higher VM density per node. + +**Default Overcommit Ratios:** +- CPU: No overcommit (1:1 mapping) +- Memory: No overcommit (1:1 mapping) + +**Recommended Production Limits (Red Hat Guidance):** +- **CPU Overcommit**: Maximum 1.8x physical cores +- **Memory Overcommit**: Maximum 0.9x physical memory + +**Consequences of Exceeding Limits:** +- **CPU**: Throttling causes slowness across all workloads on affected node +- **Memory**: OOM (Out of Memory) kills, VM crashes, data loss + +### Configuring Overcommit + +**Update HyperConverged CR using MCP Tools:** + +**Step 1: Get HyperConverged resource** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "hco.kubevirt.io/v1beta1", + "kind": "HyperConverged", + "namespace": "openshift-cnv", + "name": "kubevirt-hyperconverged" +} +``` + +**Step 2: Modify overcommit configuration** + +Add to `.spec.resourceRequirements`: + +```yaml +spec: + resourceRequirements: + vmiCPUAllocationRatio: 1.5 # Allow 1.5x CPU overcommit + vmiMemoryOvercommitPercent: 20 # Allow 20% memory overcommit +``` + +**Step 3: Update using `resources_create_or_update`** + +**Best Practices:** +- Use conservative overcommit for production (1.2x CPU max, 10% memory max) +- Use higher overcommit for dev/test (1.8x CPU, 20% memory acceptable) +- Monitor node resource usage closely after enabling overcommit +- Adjust based on actual VM behavior patterns + +**Impact on Rebalancing:** +- Higher overcommit = more VMs per node = longer migration times +- Rebalancing may be needed more frequently with overcommit +- Target node capacity calculations must account for overcommit ratios + +--- + +## Network Performance Tuning + +### MTU Configuration + +**Why MTU Matters:** +- Default MTU (1500 bytes) causes fragmentation for large data transfers +- Jumbo frames (MTU 9000) significantly improve network efficiency +- Critical for large VM migrations (>100GB memory) + +**Set MTU in NetworkAttachmentDefinition:** + +When creating dedicated migration network, include MTU setting: + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Parameters** (excerpt): +```json +{ + "resource": "apiVersion: k8s.cni.cncf.io/v1\nkind: NetworkAttachmentDefinition\nmetadata:\n name: migration-network\n namespace: openshift-cnv\nspec:\n config: '{\n \"cniVersion\": \"0.3.1\",\n \"name\": \"migration-bridge\",\n \"type\": \"macvlan\",\n \"master\": \"eth1\",\n \"mode\": \"bridge\",\n \"mtu\": 9000,\n \"ipam\": {...}\n }'" +} +``` + +**Validate MTU:** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "k8s.cni.cncf.io/v1", + "kind": "NetworkAttachmentDefinition", + "namespace": "openshift-cnv", + "name": "migration-network" +} +``` + +Check `.spec.config` for `"mtu": 9000`. + +### NAPI and Multiqueue Tuning + +For network-intensive workloads, enable multiqueue virtio-net: + +**VM Configuration:** +```yaml +spec: + template: + spec: + domain: + devices: + interfaces: + - name: default + model: virtio + masquerade: {} + ports: + - port: 80 + networkInterfaceMultiqueue: true # Enable multiqueue +``` + +**Benefits:** +- Parallelizes network processing across multiple vCPUs +- Improves throughput for high-bandwidth workloads +- Reduces latency for network-intensive applications + +**When to Use:** +- VMs with >4 vCPUs +- High network throughput requirements (>10Gbps) +- Web servers, load balancers, network appliances + +--- + +## Storage Performance Optimization + +### Storage Class Selection + +Different storage backends have different performance characteristics: + +| Storage Backend | IOPS | Latency | Throughput | Best For | +|-----------------|------|---------|------------|----------| +| OpenShift Data Foundation (ODF) | High | Low (<5ms) | Very High | General purpose, production | +| Azure NetApp Files (ANF) Premium | Very High | Very Low (<1ms) | Very High | Database, high-performance | +| NFS-CSI (SSD-backed) | Medium | Medium (5-10ms) | High | Dev/test, general use | +| AWS EBS gp3 | Medium | Medium (10-20ms) | Medium | Cost-effective, RWO only | + +**Check Storage Class Performance:** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "storage.k8s.io/v1", + "kind": "StorageClass", + "name": "" +} +``` + +Review `.parameters` for performance tier, provisioning type, and backend configuration. + +### Storage Limits (NetApp ONTAP) + +When using NetApp storage backends, configure limits to prevent resource exhaustion: + +**SVM Volume Limits:** + +Set maximum volumes per SVM to prevent Trident from consuming all storage capacity. + +**Storage Quotas:** + +Implement storage limits on SVMs to enforce resource boundaries. + +**Trident Backend Parameters:** + +Configure in Trident backend definition: +- `limitVolumeSize`: Maximum individual volume size (e.g., "100Gi") +- `limitVolumePoolSize`: Maximum FlexVol size for economy drivers (e.g., "500Gi") + +**Impact on Rebalancing:** +- Storage limits prevent VMs from growing unbounded +- Predictable storage capacity aids in target node selection +- Quota enforcement ensures fair resource distribution + +--- + +## Migration Bandwidth Management + +### Bandwidth Per Migration + +**Purpose**: Limit bandwidth consumption per migration to prevent network saturation. + +**Default**: Unlimited (no bandwidth limit) + +**When to Set:** +- Shared application network (no dedicated migration network) +- Multiple concurrent migrations planned +- Network capacity constraints + +**Configuration Using MCP Tools:** + +**Step 1: Get HyperConverged resource** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "hco.kubevirt.io/v1beta1", + "kind": "HyperConverged", + "namespace": "openshift-cnv", + "name": "kubevirt-hyperconverged" +} +``` + +**Step 2: Set bandwidth limit** + +Modify `.spec.liveMigrationConfig.bandwidthPerMigration`: + +```yaml +spec: + liveMigrationConfig: + bandwidthPerMigration: 64Mi # 64 MiB/s per migration +``` + +Common values: +- `32Mi` - Conservative (256 Mbps) +- `64Mi` - Default (512 Mbps) +- `128Mi` - High bandwidth (1 Gbps) +- Omit field for unlimited + +**Step 3: Update using `resources_create_or_update`** + +**Monitoring Bandwidth Usage:** + +**MCP Tool**: `nodes_stats_summary` (from openshift-virtualization) + +**Parameters**: +```json +{ + "name": "" +} +``` + +Review `.network.interfaces[].rxBytes` and `.network.interfaces[].txBytes` for current throughput. + +**Tuning Guidance:** +- Start conservative (32-64Mi) and increase if migrations are slow +- Monitor network utilization during migrations +- Unlimited bandwidth is acceptable with dedicated migration network + +--- + +## Concurrency Limits Tuning + +### Parallel Migrations Per Cluster + +**Default**: 5 concurrent migrations cluster-wide + +**When to Increase:** +- Dedicated migration network with high bandwidth (100Gbps) +- Routine maintenance windows requiring many migrations +- Cluster has >20 nodes + +**When to Decrease:** +- Network saturation observed +- Migration failures due to timeouts +- Shared application network + +**Configuration:** + +Modify `.spec.liveMigrationConfig.parallelMigrationsPerCluster` in HyperConverged CR: + +```yaml +spec: + liveMigrationConfig: + parallelMigrationsPerCluster: 10 # Increase from default 5 +``` + +**Conservative**: 3-5 migrations +**Moderate**: 5-10 migrations +**Aggressive**: 10-20 migrations (requires dedicated network) + +### Parallel Outbound Migrations Per Node + +**Default**: 2 concurrent outbound migrations per source node + +**Recommendation**: Keep at 2 to prevent single-node overload. + +**Why 2 is Optimal:** +- Prevents source node CPU/memory saturation +- Limits network bandwidth consumption per node +- Avoids cascading performance degradation +- Tested and validated by Red Hat + +**Only increase to 3-4 if:** +- Node has very high CPU/memory headroom (>50% free) +- Dedicated high-bandwidth migration network +- Extensive testing validates stability + +--- + +## Timeout Configuration + +### Completion Timeout Per GiB + +**Default**: 800 seconds per GiB of VM memory + +**Calculation**: For a 16GB VM, timeout = 16 * 800 = 12,800 seconds (~3.5 hours) + +**When to Increase:** +- High dirty page rate workloads (databases, caching systems) +- VMs with >100GB memory +- Network bandwidth constraints + +**When to Decrease:** +- Fast dedicated migration network (100Gbps) +- Low dirty page rate (mostly idle VMs) +- Want faster failure detection + +**Configuration:** + +Modify `.spec.liveMigrationConfig.completionTimeoutPerGiB` in HyperConverged CR: + +```yaml +spec: + liveMigrationConfig: + completionTimeoutPerGiB: 1200 # Increase for large/busy VMs +``` + +**Tuning by Workload Type:** + +| Workload Type | Recommended Timeout | Rationale | +|---------------|---------------------|-----------| +| Database (write-heavy) | 1200-1600s | High dirty page rate | +| Web server (mostly read) | 600-800s | Low dirty page rate | +| Caching (Redis/Memcached) | 1600-2000s | Very high dirty page rate | +| General purpose | 800s (default) | Balanced | + +### Progress Timeout + +**Default**: 150 seconds without progress before cancellation + +**Purpose**: Detects stuck migrations and fails fast rather than hanging indefinitely. + +**When to Increase:** +- Very large VMs (>500GB memory) +- Slow networks (<1Gbps) +- Initial memory copy takes >2 minutes + +**When to Decrease:** +- Want faster failure detection +- Prefer to retry quickly rather than wait + +**Configuration:** + +Modify `.spec.liveMigrationConfig.progressTimeout` in HyperConverged CR: + +```yaml +spec: + liveMigrationConfig: + progressTimeout: 300 # 5 minutes without progress +``` + +**Recommended Values:** +- Small VMs (<50GB): 150s (default) +- Medium VMs (50-200GB): 200-300s +- Large VMs (>200GB): 300-600s + +--- + +## Monitoring and Observability + +### Key Metrics to Monitor + +**During Rebalancing Operations:** + +1. **Migration Progress** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstanceMigration" +} +``` + +Monitor `.status.phase` for each migration (Pending → Scheduling → PreparingTarget → Running → Succeeded). + +2. **Node Resource Usage** + +**MCP Tool**: `nodes_top` (from openshift-virtualization) + +**Parameters**: +```json +{ + "name": "" +} +``` + +Track CPU and memory utilization before, during, and after migrations. + +3. **Network Throughput** + +**MCP Tool**: `nodes_stats_summary` (from openshift-virtualization) + +**Parameters**: +```json +{ + "name": "" +} +``` + +Review `.network.interfaces[]` metrics for bandwidth usage. + +4. **VM Health** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "namespace": "", + "name": "" +} +``` + +Check `.status.conditions[]` for VM health status. + +### Performance Benchmarking + +**Before Rebalancing:** +- Establish baseline performance metrics +- Document current resource utilization +- Identify performance-sensitive VMs + +**During Rebalancing:** +- Monitor migration duration +- Track network bandwidth consumption +- Watch for resource contention + +**After Rebalancing:** +- Validate improved load distribution +- Confirm no performance degradation +- Document improvements achieved + +**Tools for Benchmarking:** +- Apache JMeter (web application load testing) +- stress-ng (CPU/memory stress testing) +- fio (storage I/O benchmarking) +- iperf3 (network throughput testing) + +--- + +## Scaling Strategies + +### Scale Out vs Scale Up + +**Scale Out** (add more nodes): +- **Pros**: Better fault tolerance, more migration targets, horizontal capacity growth +- **Cons**: Higher complexity, more licensing costs, requires cluster expansion + +**Scale Up** (larger node sizes): +- **Pros**: Simpler management, fewer migration hops, better resource consolidation +- **Cons**: Larger blast radius, limited by maximum instance size, single point of failure risk + +**For Demanding Workloads:** + +From Microsoft Azure Red Hat OpenShift guidance: +> "Scale out or up for demanding workloads: Add more nodes or upsize the nodes in your Azure Red Hat OpenShift cluster for high concurrency or resource-intensive applications." + +**Recommendation:** +- Start with scale-up to minimum 8-core Azure VMs (per OpenShift Virtualization requirements) +- Scale-out when individual nodes exceed 70-80% sustained utilization +- Balance between node size and cluster size for optimal resilience + +### Node Pool Strategy + +**Workload-Specific Node Pools:** + +Create dedicated node pools for different VM workload types using labels, taints, and tolerations: + +**Example Node Pool Configuration:** + +**Pool 1: General VMs** +- Node labels: `workload-type=general` +- Node taints: None +- VM tolerations: Not required + +**Pool 2: High-Performance VMs** +- Node labels: `workload-type=high-performance` +- Node taints: `performance=dedicated:NoSchedule` +- VM tolerations: Match taint + +**Pool 3: GPU Workloads** +- Node labels: `workload-type=gpu` +- Node taints: `nvidia.com/gpu=present:NoSchedule` +- VM tolerations: Match taint + +**Apply Labels to Nodes Using MCP Tools:** + +**MCP Tool**: `resources_get` then `resources_create_or_update` (from openshift-virtualization) + +Get node, modify `.metadata.labels`, then update. + +**Benefits:** +- Simplifies maintenance (drain entire pool) +- Limits blast radius (failures contained to pool) +- Improves resource efficiency (right-sized pools) +- Enables topology spread rules (VMs across zones/pools) + +--- + +## Related Documentation + +- [Live Migration Best Practices](./live-migration-best-practices.md) - Configuration parameters and requirements +- [Anti-Patterns](./anti-patterns.md) - Common mistakes to avoid +- [Production Considerations](./production-considerations.md) - Right-sizing, workload planning, HA strategies + +--- + +**Last Updated**: 2026-02-24 +**OpenShift Virtualization Versions**: 4.17, 4.18, 4.19, 4.20 +**Status**: Production-ready guidance from official Red Hat sources diff --git a/rh-virt/skills/vm-rebalance/references/production-considerations.md b/rh-virt/skills/vm-rebalance/references/production-considerations.md new file mode 100644 index 00000000..919280a8 --- /dev/null +++ b/rh-virt/skills/vm-rebalance/references/production-considerations.md @@ -0,0 +1,868 @@ +# Production Considerations for VM Rebalancing + +**Purpose**: Production deployment guidance, workload planning, high availability strategies, and operational best practices for VM rebalancing in OpenShift Virtualization. + +**When to consult this document**: Before deploying to production, when planning capacity, or when designing HA/DR strategies. + +--- + +## Official Sources + +This document is compiled from official Red Hat documentation: + +- [Best Practices for Virtual Machine Deployments on OpenShift Virtualization](https://learn.microsoft.com/en-us/azure/openshift/best-practices-openshift-virtualization) - Microsoft Azure Red Hat OpenShift (2026-02-16) +- [Announcing Right-Sizing for OpenShift Virtualization](https://developers.redhat.com/articles/2025/04/28/announcing-right-sizing-openshift-virtualization) - Red Hat Developer (2025-04-28) +- [Best Practices to Deploy VMs in Red Hat OpenShift Virtualization](https://docs.netapp.com/us-en/netapp-solutions-virtualization/openshift/os-osv-bpg.html) - NetApp Solutions +- [OpenShift Virtualization Best Practices](https://trilio.io/openshift-virtualization/) - Trilio + +--- + +## Workload Identification and Categorization + +### Common Workload Types + +Before provisioning VMs, categorize workloads to determine performance and resource requirements: + +| Workload Type | Characteristics | Resource Profile | Migration Considerations | +|---------------|----------------|------------------|--------------------------| +| **General Purpose** | Web servers, app servers, CMS | Moderate CPU/memory | Easy to migrate, low dirty page rate | +| **Database** | RDBMS, NoSQL | High CPU, memory, consistent IOPS | High dirty page rate; schedule migrations carefully | +| **Real-time Analytics** | Operational dashboards | Low latency, high throughput | Sensitive to migration pause; use dedicated network | +| **AI/ML** | Training, inference | Very high CPU/GPU, memory | Large memory footprint; long migration times | +| **Data Streaming** | Event-driven architectures | High throughput, low latency | Network-intensive; avoid concurrent migrations | +| **Batch Processing** | Periodic jobs | Variable resources | Migrate during job idle periods | +| **HPC** | Scientific simulations | Very high CPU, memory | Extremely long migrations; consider cold migration | +| **Edge/IoT** | Sensor aggregation | Low resources | Easy to migrate, scale horizontally | +| **Media Processing** | Encoding, streaming | High CPU, network | High dirty page rate during processing | +| **Dev/Test** | Development environments | Variable | Higher overcommit acceptable | + +### Workload Assessment Using MCP Tools + +**Step 1: Inventory Current VMs** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine" +} +``` + +**Step 2: Analyze Resource Usage** + +**MCP Tool**: `pods_top` (from openshift-virtualization) + +**Parameters**: +```json +{ + "all_namespaces": true, + "label_selector": "kubevirt.io=virt-launcher" +} +``` + +**Step 3: Categorize by Usage Pattern** + +Group VMs by observed characteristics: +- CPU-intensive: >70% CPU utilization +- Memory-intensive: >80% memory utilization +- I/O-intensive: High storage throughput +- Network-intensive: High network bandwidth + +**Step 4: Plan Rebalancing Strategy** + +Based on workload type: +- **CPU-intensive**: Balance CPU across nodes +- **Memory-intensive**: Balance memory across nodes +- **I/O-intensive**: Distribute across different storage backends +- **Network-intensive**: Stagger migrations to avoid saturation + +--- + +## Right-Sizing Virtual Machines for Production + +### Health Metrics Definition + +Establish target ranges for healthy resource utilization: + +**CPU Utilization:** +- **Target**: 60-70% average usage +- **Warning**: >80% sustained +- **Critical**: >90% sustained +- **Action**: Scale up VM or rebalance to less loaded node + +**Memory Pressure:** +- **Target**: 70-80% utilization +- **Warning**: >85% with swap activity +- **Critical**: >95% or OOM events +- **Action**: Increase VM memory or reduce workload + +**Disk I/O:** +- **Target**: <10ms latency, <70% queue depth +- **Warning**: >50ms latency +- **Critical**: >100ms latency or queue saturation +- **Action**: Move to faster storage tier or distribute workload + +**Network Throughput:** +- **Target**: <70% interface capacity +- **Warning**: >80% sustained +- **Critical**: >90% or packet loss +- **Action**: Enable multiqueue, use faster NICs, rebalance + +### Monitoring Setup Using MCP Tools + +**VM-Level Metrics:** + +**MCP Tool**: `pods_top` (from openshift-virtualization) + +Provides current CPU and memory usage for VM launcher pods. + +**Node-Level Metrics:** + +**MCP Tool**: `nodes_top` (from openshift-virtualization) + +Shows aggregate node resource consumption. + +**Detailed Statistics:** + +**MCP Tool**: `nodes_stats_summary` (from openshift-virtualization) + +**Parameters**: +```json +{ + "name": "" +} +``` + +Provides comprehensive metrics including: +- Per-pod resource usage +- Container-level metrics +- Filesystem usage +- Network interface statistics +- PSI (Pressure Stall Information) metrics on cgroup v2 systems + +### Sizing Recommendations by Workload + +**Database Workloads:** +- Start with: 4-8 vCPU, 16-32Gi memory +- Storage: Premium SSD or NVMe with RWX support +- Network: Enable multiqueue virtio-net +- Special: Consider dedicated CPU placement (`dedicatedCpuPlacement: true`) + +**Web Servers:** +- Start with: 2-4 vCPU, 4-8Gi memory +- Storage: Standard SSD acceptable +- Network: Standard configuration sufficient +- Special: Scale horizontally rather than vertically + +**AI/ML Workloads:** +- Start with: 8-16 vCPU, 32-64Gi memory +- Storage: High-performance SSD +- Network: High bandwidth (consider dedicated migration network) +- Special: GPU support currently not available (plan accordingly) + +**Dev/Test Environments:** +- Start with: 1-2 vCPU, 2-4Gi memory +- Storage: Standard tier acceptable +- Network: Standard configuration +- Special: Higher overcommit ratios acceptable (1.5-1.8x CPU) + +### Minimum Requirements + +From Microsoft Azure Red Hat OpenShift documentation: +> "Minimum core requirement: OpenShift Virtualization requires a minimum of eight (8) core Azure VMs for OpenShift worker nodes." + +**Implications for Rebalancing:** +- Worker nodes must have ≥8 cores +- Plan VM placement considering this minimum +- Avoid creating nodes smaller than this threshold + +--- + +## High Availability Strategies + +### VM-Level High Availability + +**RunStrategy Configuration:** + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: critical-app +spec: + runStrategy: Always # Ensures VM restarts after failures +``` + +**RunStrategy Options:** + +| Strategy | Behavior | Use Case | +|----------|----------|----------| +| `Always` | VM runs continuously; restarts on failure | Production VMs requiring HA | +| `RerunOnFailure` | Restarts only if VM crashes | Batch workloads | +| `Manual` | User controls start/stop | Dev/test VMs | +| `Halted` | VM stays stopped | Maintenance, cold storage | + +**Eviction Strategy:** + +OpenShift Virtualization automatically sets `evictionStrategy` to `LiveMigrate` for VMs with RWX storage: + +```yaml +spec: + template: + spec: + evictionStrategy: LiveMigrate # Automatically set for RWX VMs +``` + +**Note for Single Node OpenShift (SNO):** + +From known issues: +> "In a Single Node OpenShift (SNO) cluster, a VMCannotBeEvicted alert occurs on virtual machines created from common templates that have the eviction strategy set to LiveMigrate." + +**Workaround**: Use `evictionStrategy: None` for SNO clusters. + +### Pod Anti-Affinity for VM Replicas + +For critical applications, deploy multiple VM replicas with anti-affinity: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: web-server-replica-1 + labels: + app: web-server +spec: + template: + metadata: + labels: + app: web-server + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app + operator: In + values: + - web-server + topologyKey: kubernetes.io/hostname # Different nodes +``` + +**Benefits:** +- VMs distributed across different failure domains +- Node failure affects only one replica +- Improves overall availability + +**Verify Distribution Using MCP Tools:** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachineInstance", + "labelSelector": "app=web-server" +} +``` + +Check `.status.nodeName` for each instance to confirm distribution. + +### Machine Health Checks + +**Critical for Automatic Failover:** + +From Red Hat documentation: +> "If a node fails and machine health checks are not deployed on your cluster, virtual machines (VMs) with RunStrategy: Always configured are not automatically relocated to healthy nodes." + +**Deploy Machine Health Checks:** + +Configure at cluster level to detect and remediate node failures. This enables automatic VM recovery without manual intervention. + +**Monitor Node Health:** + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "Node" +} +``` + +Filter for nodes where `.status.conditions[]` shows unhealthy states (`Ready=False`, `DiskPressure=True`, `MemoryPressure=True`). + +### Topology Spread for Zone Resilience + +For multi-zone clusters, use topology spread rules: + +```yaml +spec: + template: + spec: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + app: critical-app +``` + +**Benefits:** +- VMs spread across availability zones +- Zone failure doesn't affect all replicas +- Improved disaster recovery + +--- + +## Capacity Planning + +### Cluster Sizing Approach + +**Scale Out vs Scale Up:** + +From Microsoft Azure Red Hat OpenShift guidance: +> "Scale out or up for demanding workloads: Add more nodes or upsize the nodes in your Azure Red Hat OpenShift cluster for high concurrency or resource-intensive applications." + +**Scale Out (add more nodes):** +- **Pros**: Better fault tolerance, horizontal growth, more migration targets +- **Cons**: Higher complexity, more license costs, requires cluster expansion + +**When to Scale Out:** +- Current nodes consistently >70-80% utilized +- Need more fault isolation +- Planning for growth +- HA requirements mandate distribution + +**Scale Up (larger node sizes):** +- **Pros**: Simpler management, better resource consolidation, fewer migration hops +- **Cons**: Larger blast radius, limited by max instance size, single point of failure risk + +**When to Scale Up:** +- VMs don't fit on existing nodes +- Few large VMs rather than many small VMs +- Simplicity valued over distribution + +**Recommended Approach:** +1. Start with moderate node sizes (8-16 cores) +2. Scale out to 3-5 nodes minimum for HA +3. Scale up only when specific VMs require larger nodes +4. Maintain headroom (30-40% free capacity) for migrations and failures + +### Node Pool Strategy + +**Create workload-specific pools using labels and taints:** + +**Pool Configuration Example:** + +**General VM Pool:** +- Node size: 8-16 cores, 32-64GB RAM +- Labels: `workload-type=general` +- No taints (default scheduling) + +**High-Performance Pool:** +- Node size: 16-32 cores, 64-128GB RAM +- Labels: `workload-type=high-performance`, `cpumanager=true` +- Taints: `performance=dedicated:NoSchedule` + +**GPU Pool (future):** +- Node size: GPU-enabled instances +- Labels: `workload-type=gpu` +- Taints: `nvidia.com/gpu=present:NoSchedule` + +**Configure Labels Using MCP Tools:** + +**MCP Tool**: `resources_get` then `resources_create_or_update` (from openshift-virtualization) + +**Step 1: Get Node** + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "Node", + "name": "" +} +``` + +**Step 2: Add Labels** + +Modify `.metadata.labels`: +```json +{ + "workload-type": "high-performance", + "cpumanager": "true" +} +``` + +**Step 3: Update Node** + +Use `resources_create_or_update` with modified resource. + +**Benefits of Node Pools:** +- Simplifies maintenance (drain entire pool) +- Limits blast radius (failures contained) +- Improves efficiency (right-sized for workload) +- Enables topology spread rules + +### Capacity Headroom + +**Reserve capacity for:** +- Node failures (n-1 redundancy minimum) +- VM migrations (target nodes need free resources) +- Burst workloads (temporary spikes) +- New VM deployments + +**Recommended Headroom:** +- **Production**: 30-40% free capacity cluster-wide +- **Dev/Test**: 20-30% free capacity +- **Minimum**: 20% free capacity (below this, rebalancing becomes difficult) + +**Monitor Capacity Using MCP Tools:** + +**MCP Tool**: `nodes_top` (from openshift-virtualization) + +Calculate cluster-wide utilization: +``` +Total CPU Used / Total CPU Capacity = Cluster CPU Utilization +Total Memory Used / Total Memory Capacity = Cluster Memory Utilization +``` + +**Action Thresholds:** +- <70%: Healthy headroom +- 70-80%: Plan for expansion +- >80%: Add nodes urgently +- >90%: Emergency capacity issue + +--- + +## Storage Planning for Production + +### Storage Backend Selection + +**OpenShift Data Foundation (ODF):** +- **Best for**: General purpose, production workloads +- **Performance**: High IOPS, low latency (<5ms) +- **RWX Support**: Yes +- **Considerations**: Requires dedicated storage nodes; use taints/tolerations to isolate ODF workload + +**Azure NetApp Files (ANF):** +- **Best for**: High-performance databases, latency-sensitive apps +- **Performance**: Very high IOPS, very low latency (<1ms) +- **RWX Support**: Yes +- **Considerations**: Choose performance tier based on workload requirements + +**NFS-CSI (SSD-backed):** +- **Best for**: Dev/test, general use +- **Performance**: Medium IOPS, medium latency (5-10ms) +- **RWX Support**: Yes +- **Considerations**: Cost-effective, sufficient for non-critical workloads + +**AWS EBS gp3:** +- **Best for**: Cost-effective storage +- **Performance**: Medium IOPS, medium latency (10-20ms) +- **RWX Support**: No (RWO only) +- **Considerations**: Cannot use live migration; cold migration only + +### Storage QoS and Limits + +**NetApp ONTAP QoS:** + +From NetApp documentation: +> "Apply QoS policies to SVMs to limit the number of IOPS consumable by the Trident provisioned volumes." + +**Why QoS Matters:** +- Prevents one VM from starving others +- Protects non-Trident workloads from VM I/O impact +- Enforces fair resource distribution +- Predictable performance for all VMs + +**SVM Isolation:** + +From NetApp documentation: +> "Establish dedicated Storage Virtual Machines (SVMs) to provide isolation and administrative separation between tenants." + +**Benefits:** +- Tenant isolation +- Privilege delegation +- Resource quota enforcement +- Security boundary + +### Storage Validation Before Rebalancing + +**For Live Migration, verify RWX storage:** + +For each VM: + +**Step 1: Get VM** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +**Step 2: Extract PVC Names** + +From `.spec.template.spec.volumes[].persistentVolumeClaim.claimName`. + +**Step 3: Check PVC Access Mode** + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "v1", + "kind": "PersistentVolumeClaim", + "namespace": "", + "name": "" +} +``` + +**Required**: `.spec.accessModes` must include `"ReadWriteMany"`. + +--- + +## Network Planning + +### Dedicated Migration Network + +**Production Requirement:** + +For production clusters with large VMs or frequent migrations, a dedicated migration network is **highly recommended**. + +**Benefits:** +- Isolates migration traffic from applications +- Enables 100Gbps bandwidth for large VM migrations +- Prevents network contention +- Improves security (separate VLAN) + +**Implementation:** + +See [live-migration-best-practices.md](./live-migration-best-practices.md) for complete configuration steps. + +**Key Components:** +1. Secondary physical NIC or VLAN +2. NodeNetworkConfigurationPolicy (NNCP) +3. NetworkAttachmentDefinition (NAD) +4. HyperConverged CR configuration + +### Network Performance Tuning + +**MTU Configuration:** + +Set to 9000 (jumbo frames) for migration networks: + +```json +{ + "cniVersion": "0.3.1", + "name": "migration-bridge", + "type": "macvlan", + "mtu": 9000, + "ipam": {...} +} +``` + +**Multiqueue virtio-net:** + +Enable for VMs with >4 vCPUs and high network throughput: + +```yaml +spec: + template: + spec: + domain: + devices: + interfaces: + - name: default + model: virtio + networkInterfaceMultiqueue: true +``` + +### Network Isolation + +**Namespace Separation:** + +From best practices: +> "Use Namespaces to provide logical boundary for resources." + +**Pod Security Policies:** + +Disable privileged container capabilities for VM launcher pods to enhance security. + +**Separate Export Policies:** + +For NFS storage, implement separate export policies for infrastructure nodes vs application nodes. + +--- + +## Operational Best Practices + +### Phased Production Rollout + +From best practices: +> "Begin with non-critical or dev/test workloads before moving production systems - this phased approach allows teams to gain hands-on experience while minimizing risk." + +**Recommended Phases:** + +**Phase 1: Development/Test (2-4 weeks)** +- Deploy 5-10 test VMs +- Validate performance vs expectations +- Test live and cold migration workflows +- Benchmark resource overhead +- Train operations team + +**Phase 2: Non-Critical Production (4-8 weeks)** +- Migrate non-critical workloads (internal tools, QA environments) +- Monitor stability and performance +- Refine sizing and configurations +- Build runbooks and procedures +- Establish monitoring and alerting + +**Phase 3: Critical Production (Ongoing)** +- Migrate critical workloads in prioritized order +- Ensure HA and DR fully configured +- 24/7 monitoring and on-call support +- Document rollback procedures +- Conduct regular DR tests + +### Start Small, Scale Gradually + +From best practices: +> "When starting with OpenShift Virtualization, it's essential to start small and scale up as needed to avoid over-provisioning and wasting resources." + +**Growth Strategy:** + +**Month 1-2: Pilot** +- 5-10 VMs +- Single workload type +- Limited users +- Focus on learning + +**Month 3-6: Expansion** +- 20-50 VMs +- Multiple workload types +- Broader user base +- Refine processes + +**Month 6-12: Production Scale** +- 50-200+ VMs +- All workload types +- Organization-wide +- Mature operations + +**Benefits:** +- Avoids over-provisioning +- Iterative learning +- Cost-effective growth +- Risk mitigation + +### Monitoring and Alerting + +**Key Metrics to Monitor:** + +**VM Health:** +- Status (Running, Stopped, Error) +- Resource utilization (CPU, memory, disk, network) +- Guest agent connectivity +- Migration status + +**Node Health:** +- Resource utilization +- virt-handler pod status +- Network connectivity +- Storage backend health + +**Cluster Health:** +- Current migration count vs limits +- HyperConverged CR status +- Storage capacity and performance +- Network saturation + +**Migration Operations:** +- Success/failure rate +- Average migration duration +- Timeout occurrences +- Concurrent migration count + +**Alert Thresholds:** + +| Metric | Warning | Critical | +|--------|---------|----------| +| Node CPU | >80% | >90% | +| Node Memory | >85% | >95% | +| Migration Failures | >10% | >25% | +| virt-handler Pods Not Ready | Any | >1 | +| Cluster Migration Limit | >80% (4/5) | At limit (5/5) | + +### Backup and Disaster Recovery + +**VM Snapshots:** + +Use vm-snapshot skills for point-in-time backups before risky operations: +- Before major migrations +- Before configuration changes +- Before OS upgrades in guest +- Regular backup schedule (daily/weekly) + +**Disaster Recovery Planning:** + +**Multi-Zone Deployment:** +- Distribute VMs across availability zones +- Use topology spread constraints +- Configure zone-resilient storage + +**Backup Strategy:** +- Regular VM snapshots +- Export critical VM definitions +- Document restore procedures +- Test DR scenarios quarterly + +**RTO/RPO Targets:** + +Define recovery objectives: +- **RTO** (Recovery Time Objective): How quickly must VMs be recovered? +- **RPO** (Recovery Point Objective): How much data loss is acceptable? + +**Example Targets:** + +| Workload Tier | RTO | RPO | Strategy | +|---------------|-----|-----|----------| +| Critical | <15 min | <5 min | Multi-zone HA, frequent snapshots | +| Important | <1 hour | <1 hour | Daily snapshots, documented restore | +| Standard | <4 hours | <24 hours | Weekly snapshots, manual restore | + +--- + +## Cost Optimization + +### Resource Efficiency + +**Avoid Overprovisioning:** + +From Microsoft Azure Red Hat OpenShift guidance: +> "Avoid overprovisioning by aligning resources with actual usage patterns." + +**Cost Factors:** +- Azure compute costs (worker node instances) +- OpenShift licensing +- VM operating system licensing +- Storage costs (capacity and performance tier) +- Network egress charges + +**Optimization Strategies:** + +**1. Right-size VMs based on actual usage** + +Monitor with `pods_top` and resize VMs that are consistently under-utilized. + +**2. Use appropriate storage tiers** + +Don't use Premium storage for dev/test VMs; match tier to workload requirements. + +**3. Implement auto-scaling** + +For workloads with variable demand, use horizontal scaling rather than over-provisioning. + +**4. Consolidate with overcommit** + +In dev/test environments, use higher overcommit ratios (1.5-1.8x CPU) to maximize density. + +**5. Schedule non-critical VMs** + +Stop dev/test VMs during off-hours to reduce costs. + +### Load Balancing for Efficiency + +**Rebalancing Improves Efficiency:** +- Prevents hotspots (overloaded nodes) +- Enables better resource utilization +- Reduces need for emergency node additions +- Extends hardware lifespan (even wear) + +**Regular Rebalancing Schedule:** +- **Weekly**: Review node utilization, plan migrations if imbalance detected +- **Monthly**: Comprehensive rebalancing to optimize distribution +- **Quarterly**: Capacity planning and infrastructure rightsizing + +--- + +## Security Considerations + +### Tenant Isolation + +**Namespace Separation:** + +Deploy VMs for different tenants/teams in separate namespaces. + +**Network Policies:** + +Implement NetworkPolicies to restrict inter-VM communication: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: tenant-isolation + namespace: tenant-a +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress + ingress: + - from: + - namespaceSelector: + matchLabels: + tenant: tenant-a +``` + +**RBAC:** + +Grant users permissions only for their namespace's VMs, not cluster-wide access. + +### VM Security Hardening + +**Guest OS Security:** +- Regular patching and updates +- Disable unnecessary services +- Configure firewall rules +- Enable SELinux/AppArmor + +**Secrets Management:** +- Use Kubernetes Secrets for credentials +- Inject secrets into VMs via cloud-init +- Rotate secrets regularly +- Never store secrets in VM images + +**Access Control:** +- SSH key authentication only (disable password auth) +- Implement bastion/jump hosts +- Use VPN for remote access +- Audit access logs + +--- + +## Related Documentation + +- [Live Migration Best Practices](./live-migration-best-practices.md) - Configuration parameters and requirements +- [Performance Tuning](./performance-tuning.md) - Optimization strategies +- [Anti-Patterns](./anti-patterns.md) - Common mistakes to avoid + +--- + +**Last Updated**: 2026-02-24 +**OpenShift Virtualization Versions**: 4.17, 4.18, 4.19, 4.20 +**Status**: Production-ready guidance from official Red Hat sources diff --git a/rh-virt/skills/vm-snapshot-create/SKILL.md b/rh-virt/skills/vm-snapshot-create/SKILL.md new file mode 100644 index 00000000..3c927bd6 --- /dev/null +++ b/rh-virt/skills/vm-snapshot-create/SKILL.md @@ -0,0 +1,423 @@ +--- +name: vm-snapshot-create +description: | + Create virtual machine snapshots for backup and recovery with comprehensive storage backend analysis. + + Use when: + - "Create a snapshot of VM [name]" + - "Backup VM [name] before upgrade" + - "Take a snapshot of [vm]" + + Analyzes storage capabilities, guest agent status, and consistency guarantees before creating snapshots. + + NOT for VM cloning (use vm-clone to create independent copies). + +model: inherit +color: green +--- + +# /vm-snapshot-create Skill + +Create virtual machine snapshots in OpenShift Virtualization. Snapshots capture the state and data of a VM at a specific point in time, enabling backup, recovery, and testing workflows. + +**Implementation Note**: This skill uses generic Kubernetes resource tools (`resources_create_or_update`) to manage VirtualMachineSnapshot resources. Dedicated snapshot tools do not currently exist in the openshift-virtualization MCP server. + +## Prerequisites + +**Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) + +**Required MCP Tools**: +- `resources_create_or_update` (from openshift-virtualization) - Create VirtualMachineSnapshot +- `resources_get` (from openshift-virtualization) - Verify VM exists and get status +- `resources_list` (from openshift-virtualization) - List StorageClass, VolumeSnapshotClass + +**Required Environment Variables**: +- `KUBECONFIG` - Path to Kubernetes configuration file with cluster access + +**Required Cluster Setup**: +- OpenShift cluster (>= 4.19) +- OpenShift Virtualization operator installed +- ServiceAccount with RBAC permissions to create VirtualMachineSnapshot resources +- Storage backend with snapshot support (CSI driver with snapshot capabilities) + +## When to Use This Skill + +**Trigger this skill when:** +- User wants to create a backup of a VM before changes +- User wants to create a recovery point +- User explicitly requests snapshot creation + +**User phrases that trigger this skill:** +- "Create a snapshot of VM database-01" +- "Backup VM web-server before upgrade" +- "Take a snapshot of production-app" + +**Do NOT use this skill when:** +- User wants to clone a VM → Use `vm-clone` skill (creates independent copy) +- User wants to list snapshots → Use `vm-snapshot-list` skill +- User wants to restore from snapshot → Use `vm-snapshot-restore` skill + +## Workflow + +### Step 1: Gather Snapshot Information + +**Required Information from User:** +1. **VM Name** - Name of the VM to snapshot +2. **Namespace** - Namespace where VM exists +3. **Snapshot Name** (Optional) - Name for the snapshot (auto-generated if not provided) + +If namespace not provided, ask for it explicitly. + +### Step 2: Verify VM Exists and Get Status + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +**Expected Output**: VirtualMachine resource with status + +**Error Handling**: +- If VM not found → Report error, suggest using vm-inventory skill +- If permission denied → Report RBAC error + +**Extract VM Details:** +- Current status (Running, Stopped) +- Storage configuration (DataVolumes, PVCs) +- **IMPORTANT**: Save `status.volumeSnapshotStatuses` for storage analysis + +### Step 3: Verify Storage Snapshot Capabilities + +**CRITICAL: This comprehensive storage analysis MUST execute BEFORE asking user about VM running state.** + +This step analyzes storage backend capabilities to determine snapshot behavior and requirements. The analysis includes 9 substeps. + +[Continue with all 9 substeps from the original file: 1c.1 through 1c.9, checking volume snapshot status, hot-plugged volumes, storage class, VolumeSnapshotClass, CSI driver capabilities, guest agent status, Windows VSS, and storing analysis results] + +### Step 4: Check VM Running State (Enhanced with Storage Analysis) + +**From the VM resource in Step 2**, check `status.printableStatus`. + +**Use storage analysis results from Step 3** to provide accurate guidance. + +[Include the three scenarios: VM must be stopped, VM can run (online supported), VM is stopped - with all the guest agent and Windows VSS warnings] + +### Step 5: Stop Running VM (if user chose "stop-and-snapshot") + +**ONLY execute if user chose "stop-and-snapshot" in Step 4.** + +Use `vm_lifecycle` MCP tool or vm-lifecycle-manager skill to stop the VM. + +### Step 6: Estimate Storage Consumption + +**From the VM resource**, estimate snapshot storage: +- Initial snapshot may be same size as VM disk +- Subsequent snapshots smaller (only deltas) + +### Step 7: Present Snapshot Configuration for Confirmation + +**Include storage analysis results from Step 3 in the configuration presentation.** + +[Include the full confirmation template with storage backend analysis, guest agent status, volumes to snapshot, etc.] + +**Wait for user confirmation.** + +**Handle response:** +- If "yes" → Proceed to Step 8 (execute snapshot) +- If "no", "cancel", or anything else → Cancel operation + +### Step 8: Create the Snapshot + +**ONLY PROCEED AFTER user confirmation in Step 7.** + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Construct VirtualMachineSnapshot YAML:** + +```yaml +apiVersion: snapshot.kubevirt.io/v1beta1 +kind: VirtualMachineSnapshot +metadata: + name: + namespace: +spec: + source: + apiGroup: kubevirt.io + kind: VirtualMachine + name: +``` + +**If snapshot name not provided by user**, generate one: +- Format: `-snapshot-` +- Example: `database-01-snapshot-20260218-143022` + +**Parameters**: +```json +{ + "resource": "apiVersion: snapshot.kubevirt.io/v1beta1\nkind: VirtualMachineSnapshot\nmetadata:\n name: \n namespace: \nspec:\n source:\n apiGroup: kubevirt.io\n kind: VirtualMachine\n name: " +} +``` + +**Report progress:** +```markdown +📸 Creating VM snapshot... +✓ Snapshot `` created for VM `` +``` + +### Step 9: Monitor Snapshot Status + +**After creation, monitor snapshot readiness using `resources_get`.** + +Check `status.phase`: +- `InProgress` → Still creating +- `Succeeded` → Snapshot ready +- `Failed` → Snapshot failed + +**Wait up to 5 minutes for snapshot to complete.** + +### Step 10: Report Snapshot Creation Results + +**Extract snapshot indications** from `status.indications`: +- `GuestAgent` - Guest agent coordinated the snapshot +- `Online` - Snapshot taken while VM was running + +**On success:** + +```markdown +## ✓ VM Snapshot Created Successfully + +**VM**: `` (namespace: ``) +**Snapshot**: `` + +### Snapshot Details +- **Name**: `` +- **Status**: Ready +- **Created**: +- **VM Status at Snapshot**: + +### Snapshot Coordination (from status.indications) + +- ✅ **Guest Agent Coordination**: Active +- ✅ **Filesystem Freeze/Thaw**: Performed during snapshot +- ✅ **Actual Consistency**: Application-consistent + + + +- ⚠️ **Guest Agent Coordination**: Not active +- ⚠️ **Actual Consistency**: Crash-consistent (best-effort) + + +### Next Steps + +**To list all snapshots:** +"List snapshots for VM " + +**To restore from this snapshot:** +"Restore VM from snapshot " + +**To delete this snapshot:** +"Delete snapshot " +``` + +## Dependencies + +### Required MCP Servers +- `openshift-virtualization` - OpenShift MCP server with kubevirt toolset + +### Required MCP Tools +- `resources_create_or_update` (from openshift-virtualization) - Create VirtualMachineSnapshot +- `resources_get` (from openshift-virtualization) - Verify VM and snapshot status +- `resources_list` (from openshift-virtualization) - List StorageClass, VolumeSnapshotClass + +### Related Skills +- `vm-snapshot-list` - List snapshots after creation +- `vm-snapshot-restore` - Restore VMs from snapshots +- `vm-snapshot-delete` - Delete old snapshots +- `vm-lifecycle-manager` - Stop VMs before snapshot +- `vm-inventory` - List VMs before creating snapshots + +### Reference Documentation + +**Official Red Hat Documentation:** +- [OpenShift Virtualization Snapshots - OpenShift 4.20](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt-managing-vm-snapshots) +- [Live Snapshots in OpenShift Virtualization](https://www.redhat.com/en/blog/live-snapshots-in-openshift-virtualization) + +**Upstream Documentation:** +- [KubeVirt VM Snapshots](https://kubevirt.io/user-guide/operations/snapshot_restore_api/) +- [CSI Volume Snapshots](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) + +## Critical: Human-in-the-Loop Requirements + +**IMPORTANT:** This skill performs operations that affect VM data and storage. You MUST: + +1. **Before Creating Snapshots** + - Analyze storage backend capabilities + - Verify VM exists and get current state + - Check for hot-plugged volumes (blocks snapshots) + - Show storage consumption estimate + - Present snapshot configuration + - Ask: "Proceed with snapshot creation? (yes/no)" + - Wait for explicit "yes" + +2. **Never Auto-Execute** + - **NEVER create without user confirmation** + - **NEVER skip storage analysis** + - **NEVER skip hot-plugged volume check** + +**Why This Matters:** +- **Storage Consumption**: Snapshots consume storage proportional to VM disk size +- **Hot-Plugged Volumes**: Cannot snapshot VMs with hot-plugged volumes +- **Consistency**: Online vs offline snapshots have different consistency guarantees +- **Guest Agent**: Required for application-consistent snapshots + +## Common Issues + +### Issue 1: Snapshot Creation Fails - VolumeSnapshotClass Not Found + +**Error**: "VolumeSnapshotClass not found" or "CSI driver doesn't support snapshots" + +**Cause**: The storage backend doesn't have a VolumeSnapshotClass configured for the CSI driver, or the CSI driver doesn't support snapshots at all. + +**Solution:** +1. **Check if VolumeSnapshotClass exists**: Use `resources_list` with apiVersion="snapshot.storage.k8s.io/v1", kind="VolumeSnapshotClass" +2. **Verify CSI driver supports snapshots**: Check StorageClass provisioner field +3. **Contact cluster admin**: Request VolumeSnapshotClass configuration for your storage backend +4. **Alternative**: Use `vm-clone` skill for VM backup instead of snapshots + +### Issue 2: Snapshot Creation Blocked - Hot-Plugged Volumes Detected + +**Error**: "Cannot create snapshot - VM has hot-plugged volumes" + +**Cause**: The VM has volumes that were attached after VM creation without restarting the VM. Hot-plugged volumes block snapshot creation in OpenShift Virtualization. + +**Solution:** +1. **Stop the VM**: Use vm-lifecycle-manager skill to stop the VM +2. **Remove hot-plugged volumes**: Detach volumes that aren't needed +3. **Persist volumes to VM spec**: Add hot-plugged volumes to `spec.template.spec.volumes` to make them permanent +4. **Restart the VM**: Start the VM to apply the changes +5. **Retry snapshot**: Once hot-plugged volumes are resolved, create the snapshot + +**Related**: See [OpenShift Virtualization documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt-hot-plugging-virtual-disks) for hot-plugging details + +### Issue 3: Snapshot Created but Consistency Warning + +**Error**: Snapshot created successfully but shows "crash-consistent" without guest agent + +**Cause**: The VM doesn't have QEMU guest agent installed or running, so the snapshot couldn't coordinate filesystem freeze/thaw during creation. + +**Solution:** +1. **For Linux VMs**: Install qemu-guest-agent package + ```bash + # RHEL/CentOS/Fedora + sudo dnf install qemu-guest-agent + sudo systemctl enable --now qemu-guest-agent + ``` +2. **For Windows VMs**: Install VirtIO drivers which include the guest agent +3. **Verify agent status**: Check VM status for `AgentConnected: True` condition +4. **Future snapshots**: Once guest agent is running, subsequent snapshots will be application-consistent +5. **Current snapshot**: The crash-consistent snapshot is still usable, but may have minor inconsistencies + +## Security Considerations + +- **RBAC Enforcement**: Requires permissions for VirtualMachineSnapshot resources +- **Storage Quotas**: Respects namespace storage quotas +- **Hot-Plugged Volume Detection**: Prevents snapshots when hot-plugged volumes present +- **KUBECONFIG Security**: Credentials never exposed in output +- **Namespace Isolation**: Snapshots scoped to namespace boundaries +- **Audit Trail**: All snapshot operations logged in Kubernetes API audit logs + +## Example Usage + +### Example 1: Create Snapshot Before Upgrade + +``` +User: "Create a snapshot of database-01 before I upgrade it" + +Agent: "Which namespace is database-01 in?" + +User: "production" + +Agent: [Invokes vm-snapshot-create skill] + [Step 2: Verifies VM exists] + [Step 3: Analyzes storage backend - AWS EBS, online snapshots supported] + [Step 4: Checks VM status - Running, guest agent active] + [Step 6: Estimates storage - 50Gi] + [Step 7: Presents configuration] + +## VM Snapshot - Review Configuration + +**VM to Snapshot:** +- **Name**: `database-01` +- **Namespace**: `production` +- **Status**: Running +- **Storage**: 50Gi + +**Storage Backend Analysis:** +- **Storage Class**: `gp3-csi` +- **CSI Driver**: `ebs.csi.aws.com` +- **Online Snapshots**: ✅ Supported +- **Consistency Level**: `crash-consistent` +- **VolumeSnapshotClass**: `ebs-snapclass` ✓ + +**Guest Agent Status:** +- ✅ **QEMU Guest Agent**: Connected and running +- ✅ **Filesystem Freeze/Thaw**: Will be performed during snapshot +- ✅ **Expected Consistency**: Application-consistent (with freeze) + +**Snapshot Details:** +- **Snapshot Name**: `database-01-snapshot-20260218-143022` (auto-generated) +- **Type**: Online snapshot +- **Estimated Storage**: ~50Gi (initial snapshot) + +**Proceed with snapshot creation? (yes/no)** + +User: "yes" + +Agent: [Step 8: Creates snapshot] + +📸 Creating VM snapshot... +✓ Snapshot `database-01-snapshot-20260218-143022` created for VM `database-01` + + [Step 9: Monitors status] + +⏳ Snapshot creation in progress... + Status: InProgress + +⏳ Snapshot creation in progress... + Status: Succeeded + + [Step 10: Reports results] + +## ✓ VM Snapshot Created Successfully + +**VM**: `database-01` (namespace: `production`) +**Snapshot**: `database-01-snapshot-20260218-143022` + +### Snapshot Details +- **Name**: `database-01-snapshot-20260218-143022` +- **Status**: Ready +- **Created**: 2026-02-18 14:30:22 UTC +- **VM Status at Snapshot**: Running + +### Snapshot Coordination (from status.indications) +- ✅ **Guest Agent Coordination**: Active +- ✅ **Filesystem Freeze/Thaw**: Performed during snapshot +- ✅ **Actual Consistency**: Application-consistent + +### Next Steps + +**To list all snapshots:** +"List snapshots for VM database-01" + +**To restore from this snapshot:** +"Restore VM database-01 from snapshot database-01-snapshot-20260218-143022" + +You can now safely upgrade the database. If the upgrade fails, restore using the command above. +``` diff --git a/rh-virt/skills/vm-snapshot-delete/SKILL.md b/rh-virt/skills/vm-snapshot-delete/SKILL.md new file mode 100644 index 00000000..14f2fb49 --- /dev/null +++ b/rh-virt/skills/vm-snapshot-delete/SKILL.md @@ -0,0 +1,447 @@ +--- +name: vm-snapshot-delete +description: | + Permanently delete virtual machine snapshots to free storage space. + + Use when: + - "Delete snapshot [snapshot-name]" + - "Remove old snapshots for VM [name]" + - "Free up snapshot storage" + + Requires user confirmation before deletion. + + NOT for restoring VMs (use vm-snapshot-restore instead). + +model: inherit +color: yellow +--- + +# /vm-snapshot-delete Skill + +Permanently delete virtual machine snapshots in OpenShift Virtualization. Deleting snapshots frees storage but removes recovery points. + +## Prerequisites + +**Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) + +**Required MCP Tools**: +- `resources_get` (from openshift-virtualization) - Verify snapshot exists +- `resources_list` (from openshift-virtualization) - List related snapshots +- `resources_delete` (from openshift-virtualization) - Delete snapshot + +**Required Environment Variables**: +- `KUBECONFIG` - Path to Kubernetes configuration file with cluster access + +**Required Cluster Setup**: +- OpenShift cluster (>= 4.19) +- OpenShift Virtualization operator installed +- ServiceAccount with RBAC permissions to delete VirtualMachineSnapshot resources + +### Prerequisite Verification + +**Before executing, verify MCP server availability:** + +1. **Check MCP Server Configuration** + - Verify `openshift-virtualization` exists in `.mcp.json` + - If missing → Report to user with setup instructions + +2. **Check Environment Variables** + - Verify `KUBECONFIG` is set (check presence only, never expose value) + - If missing → Report to user + +## When to Use This Skill + +**Trigger this skill when:** +- User wants to free storage by removing old snapshots +- User wants to delete a specific snapshot +- User wants to implement snapshot retention policies + +**User phrases that trigger this skill:** +- "Delete snapshot pre-upgrade-backup" +- "Remove old snapshots for VM database-01" +- "Delete all snapshots older than 7 days" +- "Free up snapshot storage" + +**Do NOT use this skill when:** +- User wants to create snapshots → Use `vm-snapshot-create` skill +- User wants to restore from snapshot → Use `vm-snapshot-restore` skill +- User wants to list snapshots → Use `vm-snapshot-list` skill + +## Workflow + +### Step 1: Gather Delete Information + +**Required Information from User:** +1. **Snapshot Name** - Name of snapshot to delete +2. **Namespace** - Namespace where snapshot exists + +If namespace not provided, ask for it. + +### Step 2: Verify Snapshot Exists + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "snapshot.kubevirt.io/v1beta1", + "kind": "VirtualMachineSnapshot", + "namespace": "", + "name": "" +} +``` + +**Expected Output**: VirtualMachineSnapshot resource + +**Error Handling**: +- If snapshot not found → Report error + +**If snapshot not found:** +```markdown +❌ Snapshot Not Found + +**Snapshot**: `` not found in namespace ``. + +**To list available snapshots:** +"List snapshots in namespace " + +Delete operation cancelled. +``` + +**STOP workflow**. + +**If snapshot found**, extract snapshot details: +- `spec.source.name` - VM name +- `metadata.creationTimestamp` - Creation timestamp +- `status.phase` - Snapshot status +- Calculate age from creationTimestamp + +### Step 3: List Other Snapshots for Same VM + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "snapshot.kubevirt.io/v1beta1", + "kind": "VirtualMachineSnapshot", + "namespace": "", + "labelSelector": "vm.kubevirt.io/name=" +} +``` + +**Fallback**: If label selector doesn't work, list all snapshots and filter by `spec.source.name`. + +**Count snapshots** for the VM to determine if this is the last snapshot. + +### Step 4: Present Snapshot Details and Confirm Deletion + +```markdown +## ⚠️ Snapshot Deletion - Review + +**Snapshot to Delete**: `` + +### Snapshot Details +- **Snapshot Name**: `` +- **VM**: `` +- **Namespace**: `` +- **Created**: +- **Age**: +- **Status**: + +### Impact of Deletion +- ✗ Snapshot will be permanently deleted +- ✗ This recovery point will be lost +- ✗ Cannot restore VM to this snapshot state after deletion +- ✓ Storage will be freed + +### Recovery Impact +**Before deletion, consider:** +- Is this snapshot still needed for recovery? +- Are there other recovery points available? +- Could you need to restore to this state in the future? + +**Available snapshots for VM ``:** + + + +⚠️ **WARNING**: This is the ONLY snapshot for VM ``. After deletion, no snapshot recovery points will exist. + + +--- + +**Proceed with snapshot deletion? This action cannot be undone. (yes/no)** +``` + +**Wait for user confirmation.** + +**Handle response:** +- If "yes" → Proceed to Step 5 (execute deletion) +- If "no", "cancel", or anything else → Cancel operation + +**On cancellation:** +```markdown +Snapshot deletion cancelled by user. Snapshot `` preserved. +``` + +**STOP workflow**. + +### Step 5: Delete the Snapshot + +**ONLY PROCEED AFTER user confirmation in Step 4.** + +**MCP Tool**: `resources_delete` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "snapshot.kubevirt.io/v1beta1", + "kind": "VirtualMachineSnapshot", + "namespace": "", + "name": "" +} +``` + +**Example tool invocation:** +```json +resources_delete({ + "apiVersion": "snapshot.kubevirt.io/v1beta1", + "kind": "VirtualMachineSnapshot", + "namespace": "production", + "name": "old-snapshot" +}) +``` + +**Expected Output**: VirtualMachineSnapshot deleted successfully + +**Error Handling**: +- If snapshot not found → Report error (may have been deleted externally) +- If permission denied → Report RBAC error +- If snapshot in use → Report error (snapshot may be in restore process) + +**Report progress:** +```markdown +🗑️ Deleting snapshot... +✓ Snapshot `` deleted +``` + +### Step 6: Report Deletion Results + +**On success:** + +```markdown +## ✓ Snapshot Deleted Successfully + +**Snapshot**: `` (VM: ``, namespace: ``) + +### Deletion Summary +- ✓ Snapshot permanently deleted +- ✓ Storage freed +- ✓ Recovery point removed + +### Impact +- ✗ Cannot restore VM to state +- ✗ Snapshot `` no longer available + + +### Remaining Snapshots for VM `` + + + +These snapshots are still available for recovery. + + + +⚠️ **No snapshots remain** for VM ``. Consider creating new snapshots for future recovery points. + + +--- + +### Next Steps + +**To create a new snapshot:** +"Create snapshot of VM " + +**To list remaining snapshots:** +"List snapshots for VM " +``` + +**On failure:** + +```markdown +## ❌ Snapshot Deletion Failed + +**Error**: + +**Snapshot**: `` (VM: ``, namespace: ``) + +**Common Causes:** +- **Snapshot not found** - May have been deleted externally +- **Insufficient RBAC permissions** - ServiceAccount lacks delete permissions +- **Snapshot in use** - Snapshot may be in active restore process +- **Storage backend error** - CSI driver or storage backend issue + +**Troubleshooting Steps:** + +1. **Verify snapshot still exists:** + "List snapshots for VM " + +2. **Check if snapshot is being used for restore:** + Use `resources_list` to check for active VirtualMachineRestore resources + +3. **Check permissions:** + Use CLI: `oc auth can-i delete virtualmachinesnapshots -n ` + +4. **Wait and retry** if snapshot is in use by restore operation + +Would you like help troubleshooting this error? +``` + +## Common Issues + +### Issue 1: Snapshot Not Found + +**Error**: "Snapshot `` not found in namespace ``" + +**Cause**: Snapshot doesn't exist, was deleted, or wrong namespace/name. + +**Solution:** +1. List snapshots to verify name: "List snapshots in namespace " +2. Check spelling (names are case-sensitive) +3. Try listing in other namespaces if unsure + +### Issue 2: Snapshot In Use During Restore + +**Error**: "Snapshot is in use by restore operation" + +**Cause**: An active VirtualMachineRestore is using this snapshot. + +**Solution:** +1. Check for active restores: Use `resources_list` with apiVersion="snapshot.kubevirt.io/v1beta1", kind="VirtualMachineRestore" +2. Wait for restore to complete, or delete the VirtualMachineRestore resource +3. Retry snapshot deletion + +### Issue 3: Permission Denied + +**Error**: "Forbidden: User lacks permissions to delete virtualmachinesnapshots" + +**Cause**: Missing RBAC permissions for snapshot deletion. + +**Solution:** +1. Check permissions: `oc auth can-i delete virtualmachinesnapshots -n ` +2. Contact cluster admin to grant delete permissions for virtualmachinesnapshots +3. Required permissions: delete verb on snapshot.kubevirt.io/virtualmachinesnapshots + +## Dependencies + +### Required MCP Servers +- `openshift-virtualization` - OpenShift MCP server with kubevirt toolset + +### Required MCP Tools +- `resources_get` (from openshift-virtualization) - Get snapshot details + - Parameters: apiVersion, kind, namespace, name + - Source: https://github.com/openshift/openshift-mcp-server + +- `resources_list` (from openshift-virtualization) - List related snapshots + - Parameters: apiVersion, kind, namespace, labelSelector + - Source: https://github.com/openshift/openshift-mcp-server + +- `resources_delete` (from openshift-virtualization) - Delete Kubernetes resources + - Parameters: apiVersion, kind, namespace, name + - Source: https://github.com/openshift/openshift-mcp-server + +### Related Skills +- `vm-snapshot-list` - List snapshots before deletion +- `vm-snapshot-create` - Create new snapshots +- `vm-snapshot-restore` - Restore VMs from snapshots + +### Reference Documentation +- [OpenShift Virtualization Snapshots](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt-managing-vm-snapshots) +- [KubeVirt VM Snapshots](https://kubevirt.io/user-guide/operations/snapshot_restore_api/) + +## Critical: Human-in-the-Loop Requirements + +**IMPORTANT:** This skill performs destructive operations. You MUST: + +1. **Before Deleting Snapshots** + - Show snapshot details (VM, age, size) + - Confirm snapshot won't be needed for recovery + - List other available snapshots for the VM + - Ask: "Proceed with snapshot deletion? (yes/no)" + - Wait for explicit "yes" + +2. **Never Auto-Execute** + - **NEVER delete without user confirmation** + - **ALWAYS show what will be lost before deletion** + +**Why This Matters:** +- **Recovery Point Loss**: Deleting snapshots removes recovery options permanently +- **No Undo**: Snapshot deletion cannot be reversed +- **Last Snapshot Warning**: Users must know if deleting the only snapshot for a VM + +## Security Considerations + +- **RBAC Enforcement**: Requires delete permissions for VirtualMachineSnapshot resources +- **User Confirmation**: Always requires explicit "yes" before deletion +- **Last Snapshot Warning**: Warns users when deleting the only snapshot for a VM +- **Namespace Isolation**: Snapshots scoped to namespace boundaries +- **Audit Trail**: Deletions logged in Kubernetes API audit logs + +## Example Usage + +``` +User: "Delete snapshot database-01-daily-backup" + +Agent: "Which namespace is the snapshot in?" + +User: "production" + +Agent: [Invokes vm-snapshot-delete skill] + [Step 2: Verifies snapshot exists] + [Step 3: Lists other snapshots] + [Step 4: Presents deletion confirmation] + +## ⚠️ Snapshot Deletion - Review + +Snapshot to Delete: `database-01-daily-backup` + +Snapshot Details: +- Name: `database-01-daily-backup` +- VM: `database-01` +- Namespace: `production` +- Created: 2024-01-14 02:00:00 UTC +- Age: 1 day + +Impact of Deletion: +- ✗ Snapshot will be permanently deleted +- ✗ This recovery point will be lost +- ✓ Storage will be freed + +Available snapshots for VM `database-01`: +- `database-01-pre-upgrade` (created 2024-01-15 10:30) + +Proceed with snapshot deletion? This action cannot be undone. (yes/no) + +User: "yes" + +Agent: [Step 5: Deletes snapshot] + +🗑️ Deleting snapshot... +✓ Snapshot `database-01-daily-backup` deleted + + [Step 6: Reports results] + +## ✓ Snapshot Deleted Successfully + +Snapshot: `database-01-daily-backup` + +Deletion Summary: +- ✓ Snapshot permanently deleted +- ✓ Storage freed +- ✓ Recovery point removed + +Remaining Snapshots for VM `database-01`: +- `database-01-pre-upgrade` (created 2024-01-15 10:30) + +This snapshot is still available for recovery. +``` diff --git a/rh-virt/skills/vm-snapshot-list/SKILL.md b/rh-virt/skills/vm-snapshot-list/SKILL.md new file mode 100644 index 00000000..3f199f8f --- /dev/null +++ b/rh-virt/skills/vm-snapshot-list/SKILL.md @@ -0,0 +1,402 @@ +--- +name: vm-snapshot-list +description: | + List virtual machine snapshots across namespaces with status, age, and recovery information. + + Use when: + - "List snapshots for VM [name]" + - "Show snapshots in namespace [name]" + - "What snapshots exist for [vm]?" + + Read-only operation - no user confirmation required. + + NOT for creating/deleting snapshots (use vm-snapshot-create/delete instead). + +model: inherit +color: cyan +--- + +# /vm-snapshot-list Skill + +List virtual machine snapshots in OpenShift Virtualization. This read-only skill displays snapshot information including status, age, size, and recovery options. + +## Prerequisites + +**Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) + +**Required MCP Tools**: +- `resources_list` (from openshift-virtualization) - List VirtualMachineSnapshot resources +- `resources_get` (from openshift-virtualization) - Get snapshot details + +**Required Environment Variables**: +- `KUBECONFIG` - Path to Kubernetes configuration file with cluster access + +**Required Cluster Setup**: +- OpenShift cluster (>= 4.19) +- OpenShift Virtualization operator installed +- ServiceAccount with RBAC permissions to list VirtualMachineSnapshot resources + +### Prerequisite Verification + +**Before executing, verify MCP server availability:** + +1. **Check MCP Server Configuration** + - Verify `openshift-virtualization` exists in `.mcp.json` + - If missing → Report to user with setup instructions + +2. **Check Environment Variables** + - Verify `KUBECONFIG` is set (check presence only, never expose value) + - If missing → Report to user + +**Human Notification Protocol:** + +When prerequisites fail: + +``` +❌ Cannot execute vm-snapshot-list: MCP server 'openshift-virtualization' is not available + +📋 Setup Instructions: +1. Add openshift-virtualization to .mcp.json +2. Set KUBECONFIG environment variable +3. Restart Claude Code to reload MCP servers + +🔗 Documentation: https://github.com/openshift/openshift-mcp-server +``` + +## When to Use This Skill + +**Trigger this skill when:** +- User wants to list available snapshots for recovery +- User wants to see snapshot status and age +- User wants to verify snapshot existence before restore +- User wants to identify old snapshots for deletion + +**User phrases that trigger this skill:** +- "List all snapshots for web-server VM" +- "Show snapshots in namespace production" +- "What snapshots exist?" +- "Display VM snapshots" + +**Do NOT use this skill when:** +- User wants to create a snapshot → Use `vm-snapshot-create` skill +- User wants to restore from snapshot → Use `vm-snapshot-restore` skill +- User wants to delete snapshots → Use `vm-snapshot-delete` skill + +## Workflow + +### Step 1: Gather Information + +**Required Information from User:** +1. **Namespace** - Namespace to list snapshots from +2. **VM Name** (Optional) - Filter snapshots by specific VM + +If user doesn't provide namespace, ask for it. + +### Step 2: List Snapshots + +**MCP Tool**: `resources_list` (from openshift-virtualization) + +**Parameters** (with VM filter using label selector): +```json +{ + "apiVersion": "snapshot.kubevirt.io/v1beta1", + "kind": "VirtualMachineSnapshot", + "namespace": "", + "labelSelector": "vm.kubevirt.io/name=" +} +``` + +**Parameters** (all snapshots in namespace): +```json +{ + "apiVersion": "snapshot.kubevirt.io/v1beta1", + "kind": "VirtualMachineSnapshot", + "namespace": "" +} +``` + +**Note**: The label selector `vm.kubevirt.io/name=` may not always exist. If no results are returned, fall back to listing all snapshots and filtering by checking `spec.source.name` field in the results. + +**Expected Output**: List of VirtualMachineSnapshot resources + +**Parse each snapshot to extract**: +- `metadata.name` - Snapshot name +- `metadata.namespace` - Namespace +- `metadata.creationTimestamp` - Creation time +- `spec.source.name` - VM name +- `status.phase` - Status (InProgress, Succeeded, Failed) +- `status.readyToUse` - Ready for restore (true/false) + +**Error Handling**: +- If namespace not found → Report error +- If permission denied → Report RBAC error +- If no snapshots found → Report "No snapshots found" + +### Step 3: Report Snapshot List + +**If snapshots found:** + +```markdown +## VM Snapshots + +**Namespace**: `` + +**VM**: `` + + +### Available Snapshots + +| Snapshot Name | VM Name | Status | Created | Age | ReadyToUse | +|---------------|---------|--------|---------|-----|------------| +| `pre-upgrade-snapshot` | `database-01` | Succeeded ✓ | 2024-01-15 10:30 | 2 days | true | +| `backup-snapshot` | `database-01` | Succeeded ✓ | 2024-01-10 08:00 | 7 days | true | +| `test-snapshot` | `web-server` | Succeeded ✓ | 2024-01-14 14:20 | 3 days | true | + +**Total Snapshots**: 3 + +--- + +### Snapshot Details + +**Snapshot: `pre-upgrade-snapshot`** +- **VM**: `database-01` +- **Status**: Succeeded ✓ +- **Created**: 2024-01-15 10:30:00 UTC +- **Age**: 2 days +- **Ready to Use**: true + +**Snapshot: `backup-snapshot`** +- **VM**: `database-01` +- **Status**: Succeeded ✓ +- **Created**: 2024-01-10 08:00:00 UTC +- **Age**: 7 days +- **Ready to Use**: true + +**Snapshot: `test-snapshot`** +- **VM**: `web-server` +- **Status**: Succeeded ✓ +- **Created**: 2024-01-14 14:20:00 UTC +- **Age**: 3 days +- **Ready to Use**: true + +--- + +### Actions + +**To restore from a snapshot:** +``` +"Restore VM from snapshot " +``` + +**To delete a snapshot:** +``` +"Delete snapshot " +``` + +**To create a new snapshot:** +``` +"Create snapshot of VM " +``` +``` + +**If no snapshots found:** + +```markdown +## VM Snapshots + +**Namespace**: `` + +**VM**: `` + + +**No snapshots found.** + + +No snapshots exist for VM `` in namespace ``. + + +No snapshots exist in namespace ``. + + +**To create a snapshot:** +``` +"Create snapshot of VM " +``` +``` + +## Common Issues + +### Issue 1: Permission Denied + +**Error**: "Forbidden: User lacks permissions to list virtualmachinesnapshots" + +**Cause**: Missing RBAC permissions for listing snapshots. + +**Solution:** +1. Check permissions: `oc auth can-i list virtualmachinesnapshots -n ` +2. Contact cluster admin to grant list/get permissions for virtualmachinesnapshots +3. Try listing in a different namespace where you have permissions + +### Issue 2: No Snapshots Found + +**Error**: "No snapshots exist in namespace ``" + +**Cause**: Namespace has no snapshots, or wrong namespace. + +**Solution:** +1. Verify correct namespace name +2. List snapshots without VM filter to see all snapshots +3. Check other namespaces: Use `namespaces_list` to see available namespaces +4. Check if snapshots were recently deleted: Use `events_list` in namespace + +### Issue 3: Snapshot Shows Failed Status + +**Error**: Snapshot listed but `status.phase: Failed` or `readyToUse: false` + +**Cause**: Snapshot creation failed due to storage issues, hot-plugged volumes, or missing VolumeSnapshotClass. + +**Solution:** +1. Get snapshot details: Use `resources_get` to check `status.conditions` for error messages +2. Check cluster events: Use `events_list` for snapshot-related errors +3. Common fixes: + - VolumeSnapshotClass missing: Contact cluster admin + - Hot-plugged volumes: Stop VM, persist volumes, recreate snapshot + - Failed permanently: Delete and recreate snapshot + +## Dependencies + +### Required MCP Servers +- `openshift-virtualization` - OpenShift MCP server with kubevirt toolset + +### Required MCP Tools +- `resources_list` (from openshift-virtualization) - List Kubernetes resources by type + - Used for: Listing VirtualMachineSnapshot resources + - Parameters: apiVersion, kind, namespace, labelSelector (optional) + - Source: https://github.com/openshift/openshift-mcp-server + +### Related Skills +- `vm-snapshot-create` - Create VM snapshots +- `vm-snapshot-restore` - Restore VMs from snapshots +- `vm-snapshot-delete` - Delete VM snapshots +- `vm-inventory` - List VMs before creating snapshots + +### Reference Documentation +- [OpenShift Virtualization Snapshots](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt-managing-vm-snapshots) +- [KubeVirt VM Snapshots](https://kubevirt.io/user-guide/operations/snapshot_restore_api/) + +## Security Considerations + +- **RBAC Enforcement**: Requires list permissions for VirtualMachineSnapshot resources +- **Namespace Isolation**: Only lists snapshots within specified namespace +- **Read-Only Operation**: No modifications or destructive actions +- **KUBECONFIG Security**: Credentials never exposed in output + +## Example Usage + +### Example 1: List Snapshots for Specific VM + +``` +User: "List all snapshots for database-01" + +Agent: "Which namespace is database-01 in?" + +User: "production" + +Agent: [Invokes vm-snapshot-list skill] + [Lists snapshots with VM filter] + +## VM Snapshots + +**Namespace**: `production` +**VM**: `database-01` + +### Available Snapshots + +| Snapshot Name | VM Name | Status | Created | Age | ReadyToUse | +|---------------|---------|--------|---------|-----|------------| +| `database-01-pre-upgrade` | `database-01` | Succeeded ✓ | 2024-01-15 10:30 | 2 hours | true | +| `database-01-daily-backup` | `database-01` | Succeeded ✓ | 2024-01-14 02:00 | 1 day | true | + +**Total Snapshots**: 2 + +--- + +### Snapshot Details + +**Snapshot: `database-01-pre-upgrade`** +- **VM**: `database-01` +- **Status**: Succeeded ✓ +- **Created**: 2024-01-15 10:30:00 UTC +- **Age**: 2 hours +- **Ready to Use**: true + +**Snapshot: `database-01-daily-backup`** +- **VM**: `database-01` +- **Status**: Succeeded ✓ +- **Created**: 2024-01-14 02:00:00 UTC +- **Age**: 1 day +- **Ready to Use**: true + +--- + +### Actions + +**To restore from a snapshot:** +"Restore VM database-01 from snapshot " + +**To delete a snapshot:** +"Delete snapshot " + +**To create a new snapshot:** +"Create snapshot of VM database-01" +``` + +### Example 2: List All Snapshots in Namespace + +``` +User: "Show me all snapshots in namespace production" + +Agent: [Invokes vm-snapshot-list skill] + [Lists all snapshots without VM filter] + +## VM Snapshots + +**Namespace**: `production` + +### Available Snapshots + +| Snapshot Name | VM Name | Status | Created | Age | ReadyToUse | +|---------------|---------|--------|---------|-----|------------| +| `database-01-pre-upgrade` | `database-01` | Succeeded ✓ | 2024-01-15 10:30 | 2 hours | true | +| `database-01-daily-backup` | `database-01` | Succeeded ✓ | 2024-01-14 02:00 | 1 day | true | +| `web-server-backup` | `web-server` | Succeeded ✓ | 2024-01-13 03:00 | 2 days | true | + +**Total Snapshots**: 3 + +VMs with snapshots: `database-01` (2), `web-server` (1) +``` + +### Example 3: No Snapshots Found + +``` +User: "List snapshots for test-vm" + +Agent: "Which namespace is test-vm in?" + +User: "dev" + +Agent: [Invokes vm-snapshot-list skill] + [Lists snapshots - none found] + +## VM Snapshots + +**Namespace**: `dev` +**VM**: `test-vm` + +**No snapshots found.** + +No snapshots exist for VM `test-vm` in namespace `dev`. + +**To create a snapshot:** +"Create snapshot of VM test-vm" +``` diff --git a/rh-virt/skills/vm-snapshot-restore/SKILL.md b/rh-virt/skills/vm-snapshot-restore/SKILL.md new file mode 100644 index 00000000..4fbd8e63 --- /dev/null +++ b/rh-virt/skills/vm-snapshot-restore/SKILL.md @@ -0,0 +1,580 @@ +--- +name: vm-snapshot-restore +description: | + Restore virtual machines from snapshots with strict safety confirmations to prevent data loss. + + Use when: + - "Restore VM [name] from snapshot [snapshot-name]" + - "Roll back VM [name] to snapshot" + - "Recover VM [name] from backup" + + CRITICAL: Requires VM to be stopped and typed snapshot name confirmation before restore. + + NOT for creating snapshots (use vm-snapshot-create instead). + +model: inherit +color: red +--- + +# /vm-snapshot-restore Skill + +Restore virtual machines from snapshots in OpenShift Virtualization. **CRITICAL**: This operation replaces current VM state with snapshot data. ALL changes since the snapshot will be LOST. + +**Implementation Note**: This skill uses generic Kubernetes resource tools (`resources_create_or_update`) to create VirtualMachineRestore resources. Dedicated restore tools do not currently exist in the openshift-virtualization MCP server. + +## Prerequisites + +**Required MCP Server**: `openshift-virtualization` ([OpenShift MCP Server](https://github.com/openshift/openshift-mcp-server)) + +**Required MCP Tools**: +- `resources_create_or_update` (from openshift-virtualization) - Create VirtualMachineRestore +- `resources_get` (from openshift-virtualization) - Verify VM/snapshot exists, monitor restore +- `vm_lifecycle` (from openshift-virtualization) - Stop VM if running + +**Required Environment Variables**: +- `KUBECONFIG` - Path to Kubernetes configuration file with cluster access + +**Required Cluster Setup**: +- OpenShift cluster (>= 4.19) +- OpenShift Virtualization operator installed +- ServiceAccount with RBAC permissions to create VirtualMachineRestore resources + +## When to Use This Skill + +**Trigger this skill when:** +- User wants to restore a VM to a previous state +- User wants to recover from failed changes/upgrades +- User explicitly requests snapshot restore + +**User phrases that trigger this skill:** +- "Restore VM api-server from snapshot snapshot-20240115" +- "Roll back database-01 to pre-upgrade snapshot" +- "Recover VM web-server from backup" + +**Do NOT use this skill when:** +- User wants to create snapshots → Use `vm-snapshot-create` skill +- User wants to list snapshots → Use `vm-snapshot-list` skill +- User wants to clone a VM → Use `vm-clone` skill + +## Workflow + +### Step 1: Gather Restore Information + +**Required Information from User:** +1. **VM Name** - VM to restore +2. **Namespace** - Namespace where VM exists +3. **Snapshot Name** - Snapshot to restore from + +If any information missing, ask for it. + +### Step 2: Verify VM Exists + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "kubevirt.io/v1", + "kind": "VirtualMachine", + "namespace": "", + "name": "" +} +``` + +**Error Handling**: +- If VM not found → Report error +- If permission denied → Report RBAC error + +### Step 3: Check VM Running State + +**From the VM resource in Step 2**, check `status.printableStatus`. + +**If VM is Running:** +```markdown +⚠️ VM Must Be Stopped Before Restore + +**VM**: `` (namespace: ``) +**Status**: Running + +**Safety Requirement**: VMs must be stopped before restore to prevent data corruption. + +**Options:** +1. "stop-and-restore" - Stop the VM first, then restore from snapshot +2. "cancel" - Cancel restore operation + +How would you like to proceed? +``` + +**Wait for user response.** + +- If "stop-and-restore" → Stop VM using vm_lifecycle, then continue +- If "cancel" → Stop workflow + +### Step 4: Verify Snapshot Exists + +**MCP Tool**: `resources_get` (from openshift-virtualization) + +**Parameters**: +```json +{ + "apiVersion": "snapshot.kubevirt.io/v1beta1", + "kind": "VirtualMachineSnapshot", + "namespace": "", + "name": "" +} +``` + +**If snapshot not found:** +```markdown +❌ Snapshot Not Found + +**Snapshot**: `` does not exist in namespace ``. + +**To list available snapshots:** +"List snapshots for VM " + +Restore operation cancelled. +``` + +**STOP workflow**. + +**Extract snapshot details:** +- `metadata.creationTimestamp` - Creation time +- `status.phase` - Must be "Succeeded" +- `status.readyToUse` - Must be `true` +- `spec.source.name` - Verify it matches the VM name + +**If snapshot status is not Ready:** +```markdown +❌ Snapshot Not Ready + +**Snapshot**: `` +**Status**: +**Ready to Use**: + +Snapshot is not ready for restore. Only snapshots with "Succeeded" phase and readyToUse=true can be used. + +Restore operation cancelled. +``` + +**STOP workflow**. + +### Step 5: Present Restore Preview and Get Typed Confirmation + +**CRITICAL: User must type the snapshot name to confirm.** + +```markdown +## 🔴 VM RESTORE - Data Loss Warning + +**⚠️ THIS WILL REPLACE CURRENT VM STATE WITH SNAPSHOT DATA ⚠️** + +### What Will Happen + +**VM to Restore**: `` (namespace: ``) +**Snapshot to Restore From**: `` + +**Current VM State** (WILL BE LOST): +- **Last Modified**: +- **Changes Since Snapshot**: ALL changes made after WILL BE PERMANENTLY LOST + +**Snapshot State** (WILL BE RESTORED): +- **Created**: +- **Age**: + +**Time Range of Data Loss**: +- **⚠️ ALL CHANGES in the last WILL BE LOST ⚠️** + +### What Will Be Restored +- ✓ VM configuration (from snapshot time) +- ✓ Disk data (from snapshot time) + +### What Will Be Lost +- ✗ **ALL disk changes** made after +- ✗ **ALL configuration changes** made after + +--- + +**⚠️ CRITICAL: This restore is permanent. Current VM state cannot be recovered unless you create a snapshot now.** + +**To proceed with restore, type the snapshot name exactly as shown:** + +Type `` to confirm: _____ +``` + +**Wait for user to type the snapshot name.** + +**Validation:** +- Compare user input with snapshot name (case-sensitive, exact match) +- **If match**: Proceed to Step 6 +- **If mismatch**: Cancel operation + +**On mismatch:** +```markdown +❌ Confirmation Failed + +**You typed**: `` +**Expected**: `` + +Names do not match. Restore cancelled for safety. + +Operation cancelled. Current VM state preserved. +``` + +**STOP workflow**. + +### Step 6: Final Confirmation Before Restore + +**After typed verification succeeds**, ask for final explicit confirmation. + +```markdown +## ✓ Typed Verification Passed + +**Confirmation received for snapshot**: `` + +### Ready to Restore + +**VM**: `` (namespace: ``) +**From Snapshot**: `` + +**Impact**: +- Current VM state will be replaced with snapshot state +- All changes in the last will be permanently lost + +--- + +**Proceed with VM restore? This action cannot be undone.** +- Type "yes" to execute restore +- Type "cancel" to abort + +Your choice: _____ +``` + +**Wait for user response.** + +**Handle response:** +- If "yes" → Proceed to Step 7 (execute restore) +- If "cancel", "no", "wait", or anything else → Cancel operation + +**On cancellation:** +```markdown +Restore operation cancelled by user. Current VM state preserved. +``` + +**STOP workflow**. + +### Step 7: Execute Restore + +**ONLY PROCEED AFTER**: +- ✓ VM verified (exists, stopped) +- ✓ Snapshot verified (exists, ready) +- ✓ User typed snapshot name correctly +- ✓ User confirmed "yes" + +**MCP Tool**: `resources_create_or_update` (from openshift-virtualization) + +**Construct VirtualMachineRestore YAML:** + +```yaml +apiVersion: snapshot.kubevirt.io/v1beta1 +kind: VirtualMachineRestore +metadata: + name: + namespace: +spec: + target: + apiGroup: kubevirt.io + kind: VirtualMachine + name: + virtualMachineSnapshotName: +``` + +**Generate restore name**: +- Format: `restore--` +- Example: `restore-database-01-20260218-143500` + +**Parameters**: +```json +{ + "resource": "apiVersion: snapshot.kubevirt.io/v1beta1\nkind: VirtualMachineRestore\nmetadata:\n name: \n namespace: \nspec:\n target:\n apiGroup: kubevirt.io\n kind: VirtualMachine\n name: \n virtualMachineSnapshotName: " +} +``` + +**Report progress:** +```markdown +🔄 Restoring VM from snapshot... +⏳ This may take several minutes... +``` + +### Step 8: Monitor Restore Progress + +**Use `resources_get` to monitor VirtualMachineRestore status.** + +Check `status.complete`: +- `true` → Restore completed +- `false` → Restore in progress + +**Wait up to 10 minutes for restore to complete.** + +### Step 9: Report Restore Results + +**On success:** + +```markdown +## ✓ VM Restored Successfully + +**VM**: `` (namespace: ``) +**Restored From**: Snapshot `` + +### Restore Details +- **Snapshot Created**: +- **Restore Completed**: +- **VM Status**: Stopped (ready to start) + +### Data Loss Confirmation +- ⚠️ All changes made after have been lost + +### Next Steps + +**To start the restored VM:** +"Start VM in namespace " +``` + +**On failure:** + +```markdown +## ❌ VM Restore Failed + +**Error**: + +**VM**: `` +**Snapshot**: `` + +**Current VM State**: UNKNOWN - may be partially restored or unchanged + +**CRITICAL**: Do not start VM until restore issue is resolved + +**Recovery Options:** +1. Try restore again after resolving the error +2. Restore from a different snapshot +3. Contact cluster admin for investigation +``` + +## Dependencies + +### Required MCP Servers +- `openshift-virtualization` - OpenShift MCP server with kubevirt toolset + +### Required MCP Tools +- `resources_create_or_update` (from openshift-virtualization) - Create VirtualMachineRestore +- `resources_get` (from openshift-virtualization) - Verify and monitor +- `vm_lifecycle` (from openshift-virtualization) - Stop VM if running + +### Related Skills +- `vm-snapshot-list` - List snapshots before restore +- `vm-snapshot-create` - Create snapshots before risky operations +- `vm-snapshot-delete` - Delete old snapshots +- `vm-lifecycle-manager` - Start VM after restore + +### Reference Documentation + +**Official Red Hat Documentation:** +- [OpenShift Virtualization Snapshots - OpenShift 4.20](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/virtualization/index#virt-managing-vm-snapshots) + +**Upstream Documentation:** +- [KubeVirt VM Snapshots](https://kubevirt.io/user-guide/operations/snapshot_restore_api/) + +## Critical: Human-in-the-Loop Requirements + +**IMPORTANT:** This skill performs DESTRUCTIVE operations. You MUST: + +1. **Before Restoring Snapshots** (CRITICAL - Data Loss Risk) + - **REQUIRE VM to be stopped first** if currently running + - Display what will be lost (current VM state since snapshot) + - Show snapshot details (creation time, age) + - **Require typed confirmation** - user must type snapshot name exactly + - Ask: "Proceed with restore? This will replace current VM state. (yes/cancel)" + - Wait for explicit "yes" + +2. **Never Auto-Execute** + - **NEVER restore without user confirmation** + - **NEVER restore to running VMs** without stopping first + - **NEVER skip typed verification for restore operations** + +**Why This Matters:** +- **Data Loss on Restore**: Restoring replaces current VM state - all changes since snapshot are PERMANENTLY LOST +- **No Undo**: Restore cannot be reversed - current data cannot be recovered +- **Typed Confirmation**: Prevents accidental restores to wrong snapshots + +## Common Issues + +### Issue 1: Restore Fails - Insufficient Storage Capacity + +**Error**: "Failed to restore: insufficient storage capacity" or "PVC provisioning failed" + +**Cause**: The namespace doesn't have enough storage quota or the storage backend is full. + +**Solution:** +1. **Check namespace storage quota**: Use `resources_list` with apiVersion="v1", kind="ResourceQuota" to check available storage +2. **Check PVC status**: Use `resources_list` to check PersistentVolumeClaims in the namespace +3. **Delete unnecessary snapshots**: Use vm-snapshot-delete skill to free up storage +4. **Request quota increase**: Contact cluster admin to increase namespace storage quota +5. **Try restore again**: Once storage is available, retry the restore operation + +### Issue 2: Restore Stuck in Progress + +**Error**: VirtualMachineRestore status shows `complete: false` for extended period + +**Cause**: The storage backend is slow, the snapshot is corrupted, or there's a CSI driver issue. + +**Solution:** +1. **Check VirtualMachineRestore status**: Use `resources_get` to check `status.conditions` for detailed error messages +2. **Check snapshot health**: Use vm-snapshot-list skill to verify snapshot is in "Succeeded" state +3. **Check storage backend logs**: Review CSI driver logs for errors (requires cluster admin access) +4. **Wait longer**: Large VMs may take 10+ minutes to restore +5. **Cancel and retry**: Delete the VirtualMachineRestore resource and try again +6. **Alternative snapshot**: If restore continues to fail, try restoring from a different snapshot + +**Related**: If the issue persists, there may be a storage backend problem requiring cluster administrator investigation. + +### Issue 3: User Typed Wrong Snapshot Name + +**Error**: "Confirmation Failed - Names do not match" + +**Cause**: User made a typo when typing the snapshot name for confirmation. + +**Solution:** +1. **Retry with exact name**: Copy-paste the snapshot name from the previous message +2. **Check capitalization**: Snapshot names are case-sensitive +3. **Check for spaces**: Ensure no leading/trailing spaces in the typed name +4. **Use autocomplete**: If available in your terminal, use tab completion +5. **List snapshots**: Use vm-snapshot-list skill to see all available snapshots and verify the correct name + +**This safety mechanism is intentional** - it prevents accidental data loss from restoring to the wrong snapshot. + +## Security Considerations + +- **Data Protection**: Restore requires TWO confirmations (typed + yes) to prevent accidental data loss +- **VM Stop Requirement**: VMs must be stopped before restore +- **Typed Verification**: User must type exact snapshot name +- **RBAC Enforcement**: Requires permissions for VirtualMachineRestore resources +- **Namespace Isolation**: Restore scoped to namespace boundaries +- **Audit Trail**: All restore operations logged in Kubernetes API audit logs +- **No Rollback**: Restore is permanent - current state cannot be recovered after restore + +## Example Usage + +### Example 1: Restore Failed Upgrade + +``` +User: "The database upgrade failed. Restore database-01 from snapshot pre-upgrade-snapshot" + +Agent: "Which namespace is database-01 in?" + +User: "production" + +Agent: [Invokes vm-snapshot-restore skill] + [Step 2: Verifies VM exists] + [Step 3: Checks VM status - Running] + +⚠️ VM Must Be Stopped Before Restore + +**VM**: `database-01` (namespace: `production`) +**Status**: Running + +**Safety Requirement**: VMs must be stopped before restore to prevent data corruption. + +**Options:** +1. "stop-and-restore" - Stop the VM first, then restore from snapshot +2. "cancel" - Cancel restore operation + +How would you like to proceed? + +User: "stop-and-restore" + +Agent: [Stops VM using vm_lifecycle] + +⏸️ Stopping VM before restore... + +VM `database-01` is being stopped gracefully. + + [Waits 10 seconds, verifies stopped] + [Step 4: Verifies snapshot exists and is ready] + [Step 5: Presents restore preview] + +## 🔴 VM RESTORE - Data Loss Warning + +**⚠️ THIS WILL REPLACE CURRENT VM STATE WITH SNAPSHOT DATA ⚠️** + +**VM to Restore**: `database-01` (namespace: `production`) +**Snapshot to Restore From**: `pre-upgrade-snapshot` + +**Current VM State** (WILL BE LOST): +- **Last Modified**: 2026-02-18 16:30:00 UTC +- **Changes Since Snapshot**: ALL changes made after 2026-02-18 10:00:00 WILL BE PERMANENTLY LOST + +**Snapshot State** (WILL BE RESTORED): +- **Created**: 2026-02-18 10:00:00 UTC +- **Age**: 6 hours 30 minutes + +**Time Range of Data Loss**: +- **⚠️ ALL CHANGES in the last 6 hours 30 minutes WILL BE LOST ⚠️** + +**To proceed with restore, type the snapshot name exactly as shown:** + +Type `pre-upgrade-snapshot` to confirm: _____ + +User: "pre-upgrade-snapshot" + +Agent: [Step 6: Typed verification passed] + [Step 7: Final confirmation] + +## ✓ Typed Verification Passed + +**Confirmation received for snapshot**: `pre-upgrade-snapshot` + +**VM**: `database-01` +**From Snapshot**: `pre-upgrade-snapshot` + +**Impact**: +- Current VM state will be replaced with snapshot state +- All changes in the last 6 hours 30 minutes will be permanently lost + +**Proceed with VM restore? This action cannot be undone.** +- Type "yes" to execute restore +- Type "cancel" to abort + +Your choice: _____ + +User: "yes" + +Agent: [Step 8: Executes restore] + +🔄 Restoring VM from snapshot... +⏳ This may take several minutes... + + [Step 9: Monitors progress] + +⏳ Restore in progress... + + [Step 10: Reports results] + +## ✓ VM Restored Successfully + +**VM**: `database-01` (namespace: `production`) +**Restored From**: Snapshot `pre-upgrade-snapshot` + +### Restore Details +- **Snapshot Created**: 2026-02-18 10:00:00 UTC +- **Restore Completed**: 2026-02-18 16:35:22 UTC +- **VM Status**: Stopped (ready to start) + +### Data Loss Confirmation +- ⚠️ All changes made after 2026-02-18 10:00:00 UTC have been lost + +### Next Steps + +**To start the restored VM:** +"Start VM database-01 in namespace production" + +The database has been restored to its pre-upgrade state. The failed upgrade has been rolled back. +```