Skip to content

Add dump and diagnose command - #121

Closed
Patryk-Stefanski wants to merge 2 commits into
mainfrom
add-dump-diagnose
Closed

Add dump and diagnose command#121
Patryk-Stefanski wants to merge 2 commits into
mainfrom
add-dump-diagnose

Conversation

@Patryk-Stefanski

@Patryk-Stefanski Patryk-Stefanski commented Dec 2, 2025

Copy link
Copy Markdown

Add diagnose and dump commands for Kuadrant troubleshooting

Overview

This PR introduces two new diagnostic commands to kuadrantctl that help users troubleshoot and investigate issues with their Kuadrant and Gateway API resources.

This could also be paired up with a claude / command to help debug/fix the environment.

Use Cases

  • Troubleshooting: Quickly identify why Gateways aren't programmed or policies aren't being enforced
  • Support Tickets: Export complete resource manifests to share with support teams
  • Documentation: Generate snapshots of working configurations

What's New

1. kuadrantctl diagnose - Health Check Command

A comprehensive diagnostic tool that analyzes Kuadrant and Gateway API resources to identify configuration issues and unhealthy states.

Features:

  • Scans all Kuadrant and Gateway API resources in your cluster
  • Identifies unprogrammed Gateways, unenforced policies, and configuration problems
  • Provides detailed issue reports with reasons and messages from resource conditions
  • Supports namespace filtering or cluster-wide diagnostics

Usage:

# Diagnose all namespaces
kuadrantctl diagnose

# Diagnose specific namespace
kuadrantctl diagnose -n my-namespace

# Explicit all-namespaces flag
kuadrantctl diagnose -A

Sample Output:
================================================================================
KUADRANT DIAGNOSTIC REPORT
================================================================================

SUMMARY:
--------------------------------------------------------------------------------
Gateways:                   2 total, 1 unprogrammed
HTTPRoutes:                 5 total, 1 unaccepted
AuthPolicies:               3 total, 1 unenforced
RateLimitPolicies:          2 total, 0 unenforced
DNSPolicies:                1 total, 0 unenforced
TLSPolicies:                1 total, 0 unenforced
Kuadrants:                  1 total, 0 unready

Total issues found: 3

--------------------------------------------------------------------------------
GATEWAY ISSUES:
--------------------------------------------------------------------------------

  Resource: gateway-namespace/my-gateway
  Status:   Not Programmed
  Issues:
    - Not Programmed: NoResources - No backend resources available
    - Listener http - Programmed: Pending - Waiting for IP assignment

--------------------------------------------------------------------------------
HTTPROUTE ISSUES:
--------------------------------------------------------------------------------

  Resource: app-namespace/api-route
  Status:   Not Accepted
  Issues:
    - Not Accepted by parent my-gateway: NoMatchingListenerHostname - HTTPRoute hostname doesnt match any Gateway listener

--------------------------------------------------------------------------------
AUTHPOLICY ISSUES:
--------------------------------------------------------------------------------

  Resource: app-namespace/api-auth
  Status:   Not Enforced
  Issues:
    - Not Enforced: TargetNotFound - Target HTTPRoute not found in namespace

================================================================================
  1. kuadrantctl dump - Resource Export Command

Exports all Kuadrant and Gateway API resources to files for offline investigation, debugging, or sharing with support teams.

Features:

  • Dumps all Kuadrant resources (Gateways, HTTPRoutes, AuthPolicies, RateLimitPolicies, DNSPolicies, TLSPolicies, Kuadrants)
  • Organizes resources by type in separate directories
  • Removes cluster-specific metadata (UIDs, resource versions, managed fields) for cleaner output
  • Supports YAML and JSON output formats
  • Creates timestamped directories automatically

Usage:

# Dump all resources to timestamped directory
kuadrantctl dump

# Dump to specific directory
kuadrantctl dump -o my-debug-folder

# Dump specific namespace
kuadrantctl dump -n my-namespace

# Dump in JSON format
kuadrantctl dump --format json

Sample Output:
Resources dumped to: kuadrant-dump-20250102-143055

Directory Structure:
kuadrant-dump-20250102-143055/
├── gateways/
│   ├── gateway-namespace-my-gateway.yaml
│   └── istio-system-istio-gateway.yaml
├── gatewayclasses/
│   └── istio.yaml
├── httproutes/
│   ├── app-namespace-api-route.yaml
│   └── app-namespace-web-route.yaml
├── authpolicies/
│   └── app-namespace-api-auth.yaml
├── ratelimitpolicies/
│   └── app-namespace-api-ratelimit.yaml
├── dnspolicies/
│   └── gateway-namespace-dns-policy.yaml
├── tlspolicies/
│   └── gateway-namespace-tls-policy.yaml
└── kuadrants/
    └── kuadrant-system-kuadrant.yaml

How It Works Under the Hood

Both commands leverage the controller-runtime client to interact with the Kubernetes API:

  1. diagnose command:
    - Inspects status conditions on each resource (e.g., Programmed, Accepted, Enforced)
    - Identifies resources where condition status is not True
    - Aggregates issues and generates summary statistics
    - Outputs formatted report to stdout
  2. dump command:
    - Retrieves full resource manifests from the cluster
    - Strips cluster-specific metadata (UID, resourceVersion, managedFields)
    - Marshals to YAML/JSON
    - Writes each resource to a separate file organized by type

@codecov-commenter

codecov-commenter commented Dec 2, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 468 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (9c811a8) to head (113ec0c).
⚠️ Report is 15 commits behind head on main.

Files with missing lines Patch % Lines
cmd/diagnose.go 0.00% 358 Missing ⚠️
cmd/dump.go 0.00% 89 Missing ⚠️
cmd/root.go 0.00% 21 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #121       +/-   ##
==========================================
- Coverage   54.57%   0.00%   -54.58%     
==========================================
  Files          17       6       -11     
  Lines         885     685      -200     
==========================================
- Hits          483       0      -483     
- Misses        330     685      +355     
+ Partials       72       0       -72     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eguzki eguzki left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet 🍬

LGTM

@eguzki

eguzki commented Dec 2, 2025

Copy link
Copy Markdown
Collaborator

Add some doc?

@Boomatang Boomatang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is some changes I would like to see.

  • flag validation
  • namespace required explicitly, not defaulting to all, if anything the default should be from the kubeconfig. Like what the kubectl does.
  • Created directories only when ready to write data to location.

Comment thread cmd/dump.go
Comment on lines +58 to +59
if err := os.MkdirAll(dumpOutputDir, 0755); err != nil {
return fmt.Errorf("failed to create output directory: %w", err)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In dumpResourceType you call os.MkdirAll for the resource. The path used is for build from the dumpOutputDir and the resource name. Why do we need to make this call here? There is a number of possible errors that can happen after this which could cause blank directories to be created.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to make it here. I'll refactor the code to move the directory creation to just before the file write, ensuring we only create directories when we actually have data to write.

Comment thread cmd/dump.go

cmd.Flags().StringVarP(&dumpNamespace, "namespace", "n", "", "Namespace to dump resources from (default: all namespaces)")
cmd.Flags().BoolVarP(&dumpAllNamespaces, "all-namespaces", "A", false, "Dump resources from all namespaces")
cmd.Flags().StringVarP(&dumpOutputDir, "output", "o", "", "Output directory (default: ./kuadrant-dump-<timestamp>)")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On allowing the user to set the output directory. What happens if the directory is not empty? Are we overriding existing data in that directory?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, it's overwriting the existing data in that directory. Ill add a --force flag and check if the directory is non-empty before proceeding

Comment thread cmd/dump.go Outdated
Comment on lines +220 to +221
listObj client.ObjectList,
itemsFunc func(client.ObjectList) []runtime.Object,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not pass in the struct is being looped over?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll refactor the code to use a generic approach.

Comment thread cmd/dump.go Outdated
Comment on lines +249 to +251
obj.SetResourceVersion("")
obj.SetUID("")
obj.SetGeneration(0)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you clearing these? If I am dumping resources, I would want to be able to check the resource version and generation against other points of information. The UID maybe I don't want that, but by and large I would prefer to have it.

Comment thread cmd/dump.go Outdated
Comment on lines +264 to +271
if dumpFormat == "json" {
data, err = yaml.Marshal(item)
// Note: Using yaml.Marshal for both since we'd need proper JSON encoder setup
// This still produces valid JSON-compatible output
} else {
// Default to YAML
data, err = yaml.Marshal(item)
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have this check if there is no difference?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some stale code, that shouldve been removed

Comment thread cmd/dump.go Outdated
Comment on lines +39 to +40
cmd.Flags().StringVarP(&dumpNamespace, "namespace", "n", "", "Namespace to dump resources from (default: all namespaces)")
cmd.Flags().BoolVarP(&dumpAllNamespaces, "all-namespaces", "A", false, "Dump resources from all namespaces")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can there be some input validation on the inputs. The user should not be able to set the namespace and all-namespaces at the same time.

I also don't think the default should be to dump all the namespaces if not set.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ill add some validation , however the namesapce will be defaulted to all as we cant assume that all users will install their resources in a specific namespace and dumping the required resources from all namespaces is more convenient.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can't set the default namespace from the kubeconfig, we should force the setting of the field. I understand it maybe more convenient to have it set to all namespaces, but that is not predictable. Take for example, the user runs kubectl get kuadrant, what output would you expect.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect the kuadrant resource from whatever namespace the kubeconfig is currently set to. And that makes sense when you're only retrieving one resource that is in a specific namespace.

When retrieving multiple resources that could be in different namespaces, I think it makes sense to dump resources from all namespaces by default, as this is more than likely what most users will want to do.

@eguzki wdyt ? default to specific namespace or all-namespaces?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as the desired behavior is documented, both approaches LGTM. I guess I am trying to say pick the easiest to implement and make it clear in doc.

Comment thread cmd/diagnose.go
Comment on lines +71 to +72
cmd.Flags().StringVarP(&diagnoseNamespace, "namespace", "n", "", "Namespace to diagnose (default: all namespaces)")
cmd.Flags().BoolVarP(&diagnoseAllNamespaces, "all-namespaces", "A", false, "Diagnose resources from all namespaces")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment on validation and default as before.

Comment thread cmd/diagnose.go Outdated
Comment on lines +82 to +105
// Setup Kubernetes client
cfg, err := config.GetConfig()
if err != nil {
return fmt.Errorf("failed to get kubeconfig: %w", err)
}

// Register Gateway API and Kuadrant schemes
if err := gatewayapiv1.AddToScheme(scheme.Scheme); err != nil {
return fmt.Errorf("failed to add Gateway API to scheme: %w", err)
}
if err := kuadrantv1alpha1.AddToScheme(scheme.Scheme); err != nil {
return fmt.Errorf("failed to add Kuadrant v1alpha1 API to scheme: %w", err)
}
if err := kuadrantv1beta1.AddToScheme(scheme.Scheme); err != nil {
return fmt.Errorf("failed to add Kuadrant v1beta1 API to scheme: %w", err)
}
if err := kuadrantv1beta2.AddToScheme(scheme.Scheme); err != nil {
return fmt.Errorf("failed to add Kuadrant v1beta2 API to scheme: %w", err)
}

k8sClient, err := client.New(cfg, client.Options{Scheme: scheme.Scheme})
if err != nil {
return fmt.Errorf("failed to create Kubernetes client: %w", err)
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks to be the same in the other command. Can this be refactored to be a shared command.

Comment thread cmd/diagnose.go
Comment on lines +362 to +377
// Check status conditions
for _, condition := range policy.Status.Conditions {
switch condition.Type {
case "Accepted":
if string(condition.Status) != string(metav1.ConditionTrue) {
issues = append(issues, fmt.Sprintf("Not Accepted: %s - %s", condition.Reason, condition.Message))
}
case "Enforced":
if string(condition.Status) != string(metav1.ConditionTrue) {
issues = append(issues, fmt.Sprintf("Not Enforced: %s - %s", condition.Reason, condition.Message))
report.Summary.UnenforcedRateLimitPolicies++
} else {
isEnforced = true
}
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These blocks seem to be repeated in all the diagnose functions, can it be refactored to be more general?

Comment thread cmd/diagnose.go Outdated
Comment on lines +598 to +669
// Print Gateway issues
if len(report.GatewayIssues) > 0 {
fmt.Println("\n" + strings.Repeat("-", 80))
fmt.Println("GATEWAY ISSUES:")
fmt.Println(strings.Repeat("-", 80))
for _, issue := range report.GatewayIssues {
printResourceIssue(issue)
}
}

// Print HTTPRoute issues
if len(report.HTTPRouteIssues) > 0 {
fmt.Println("\n" + strings.Repeat("-", 80))
fmt.Println("HTTPROUTE ISSUES:")
fmt.Println(strings.Repeat("-", 80))
for _, issue := range report.HTTPRouteIssues {
printResourceIssue(issue)
}
}

// Print AuthPolicy issues
if len(report.AuthPolicyIssues) > 0 {
fmt.Println("\n" + strings.Repeat("-", 80))
fmt.Println("AUTHPOLICY ISSUES:")
fmt.Println(strings.Repeat("-", 80))
for _, issue := range report.AuthPolicyIssues {
printResourceIssue(issue)
}
}

// Print RateLimitPolicy issues
if len(report.RateLimitPolicyIssues) > 0 {
fmt.Println("\n" + strings.Repeat("-", 80))
fmt.Println("RATELIMITPOLICY ISSUES:")
fmt.Println(strings.Repeat("-", 80))
for _, issue := range report.RateLimitPolicyIssues {
printResourceIssue(issue)
}
}

// Print DNSPolicy issues
if len(report.DNSPolicyIssues) > 0 {
fmt.Println("\n" + strings.Repeat("-", 80))
fmt.Println("DNSPOLICY ISSUES:")
fmt.Println(strings.Repeat("-", 80))
for _, issue := range report.DNSPolicyIssues {
printResourceIssue(issue)
}
}

// Print TLSPolicy issues
if len(report.TLSPolicyIssues) > 0 {
fmt.Println("\n" + strings.Repeat("-", 80))
fmt.Println("TLSPOLICY ISSUES:")
fmt.Println(strings.Repeat("-", 80))
for _, issue := range report.TLSPolicyIssues {
printResourceIssue(issue)
}
}

// Print Kuadrant issues
if len(report.KuadrantIssues) > 0 {
fmt.Println("\n" + strings.Repeat("-", 80))
fmt.Println("KUADRANT ISSUES:")
fmt.Println(strings.Repeat("-", 80))
for _, issue := range report.KuadrantIssues {
printResourceIssue(issue)
}
}

fmt.Println("\n" + strings.Repeat("=", 80))
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be converted in to some loop?

Signed-off-by: Patryk Stefanski <pstefans@redhat.com>
@Patryk-Stefanski

Copy link
Copy Markdown
Author

@Boomatang fyi I'm putting this PR on hold. I want to chat with Chris and the wider team if it's worth keeping and maintaining this repo, or at the very least removing some commands if we're not planning on supporting them.

@Boomatang

Copy link
Copy Markdown
Member

if it's worth keeping and maintaining this repo

Can you share links to where these conversations are going on? In the dns-operator we are making design for kuadrant-dns plugin based on this repo existing.

@Patryk-Stefanski

Copy link
Copy Markdown
Author

It will be raised and discussed at the next kudarant community call

@Patryk-Stefanski
Patryk-Stefanski force-pushed the add-dump-diagnose branch 2 times, most recently from a7df22c to e10a5ff Compare December 12, 2025 13:48
@Patryk-Stefanski

Copy link
Copy Markdown
Author

@Boomatang this is ready for review again, I addressed your comments and removed the generate commands as the outcome of the discussion on this weeks community call was that no one had hard opinions/thoughts on keeping those commands in.

Signed-off-by: Patryk Stefanski <pstefans@redhat.com>
@Boomatang

Copy link
Copy Markdown
Member

Was there any feedback from Chris on the future of those commands? During the community call it was mentioned that you would be speaking with them during the week.

@Boomatang

Copy link
Copy Markdown
Member

Also can you split this in to two PRs. The adding of one function has nothing to do with the removal of the others.

@Patryk-Stefanski

Copy link
Copy Markdown
Author

@Boomatang was not planning on speaking with Chris about this, when you say those commands do you mean the new ones Im adding in or the generate commands Im removing ?

@Patryk-Stefanski

Patryk-Stefanski commented Dec 19, 2025

Copy link
Copy Markdown
Author

Also can you split this in to two PRs. The adding of one function has nothing to do with the removal of the others.

New commands require the most recent kuadrant api. The generate commands rely on the outdated api, since this PR is moving to use the most up to date api. I think it makes sense to remove the commands in this PR as otherwise there will exist a version in which those commands are broken.

@Patryk-Stefanski

Copy link
Copy Markdown
Author

Closing this for now, as it's gotten stale, and with similar results can be achieved with the help of Claude

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants