hermes-plugin-auditor

Audit Hermes agent plugins - verify manifests, hooks, toolsets, and capabilities

Problem

Hermes agent plugins are powerful but fragile:

• Silent plugin failures: Plugins in plugins.enabled that don't load - no errors, just missing functionality
• Manifest drift: plugin.yaml missing required fields, wrong hook names, stale toolset lists
• Hook mystery: Which plugins handle on_session_start? Why didn't my hook fire?
• Dependency rot: Declared dependencies not installed, version conflicts
• Toolset confusion: Multiple plugins providing same toolset, no visibility

Solution

hermes-plugin-auditor provides read-only inspection of all Hermes plugins:

Check	Detects
Manifest Validation	Missing required fields, unknown hooks, invalid toolsets
Hook Verification	Hook handler files exist, correct Python import paths
Toolset Inventory	All toolsets provided by all plugins
Dependency Check	Declared Python dependencies installed
Load Status	Enabled vs disabled, import errors
Config Cross-Ref	Enabled plugins actually exist and have manifests

Installation

pip install hermes-plugin-auditor

Or run directly:

python -m hermes_plugin_auditor audit

Quick Start

# Audit all plugins in default profile
hermes-plugin-auditor audit

# Audit specific profile
hermes-plugin-auditor audit --profile production

# Show only enabled plugins
hermes-plugin-auditor audit --only-enabled

# Audit specific plugin
hermes-plugin-auditor audit --plugin session-continuity

# Machine-readable JSON output
hermes-plugin-auditor audit --json

# List all discovered plugins
hermes-plugin-auditor list

# Show all registered hooks
hermes-plugin-auditor hooks

# Filter hooks by type
hermes-plugin-auditor hooks --type on_session_start

# Show all toolsets
hermes-plugin-auditor toolsets

Commands

audit - Full Plugin Audit

hermes-plugin-auditor audit [OPTIONS]

Options:
  --profile TEXT      Hermes profile (default: default)
  --hermes-home PATH  Override Hermes home directory
  --json              Output as JSON
  --verbose, -v       Show handler details
  --only-enabled      Only audit enabled plugins
  --plugin TEXT       Audit specific plugin only

Exit codes: 0 = all good, 1 = issues found

list - List Discovered Plugins

hermes-plugin-auditor list [OPTIONS]

Options:
  --profile TEXT      Hermes profile
  --hermes-home PATH  Override Hermes home
  --json              Output as JSON

hooks - Show Registered Hooks

hermes-plugin-auditor hooks [OPTIONS]

Options:
  --profile TEXT      Hermes profile
  --hermes-home PATH  Override Hermes home
  --json              Output as JSON
  --type TEXT         Filter by hook type (e.g., on_session_start)

toolsets - Show Provided Toolsets

hermes-plugin-auditor toolsets [OPTIONS]

Options:
  --profile TEXT      Hermes profile
  --hermes-home PATH  Override Hermes home
  --json              Output as JSON

Example Output

Human-readable (default)

╭──────────────────────────────────────────────────────────────╮
│ session-continuity                    ● ENABLED            │
├──────────────────────────────────────────────────────────────┤
│ Version: 1.0.0                                                 │
│ Description: Detect PC offline gaps across Hermes sessions   │
│ Author: sudo_sonic                                             │
│                                                                │
│ Hooks Registered:                                              │
│   • on_session_start: 1 handler(s)                            │
│     - session_continuity:on_session_start                     │
│                                                                │
│ Toolsets Provided:                                             │
│                                                                │
│ Warnings:                                                      │
│  ⚠ hook_handler_not_found: Hook handler not found:           │
│     'session_continuity:on_session_start' for hook            │
│     'on_session_start'                                        │
│                                                                │
│ Issues:                                                        │
╰──────────────────────────────────────────────────────────────╯

╭──────────────────────────────────────────────────────────────╮
│ a2a                                  disabled                │
├──────────────────────────────────────────────────────────────┤
│ Version: 1.0.0                                                 │
│ Description: A2A protocol support for Hermes                 │
│                                                                │
│ Hooks Registered:                                              │
│   • on_startup: 1 handler(s)                                 │
│     - a2a:on_startup                                          │
│                                                                │
│ Toolsets Provided:                                             │
│   • a2a: ['a2a_call', 'a2a_discover', 'a2a_history', ...]    │
│                                                                │
│ Warnings:                                                      │
│  ⚠ dependency_missing: Declared dependency not installed:    │
│     'httpx'                                                   │
╰──────────────────────────────────────────────────────────────╯

════════════════════════════════════════════════════════════════
                    Plugin Audit Summary                        
════════════════════════════════════════════════════════════════
Total Plugins: 12
Enabled: 4    Disabled: 8    Errors: 2    Warnings: 15

Hooks Registered:
  • on_session_start: 2 handlers from 2 plugin(s)
  • on_startup: 5 handlers from 4 plugin(s)
  • on_message: 1 handlers from 1 plugin(s)

Total Toolsets: 8

JSON Output

{
  "session-continuity": {
    "plugin_name": "session-continuity",
    "plugin_path": "/home/user/.hermes/plugins/session-continuity",
    "status": "enabled",
    "manifest": {
      "name": "session-continuity",
      "version": "1.0.0",
      "description": "Detect PC offline gaps across Hermes sessions",
      "hooks": {
        "on_session_start": ["session_continuity:on_session_start"]
      },
      "toolsets": {}
    },
    "issues": [],
    "warnings": [
      {
        "code": "hook_handler_not_found",
        "message": "Hook handler not found: 'session_continuity:on_session_start' for hook 'on_session_start'"
      }
    ],
    "is_enabled": true,
    "hook_handlers_found": {
      "on_session_start": ["session_continuity:on_session_start"]
    },
    "toolsets_provided": {}
  }
}

As a Library

from hermes_plugin_auditor import PluginAuditor, audit_plugins

# One-off audit all plugins
results = audit_plugins(hermes_home="/custom/.hermes", profile="production")

for name, result in results.items():
    if result.has_errors():
        print(f"{name}: ERRORS")
        for issue in result.issues:
            print(f"  - {issue['code']}: {issue['message']}")

# Reusable auditor
auditor = PluginAuditor(profile="staging")
summary = auditor.get_summary()
print(f"Total plugins: {summary['total_plugins']}")
print(f"Enabled: {summary['enabled']}")
print(f"Hooks: {list(summary['hooks_registered'].keys())}")

# Audit single plugin
result = auditor.audit_plugin("my-plugin")
if result.hook_handlers_found:
    for hook_type, handlers in result.hook_handlers_found.items():
        print(f"{hook_type}: {handlers}")

Validation Rules

Manifest (plugin.yaml) Requirements

Field	Required	Type	Description
name	Yes	string	Plugin name (should match directory)
version	Yes	string	Semantic version
description	Yes	string	Human-readable description
hooks	Yes	object	Map of hook_type → handler list
author	No	string	Author name
license	No	string	License identifier
dependencies	No	list[str]	Python package dependencies
toolsets	No	object	Map of toolset_name → tool list
min_hermes_version	No	string	Minimum Hermes version

Hook Types (Known)

• on_session_start
• on_session_end
• on_message
• on_tool_call
• on_tool_result
• on_error
• on_startup
• on_shutdown

Hook Handler Format

Handlers in manifest can be:

• "module:function" - Module and function name
• "module" - Just module, assumes hook_type function
• Dotted paths like "my_plugin.handlers:on_session_start"

CI/CD Integration

# .github/workflows/audit-plugins.yml
name: Audit Hermes Plugins
on: [push, pull_request]
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install hermes-plugin-auditor
      - run: hermes-plugin-auditor audit --json > audit.json
        env:
          HERMES_HOME: ${{ github.workspace }}/.hermes
      - uses: actions/upload-artifact@v4
        with:
          name: plugin-audit
          path: audit.json
      # Fail if errors found
      - run: |
          python -c "
          import json, sys
          with open('audit.json') as f:
              data = json.load(f)
          has_errors = any(r.get('issues', []) for r in data.values())
          sys.exit(1 if has_errors else 0)
          "

Why This Exists

Hermes plugins are the extension mechanism. A typo in plugin.yaml silently disables a feature. A missing httpx dependency crashes at runtime. This tool catches those before they hit production.

License

MIT License - see LICENSE