diff --git a/docs/METRICS.md b/docs/METRICS.md index d3acee8de4..4c9c37e53a 100644 --- a/docs/METRICS.md +++ b/docs/METRICS.md @@ -11,8 +11,8 @@ and where those numbers live*. ## Why it exists PortOS's [Layered Intelligence loop](./plans/2026-07-07-layered-intelligence-loop.md) -perpetually reviews each managed app and files one high-value improvement proposal -per run. To reason well it needs to evaluate the app against **its own** +perpetually reviews each managed app and files at most one high-value improvement +proposal per run. To reason well it needs to evaluate the app against **its own** performance — user success, product KPIs, production telemetry — not against how reliably PortOS's coding agents happen to change it. @@ -21,6 +21,30 @@ telemetry/analytics service, a metrics dashboard, an events pipeline. An agent h no way to discover those unless the repo tells it where they are and how to read them. `METRICS.md` is that map. +## Layered Intelligence decision contract + +Layered Intelligence uses a metrics-first decision path: + +1. When a metric shows a meaningful gap, it proposes the single highest-value + app change that can plausibly move that metric. +2. When the metrics are missing, unavailable, stale, or inconclusive, its + reasoning agent may inspect the repository read-only to understand the + relevant product flow and existing instrumentation. This is context discovery, + not permission to turn the run into a generic code-quality audit. +3. When that inspection identifies a specific missing signal or context that + blocks evaluation against a named goal, it files an `app-data-gap` issue for + the minimum visibility needed. The issue must identify the unanswered product + question, the expected data source, and acceptance criteria; “add more + telemetry” is not decision-complete. +4. It files nothing only when the available evidence is sufficient and healthy, + the needed visibility already exists, or no decision-complete proposal fits + the configured scopes (subject to the loop's filing safety and delivery + gates). Inconclusive metrics alone do not prove the app is healthy. + +The reasoning agent never edits code in this task. A deterministic output hook +validates and deduplicates its structured proposal before filing it to the app's +configured tracker. + ## Who reads it - **Layered Intelligence** gathers repo-root `METRICS.md` as the `appMetrics` diff --git a/server/services/autonomousJobs/layeredIntelligenceHooks.js b/server/services/autonomousJobs/layeredIntelligenceHooks.js index 223efe7f8b..7f9b75a119 100644 --- a/server/services/autonomousJobs/layeredIntelligenceHooks.js +++ b/server/services/autonomousJobs/layeredIntelligenceHooks.js @@ -208,6 +208,9 @@ function buildCompletionContract() { 'You are a REASONING agent, not a coding agent. Do NOT edit code, run `/do:pr`,', 'commit, or open a pull request — any changes you make to this worktree are', 'discarded. Your ONLY output is the JSON described above.', + 'You MAY inspect repository files and run read-only commands when the gathered', + 'metrics are insufficient to understand the app context or identify a precise', + 'visibility gap.', '', 'When you have decided, write your result to the completion sentinel. Its', `absolute path is printed in the **${PROGRAMMATIC_OUTPUT_COMPLETION_HEADING}**`, diff --git a/server/services/autonomousJobs/layeredIntelligenceHooks.test.js b/server/services/autonomousJobs/layeredIntelligenceHooks.test.js index 32d3253ce6..a01fad1c3b 100644 --- a/server/services/autonomousJobs/layeredIntelligenceHooks.test.js +++ b/server/services/autonomousJobs/layeredIntelligenceHooks.test.js @@ -306,6 +306,7 @@ describe('buildTaskInput', () => { // this hook cannot know) — by the SHARED heading constant, so the pointer // can't drift from the section agentPromptBuilder emits. expect(res.prompt).toContain('completion sentinel'); + expect(res.prompt).toContain('inspect repository files and run read-only commands'); expect(res.prompt).toContain(PROGRAMMATIC_OUTPUT_COMPLETION_HEADING); // ...and it must NOT name any other filename. A prompt that says 'not a bare // `.agent-done`' is the only place the agent would learn that wrong diff --git a/server/services/layeredIntelligence.playbook.md b/server/services/layeredIntelligence.playbook.md index fb24736c5f..74fac6b116 100644 --- a/server/services/layeredIntelligence.playbook.md +++ b/server/services/layeredIntelligence.playbook.md @@ -20,9 +20,9 @@ # LI Proposal Playbook -You (the reasoner) file blindly by default: you propose work without checking how -your past proposals fared, which produces a low merge rate and repeated -NOT_PLANNED rejections. This playbook is your correction. Treat it as a hard +Your purpose is to turn evidence about whether the app is meeting its goals into +at most one decision-complete improvement issue. This playbook prevents weakly +grounded proposals and repeated NOT_PLANNED rejections. Treat it as a hard constraint on **which scope**, **which kind of work**, and **whether it aligns with goals** — applied before you commit to any proposal. @@ -38,9 +38,13 @@ require extra justification for scopes that historically get rejected. block appears above (your execution health is degraded), `loop-meta` and `portos-self` are HARD-EXCLUDED and will be dropped before filing — a degraded loop cannot repair itself, so that work is deferred to a human (see §4). -- **Prefer `app-data-gap`.** Adding the telemetry / metrics / instrumentation - needed to reason well (e.g. a missing METRICS.md, an unmeasured KPI) lands - reliably: it is unambiguous, low-risk, and unblocks future higher-value work. +- **Prefer `app-data-gap` when visibility is the blocker.** Adding the telemetry, + metrics, instrumentation, or product context needed to answer a specific + performance question (e.g. a missing METRICS.md, an unmeasured KPI, or no + observable completion signal for a stated goal) lands reliably: it is + unambiguous, low-risk, and unblocks future higher-value work. First inspect the + relevant repository flow read-only so the proposal names the minimum missing + signal and its likely source. Never file a generic “add more telemetry” issue. - **De-prioritize `app-improvement` unless alignment is explicit.** A generic "improve the app" proposal without a clear tie to a stated goal, a measured gap, or committed backlog is the single largest source of rejections. Only @@ -49,8 +53,10 @@ require extra justification for scopes that historically get rejected. - **`portos-self`** (PortOS install only): scoped, verifiable self-improvements to PortOS as an app — hold to the same bar as `app-improvement`. -If nothing clears this bar, return `proposal: null`. Filing nothing is a -legitimate, and often the correct, outcome. +If a metric is inconclusive, do not treat that as healthy and stop immediately: +first determine whether a specific visibility gap prevents a sound decision. If +visibility is already adequate and nothing clears this bar, return +`proposal: null`. Filing nothing is a legitimate, and often the correct, outcome. ## 2. Success Pattern Catalog diff --git a/server/services/layeredIntelligence.test.js b/server/services/layeredIntelligence.test.js index ba8be25e08..0bf1ca99b7 100644 --- a/server/services/layeredIntelligence.test.js +++ b/server/services/layeredIntelligence.test.js @@ -1689,7 +1689,22 @@ describe('buildPrompt', () => { it('frames the mission around the app\'s own goals and performance', () => { const out = buildPrompt({ app, isPortos: false, config: { allowedScopes: ['app-improvement'], rules: '' } }); - expect(out).toContain('its OWN goals and purpose'); + expect(out).toContain('its OWN goals'); + }); + + it('uses read-only app context to distinguish healthy metrics from a visibility gap', () => { + const out = buildPrompt({ + app, + isPortos: false, + config: { allowedScopes: ['app-improvement', 'app-data-gap'], rules: '' }, + sources: { appMetrics: 'Activation: unavailable' } + }); + expect(out).toContain('inspect this repository READ-ONLY'); + expect(out).toContain('Do not edit files'); + expect(out).toContain('SPECIFIC visibility gap'); + expect(out).toContain('never file a generic "add more telemetry" issue'); + expect(out).toContain('Inconclusive metrics alone are not evidence that the app is healthy'); + expect(out).toContain('AT MOST ONE decision-complete tracker issue'); }); it('always injects the static proposal playbook block (#2763)', () => { diff --git a/server/services/layeredIntelligence/prompt.js b/server/services/layeredIntelligence/prompt.js index 4d5c939002..77c65540b5 100644 --- a/server/services/layeredIntelligence/prompt.js +++ b/server/services/layeredIntelligence/prompt.js @@ -169,7 +169,13 @@ export function buildPrompt({ app, config, sources = {}, openIssues = [], isPort ? `\n### liHardExclusions\n${hardExclusionNotice.trim()}\n` : ''; - return `You are the Layered Intelligence reasoner for the app "${app.name}". Your job is to evaluate how THIS app is performing against its OWN goals and purpose${isPortos ? '' : ', not how well PortOS\'s tooling manages it'}. Decide the SINGLE highest-value improvement to propose this run (signal, not noise), grounded in the app's own goals and its own performance metrics (user success, KPIs, production telemetry). You never write code; you return structured JSON that a deterministic system files as ONE tracker issue. Optional \`model\` / \`effort\` dispatch hints and \`goodFirstIssue\` / \`helpWanted\` labels are independent of each other and of \`complexity\` — set them only when justified; omit rather than guessing; never stamp medium on both axes; never mark a wide mechanical sweep as a good first issue. + return `You are the Layered Intelligence reasoner for the app "${app.name}". Your purpose is to turn evidence about how THIS app performs against its OWN goals${isPortos ? '' : ', not how well PortOS\'s tooling manages it'} into AT MOST ONE decision-complete tracker issue that improves the app. Start with the app's own performance metrics (user success, KPIs, production telemetry), not a generic code-quality audit. You never write code; you return structured JSON that a deterministic system validates, deduplicates, and files. Optional \`model\` / \`effort\` dispatch hints and \`goodFirstIssue\` / \`helpWanted\` labels are independent of each other and of \`complexity\` — set them only when justified; omit rather than guessing; never stamp medium on both axes; never mark a wide mechanical sweep as a good first issue. + +Decision path (follow in order): +1. If a metric shows a meaningful gap, identify the single highest-value app improvement that can plausibly move that metric and file that proposal. +2. If the metrics are missing, stale, unavailable, or too narrow/inconclusive to support an app improvement, inspect this repository READ-ONLY to understand the relevant user flow, architecture, existing instrumentation, and documented data sources. Do not edit files, and do not substitute an unrelated code audit for missing product evidence. +3. If that inspection reveals a SPECIFIC visibility gap that prevents the app from being evaluated against a named goal, file an app-data-gap proposal for the minimum metrics, instrumentation, or context needed. Name the goal, the unanswered product question, where the signal should come from, and acceptance criteria that make a future run actionable; never file a generic "add more telemetry" issue. +4. Return proposal: null only when the available evidence is sufficient and shows no worthwhile improvement, the needed visibility already exists, no decision-complete proposal fits an allowed scope, or a filing safety/delivery gate below requires no proposal. Inconclusive metrics alone are not evidence that the app is healthy. ${handoffNote}${metricsGuidance}${hardExclusionBlock} Rules & guidance from the operator: ${config.rules?.trim() || '(none)'} diff --git a/server/services/taskSchedule.test.js b/server/services/taskSchedule.test.js index 768d212d95..f06f3a0346 100644 --- a/server/services/taskSchedule.test.js +++ b/server/services/taskSchedule.test.js @@ -277,7 +277,8 @@ describe('taskSchedule', () => { describe('layered-intelligence (programmatic-I/O agent task)', () => { it('is registered as a self-improvement task with a description and an on-demand default', () => { expect(SELF_IMPROVEMENT_TASK_TYPES).toContain('layered-intelligence'); - expect(TASK_TYPE_DESCRIPTIONS['layered-intelligence']).toBeTruthy(); + expect(TASK_TYPE_DESCRIPTIONS['layered-intelligence']).toContain('performance metrics'); + expect(TASK_TYPE_DESCRIPTIONS['layered-intelligence']).toContain('visibility gap'); expect(DEFAULT_TASK_INTERVALS['layered-intelligence']).toMatchObject({ type: INTERVAL_TYPES.ON_DEMAND, enabled: true }); }); diff --git a/server/services/taskScheduleRegistry.js b/server/services/taskScheduleRegistry.js index 12f8d93134..fd670f3b52 100644 --- a/server/services/taskScheduleRegistry.js +++ b/server/services/taskScheduleRegistry.js @@ -527,7 +527,7 @@ export const TASK_TYPE_DESCRIPTIONS = { 'stash-cleanup': 'Triage git stash list — drop entries superseded by or stale relative to main, leave real unlanded work in place', 'repo-sync': 'Sync every managed app with origin — back on the default branch, pushed and pulled, merged branches/worktrees and redundant stashes cleared', 'plan-feature': "Brainstorm one feature and file its decision-complete plan to the app's work tracker (no code)", - 'layered-intelligence': "Read this app's goals + telemetry, ask a reasoning model for one improvement, and file one deduplicated tracker issue — no code, no agent" + 'layered-intelligence': "Use app goals + performance metrics to file at most one deduplicated improvement issue; inspect read-only context and file a visibility gap when evidence is insufficient — no code" }; export function getTaskTypeDescription(taskType) {