Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions docs/METRICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ and where those numbers live*.
## Why it exists

PortOS's [Layered Intelligence loop](./plans/2026-07-07-layered-intelligence-loop.md)
perpetually reviews each managed app and files one high-value improvement proposal
per run. To reason well it needs to evaluate the app against **its own**
perpetually reviews each managed app and files at most one high-value improvement
proposal per run. To reason well it needs to evaluate the app against **its own**
performance — user success, product KPIs, production telemetry — not against how
reliably PortOS's coding agents happen to change it.

Expand All @@ -21,6 +21,30 @@ telemetry/analytics service, a metrics dashboard, an events pipeline. An agent h
no way to discover those unless the repo tells it where they are and how to read
them. `METRICS.md` is that map.

## Layered Intelligence decision contract

Layered Intelligence uses a metrics-first decision path:

1. When a metric shows a meaningful gap, it proposes the single highest-value
app change that can plausibly move that metric.
2. When the metrics are missing, unavailable, stale, or inconclusive, its
reasoning agent may inspect the repository read-only to understand the
relevant product flow and existing instrumentation. This is context discovery,
not permission to turn the run into a generic code-quality audit.
3. When that inspection identifies a specific missing signal or context that
blocks evaluation against a named goal, it files an `app-data-gap` issue for
the minimum visibility needed. The issue must identify the unanswered product
question, the expected data source, and acceptance criteria; “add more
telemetry” is not decision-complete.
4. It files nothing only when the available evidence is sufficient and healthy,
the needed visibility already exists, or no decision-complete proposal fits
the configured scopes (subject to the loop's filing safety and delivery
gates). Inconclusive metrics alone do not prove the app is healthy.

The reasoning agent never edits code in this task. A deterministic output hook
validates and deduplicates its structured proposal before filing it to the app's
configured tracker.

## Who reads it

- **Layered Intelligence** gathers repo-root `METRICS.md` as the `appMetrics`
Expand Down
3 changes: 3 additions & 0 deletions server/services/autonomousJobs/layeredIntelligenceHooks.js
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,9 @@ function buildCompletionContract() {
'You are a REASONING agent, not a coding agent. Do NOT edit code, run `/do:pr`,',
'commit, or open a pull request — any changes you make to this worktree are',
'discarded. Your ONLY output is the JSON described above.',
'You MAY inspect repository files and run read-only commands when the gathered',
'metrics are insufficient to understand the app context or identify a precise',
'visibility gap.',
'',
'When you have decided, write your result to the completion sentinel. Its',
`absolute path is printed in the **${PROGRAMMATIC_OUTPUT_COMPLETION_HEADING}**`,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,7 @@ describe('buildTaskInput', () => {
// this hook cannot know) — by the SHARED heading constant, so the pointer
// can't drift from the section agentPromptBuilder emits.
expect(res.prompt).toContain('completion sentinel');
expect(res.prompt).toContain('inspect repository files and run read-only commands');
expect(res.prompt).toContain(PROGRAMMATIC_OUTPUT_COMPLETION_HEADING);
// ...and it must NOT name any other filename. A prompt that says 'not a bare
// `.agent-done`' is the only place the agent would learn that wrong
Expand Down
22 changes: 14 additions & 8 deletions server/services/layeredIntelligence.playbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@

# LI Proposal Playbook

You (the reasoner) file blindly by default: you propose work without checking how
your past proposals fared, which produces a low merge rate and repeated
NOT_PLANNED rejections. This playbook is your correction. Treat it as a hard
Your purpose is to turn evidence about whether the app is meeting its goals into
at most one decision-complete improvement issue. This playbook prevents weakly
grounded proposals and repeated NOT_PLANNED rejections. Treat it as a hard
constraint on **which scope**, **which kind of work**, and **whether it aligns
with goals** — applied before you commit to any proposal.

Expand All @@ -38,9 +38,13 @@ require extra justification for scopes that historically get rejected.
block appears above (your execution health is degraded), `loop-meta` and
`portos-self` are HARD-EXCLUDED and will be dropped before filing — a degraded
loop cannot repair itself, so that work is deferred to a human (see §4).
- **Prefer `app-data-gap`.** Adding the telemetry / metrics / instrumentation
needed to reason well (e.g. a missing METRICS.md, an unmeasured KPI) lands
reliably: it is unambiguous, low-risk, and unblocks future higher-value work.
- **Prefer `app-data-gap` when visibility is the blocker.** Adding the telemetry,
metrics, instrumentation, or product context needed to answer a specific
performance question (e.g. a missing METRICS.md, an unmeasured KPI, or no
observable completion signal for a stated goal) lands reliably: it is
unambiguous, low-risk, and unblocks future higher-value work. First inspect the
relevant repository flow read-only so the proposal names the minimum missing
signal and its likely source. Never file a generic “add more telemetry” issue.
- **De-prioritize `app-improvement` unless alignment is explicit.** A generic
"improve the app" proposal without a clear tie to a stated goal, a measured
gap, or committed backlog is the single largest source of rejections. Only
Expand All @@ -49,8 +53,10 @@ require extra justification for scopes that historically get rejected.
- **`portos-self`** (PortOS install only): scoped, verifiable self-improvements
to PortOS as an app — hold to the same bar as `app-improvement`.

If nothing clears this bar, return `proposal: null`. Filing nothing is a
legitimate, and often the correct, outcome.
If a metric is inconclusive, do not treat that as healthy and stop immediately:
first determine whether a specific visibility gap prevents a sound decision. If
visibility is already adequate and nothing clears this bar, return
`proposal: null`. Filing nothing is a legitimate, and often the correct, outcome.

## 2. Success Pattern Catalog

Expand Down
17 changes: 16 additions & 1 deletion server/services/layeredIntelligence.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -1689,7 +1689,22 @@ describe('buildPrompt', () => {

it('frames the mission around the app\'s own goals and performance', () => {
const out = buildPrompt({ app, isPortos: false, config: { allowedScopes: ['app-improvement'], rules: '' } });
expect(out).toContain('its OWN goals and purpose');
expect(out).toContain('its OWN goals');
});

it('uses read-only app context to distinguish healthy metrics from a visibility gap', () => {
const out = buildPrompt({
app,
isPortos: false,
config: { allowedScopes: ['app-improvement', 'app-data-gap'], rules: '' },
sources: { appMetrics: 'Activation: unavailable' }
});
expect(out).toContain('inspect this repository READ-ONLY');
expect(out).toContain('Do not edit files');
expect(out).toContain('SPECIFIC visibility gap');
expect(out).toContain('never file a generic "add more telemetry" issue');
expect(out).toContain('Inconclusive metrics alone are not evidence that the app is healthy');
expect(out).toContain('AT MOST ONE decision-complete tracker issue');
});

it('always injects the static proposal playbook block (#2763)', () => {
Expand Down
8 changes: 7 additions & 1 deletion server/services/layeredIntelligence/prompt.js
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,13 @@ export function buildPrompt({ app, config, sources = {}, openIssues = [], isPort
? `\n### liHardExclusions\n${hardExclusionNotice.trim()}\n`
: '';

return `You are the Layered Intelligence reasoner for the app "${app.name}". Your job is to evaluate how THIS app is performing against its OWN goals and purpose${isPortos ? '' : ', not how well PortOS\'s tooling manages it'}. Decide the SINGLE highest-value improvement to propose this run (signal, not noise), grounded in the app's own goals and its own performance metrics (user success, KPIs, production telemetry). You never write code; you return structured JSON that a deterministic system files as ONE tracker issue. Optional \`model\` / \`effort\` dispatch hints and \`goodFirstIssue\` / \`helpWanted\` labels are independent of each other and of \`complexity\` — set them only when justified; omit rather than guessing; never stamp medium on both axes; never mark a wide mechanical sweep as a good first issue.
return `You are the Layered Intelligence reasoner for the app "${app.name}". Your purpose is to turn evidence about how THIS app performs against its OWN goals${isPortos ? '' : ', not how well PortOS\'s tooling manages it'} into AT MOST ONE decision-complete tracker issue that improves the app. Start with the app's own performance metrics (user success, KPIs, production telemetry), not a generic code-quality audit. You never write code; you return structured JSON that a deterministic system validates, deduplicates, and files. Optional \`model\` / \`effort\` dispatch hints and \`goodFirstIssue\` / \`helpWanted\` labels are independent of each other and of \`complexity\` — set them only when justified; omit rather than guessing; never stamp medium on both axes; never mark a wide mechanical sweep as a good first issue.

Decision path (follow in order):
1. If a metric shows a meaningful gap, identify the single highest-value app improvement that can plausibly move that metric and file that proposal.
2. If the metrics are missing, stale, unavailable, or too narrow/inconclusive to support an app improvement, inspect this repository READ-ONLY to understand the relevant user flow, architecture, existing instrumentation, and documented data sources. Do not edit files, and do not substitute an unrelated code audit for missing product evidence.
3. If that inspection reveals a SPECIFIC visibility gap that prevents the app from being evaluated against a named goal, file an app-data-gap proposal for the minimum metrics, instrumentation, or context needed. Name the goal, the unanswered product question, where the signal should come from, and acceptance criteria that make a future run actionable; never file a generic "add more telemetry" issue.
4. Return proposal: null only when the available evidence is sufficient and shows no worthwhile improvement, the needed visibility already exists, no decision-complete proposal fits an allowed scope, or a filing safety/delivery gate below requires no proposal. Inconclusive metrics alone are not evidence that the app is healthy.
${handoffNote}${metricsGuidance}${hardExclusionBlock}
Rules & guidance from the operator:
${config.rules?.trim() || '(none)'}
Expand Down
3 changes: 2 additions & 1 deletion server/services/taskSchedule.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,8 @@ describe('taskSchedule', () => {
describe('layered-intelligence (programmatic-I/O agent task)', () => {
it('is registered as a self-improvement task with a description and an on-demand default', () => {
expect(SELF_IMPROVEMENT_TASK_TYPES).toContain('layered-intelligence');
expect(TASK_TYPE_DESCRIPTIONS['layered-intelligence']).toBeTruthy();
expect(TASK_TYPE_DESCRIPTIONS['layered-intelligence']).toContain('performance metrics');
expect(TASK_TYPE_DESCRIPTIONS['layered-intelligence']).toContain('visibility gap');
expect(DEFAULT_TASK_INTERVALS['layered-intelligence']).toMatchObject({ type: INTERVAL_TYPES.ON_DEMAND, enabled: true });
});

Expand Down
2 changes: 1 addition & 1 deletion server/services/taskScheduleRegistry.js
Original file line number Diff line number Diff line change
Expand Up @@ -527,7 +527,7 @@ export const TASK_TYPE_DESCRIPTIONS = {
'stash-cleanup': 'Triage git stash list — drop entries superseded by or stale relative to main, leave real unlanded work in place',
'repo-sync': 'Sync every managed app with origin — back on the default branch, pushed and pulled, merged branches/worktrees and redundant stashes cleared',
'plan-feature': "Brainstorm one feature and file its decision-complete plan to the app's work tracker (no code)",
'layered-intelligence': "Read this app's goals + telemetry, ask a reasoning model for one improvement, and file one deduplicated tracker issue — no code, no agent"
'layered-intelligence': "Use app goals + performance metrics to file at most one deduplicated improvement issue; inspect read-only context and file a visibility gap when evidence is insufficient — no code"
};

export function getTaskTypeDescription(taskType) {
Expand Down