Skip to content

Decide whether workflow tasks should be first-class library components #38

@ProfessorPolymorphic

Description

@ProfessorPolymorphic

Context

A Vandalizer workflow is a sequential invocation of tasks. Today, workflow tasks
are authored as prompt_inline (and, for Extraction tasks, an inline searchset)
inside workflows/<slug>/manifest.yaml. There is no shared, reusable definition
of a task — the same task is re-authored per workflow.

Concrete trigger. We want a second award-compliance workflow that produces
human-readable output for Vandalizer users. The existing award-compliance-extraction
workflow emits schema-conformant JSON for ingest pipelines; the new one would reuse
its two extractor tasks (extract-compliance-framework, extract-financial-management)
and swap only the consolidator for a Markdown renderer. Today that reuse requires
copy-pasting both extractor prompts and their ~10-item searchsets into a second
manifest — two copies that will drift.

Proposal

Adopt the principle that a workflow is an ordered DAG of first-class
component-tasks
: every task references a component in the library instead of
inlining its prompt and searchset. prompt_ref already exists, and the
nsf-budget-justification -udm / -review / -render split shows the repo is
already moving this way. Related: #13 (single source of truth for prompt.md +
schema.json).

Open questions for the team

These are interdependent, which is why they sit in one issue rather than several.

1. Where does a component store its searchset?

A component today is prompt.md + optional schema.json. An Extraction task is
prompt + searchset, and there is nowhere in the component model to put the
searchset. prompt_ref slices prompt.md; build_vandalizer_workflows.py only
reads an inline task["searchset"].

Proposed: a searchset.yaml in the component directory. This is the keystone
decision — it requires changes to build_vandalizer_workflows.py,
.github/scripts/lint_components.py, build_component_catalog.py,
docs/contracts.md, CLAUDE.md, and the templates/new-component scaffold.

2. Are consolidator/renderer "glue" tasks first-class components?

Extractor tasks are clearly reusable contracts. Consolidator/renderer tasks are
workflow-specific glue — their contract is implicitly coupled to the fragment
shape their upstream extractors emit, and the component model has no way to
express "component C consumes A+B's output." Do glue tasks become components
anyway (consistent and independently evaluable, but single-consumer), or do they
stay prompt_inline in the workflow?

3. How do we migrate award-compliance-extraction-udm?

That component is currently a complete single-prompt extractor. Decomposing the
workflow into extractor components means it is either replaced by two fragment
components or kept alongside as an all-in-one. This is a contract-surface change
and a cross-repo trigger per CLAUDE.md (component_catalog.json, possibly
evaluation-harness / evaluation-data-sets).

Cost to weigh

Every component carries README.md, CHANGELOG.md, evals/, version
frontmatter, and a catalog entry. Promoting every task multiplies that envelope —
better eval granularity, but more surface to maintain.

Proposed sequencing (if adopted)

Once this issue resolves the three questions above, implementation lands as
separate issues:

  1. Settle and document the component model (docs/contracts.md + CLAUDE.md).
  2. Extend tooling (builder + lint + catalog) for searchset-bearing component refs.
  3. Pilot: extract the two award-compliance extractor components, migrate the
    existing structured workflow to reference them, prove the full round-trip green.
  4. Add the human-readable award-compliance workflow as the second consumer.

Decision checklist

  • Searchset home in the component model
  • Glue tasks: components or inline
  • award-compliance-extraction-udm migration approach

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions