Skip to content

Feature: refactor FEMIC around named pipelines built from recipe sequences #163

@gparadis

Description

@gparadis

Summary

Refactor FEMIC away from the legacy monolithic 00_data-prep.py, 01a_run-tsa.py, and 01b_run-tsa.py mental model toward a named-pipeline system built from reusable recipes, overlays, and runbook-defined restart seams.

Why

FEMIC is already moving toward recipe-driven execution, but the overall workflow model is still fragmented across legacy stage scripts and ad hoc runtime seams. We want a simpler and more flexible user-facing concept:

  • FEMIC runs a named pipeline;
  • each pipeline is just a sequence of named recipes;
  • recipes can come from the system registry, user-contributed registries, or local registries;
  • pipelines can be customized with overlays and overrides; and
  • model instances become reproducible because their runbooks describe exactly which pipeline and recipe sequence to execute.

This should preserve recent recipe-based work rather than invalidate it.

Scope of the umbrella

Track the larger architectural shift needed to support:

  • named pipeline registries;
  • recipe-sequence execution as the primary workflow model;
  • restart seams and canonical checkpoint contracts inside pipelines;
  • downstream recipes that can target Patchworks, ws3, reporting, scenario sets, indicator banks, and other extensible workflow units.

Child workstreams

  • #164 closed: explicit AFLB -> strata/AU/yield -> THLB interruption seam inside the THLB workflow.
  • #167 active: define named pipeline registry and runbook contracts.
  • #168 closed: implement the first named-pipeline proof runner from runbooks.
  • #169 active: validate TSA29 strict named pipeline against the locked-chain contract.

Rollout note

  • #168 proved orchestration.
  • #169 proves reproducibility against the validated strict contract.

Constraints

  • This is an umbrella/architecture issue, not one giant implementation ticket.
  • Child issues should stay narrowly scoped and executable.
  • The design should remain consistent with the recipe-based work already completed in the TSA29 THLB reconciliation lane.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions