Bug: fix MKRF bad curve cases before runtime generation

[gparadis]

Summary

Investigate and fix bad curve cases in the MKRF rebuild lane before continuing canonical runtime-package generation.

Parent issue: #173

Why this exists

The current rebuilt curve bundle has at least two clearly bad comparison cases:

• tipsy_vdyp_tsamkrf-11-CWHvm1_CW+FDC.png
• tipsy_vdyp_tsamkrf-12-CWHdmx_CW+FDC.png

In both cases the first-growth VDYP curve is effectively near-null while the managed curve is large, which indicates an upstream problem in source field choice, stand-to-unit assignment, unit definition, or curve fitting/aggregation. This needs to be resolved before we keep building the canonical runtime package.

Scope

• audit the bad first-growth curve cases against raw VDYP source rows and assignment lineage;
• determine whether the problem is source column choice, units, grouping, or fit/aggregation logic;
• fix the curve-build lane and regenerate affected artifacts and plots;
• record the acceptance rule for curve-quality gating before runtime generation continues.

Out of scope

• full canonical runtime-package generation
• broad model benchmarking after runtime generation

Immediate evidence

• external/femic-mkrf-instance/data/model_input_bundle/first_growth_au_curves.csv
• external/femic-mkrf-instance/data/model_input_bundle/first_growth_au_fit_diagnostics.csv
• external/femic-mkrf-instance/data/model_input_bundle/managed_au_curves.csv
• external/femic-mkrf-instance/plots/tipsy_vdyp_tsamkrf-11-CWHvm1_CW+FDC.png
• external/femic-mkrf-instance/plots/tipsy_vdyp_tsamkrf-12-CWHdmx_CW+FDC.png

[gparadis]

P60.7a is complete.

Implemented:

• checked-in femic instance mkrf-audit-bad-curves
• published data/model_input_bundle/bad_curve_audit_summary.csv
• published data/model_input_bundle/bad_curve_audit_detail.csv

Current audit result:

• flagged units: 15
• selected units: 31

This closes the audit slice and moves the active edge to P60.7b: correct the flagged curve cases by fixing source-field choice, grouping, assignment, or fit logic before runtime generation continues.

Validation:

• python -m pytest tests/test_mkrf_bad_curves.py tests/test_docs_contract.py::test_roadmap_is_a_compact_ordered_control_surface
• python -m ruff check src/femic/workflows/mkrf.py src/femic/cli/main.py tests/test_mkrf_bad_curves.py
• real run of femic instance mkrf-audit-bad-curves against Resultant.gdb + VDYP_Yields.csv

Commits:

• instance: bf25630 P60.7a Publish MKRF bad-curve audit bundle
• parent: 63c173f P60.7a Audit MKRF bad-curve cases

[gparadis]

P60.7b first correction slice is in.

Implemented:

• first-growth builder now excludes stands with AGE_2020 < 80 from the canonical VDYP first-growth lane
• plot rebuild now removes stale generated plot files before writing the new bundle

Regenerated:

• first_growth_au_curves.csv
• first_growth_au_fit_diagnostics.csv
• bad_curve_audit_summary.csv
• bad_curve_audit_detail.csv
• selected-unit plot bundle

Observed effect:

• flagged units dropped from 15 to 11
• managed-vs-first-growth plot count dropped from 4 to 2
• two previously bad cw/fdc comparison plots were removed because those units no longer have first-growth curve coverage after the age floor

Validation:

• python -m pytest tests/test_mkrf_plots.py tests/test_mkrf_first_growth.py tests/test_mkrf_bad_curves.py tests/test_docs_contract.py::test_roadmap_is_a_compact_ordered_control_surface
• python -m ruff check src/femic/pipeline/mkrf_first_growth.py src/femic/workflows/mkrf.py tests/test_mkrf_plots.py tests/test_mkrf_first_growth.py tests/test_mkrf_bad_curves.py
• real reruns of:
  • femic instance mkrf-build-first-growth-curves
  • femic instance mkrf-audit-bad-curves
  • femic instance mkrf-recompile-plots

Commits:

• instance: 913e135 P60.7b Apply MKRF first-growth age floor
• parent: 9a2e8c1 P60.7b Apply MKRF first-growth age floor

[gparadis]

P60.7b checkpoint: surviving bad-curve cases are now classified explicitly in the checked-in audit output.

Commits:

• parent: 39aaba2 P60.7b Classify MKRF surviving bad curves
• instance: 9bf5937 P60.7b Classify MKRF surviving bad curves

What changed:

• extended femic instance mkrf-audit-bad-curves to publish age-share diagnostics:
  • age_lt_20_count/share
  • age_lt_30_count/share
  • age_lt_80_count/share
  • age_gte_80_count/share
• added deterministic curve_issue_class classification to bad_curve_audit_summary.csv

Current post-age-floor flagged-case split:

• 7 units: no_first_growth_after_age_floor
• 2 units: young_skewed_population
• 2 units: mixed_population

This narrows the remaining work:

• most surviving flagged rows are now simply documenting that no canonical first-growth curve remains after the AGE_2020 >= 80 rule;
• the actual fix work is now concentrated on the 2 young-skewed cases and the 2 mixed-population cases.

[gparadis]

P60.7b checkpoint: applied a targeted sibling-borrow override for the clearly too-young cwh_dm_x_dr_cw first-growth case.

Commits:

• parent: 47a0b1d P60.7b Borrow MKRF young-skewed sibling curve
• instance: c44b3cc P60.7b Borrow MKRF young-skewed sibling curve

What changed:

• post-fit first-growth bundle now allows a narrow borrow override when a unit has:
  • almost no old-stand support; and
  • a low fitted terminal first-growth volume; and
  • a sane reversed-species sibling in the same BEC bucket
• this currently fixes cwh_dm_x_dr_cw by borrowing from cwh_dm_x_cw_dr
• diagnostics now record:
  • selected_path = borrowed_young_skewed_sibling
  • borrowed_from_au_id
  • borrow_reason

Result:

• surviving flagged units dropped from 11 to 10
• class split is now:
  • 7 no-first-growth-after-age-floor
  • 1 young-skewed-population
  • 2 mixed-population

[gparadis]

P60.7b correction slice completed.

Commits:

• instance: a94c10f P60.7b Reclassify MKRF insufficient-support curves
• parent: 0daf7e9 P60.7b Reclassify MKRF insufficient-support curves

What changed:

• the canonical first-growth builder now refuses to fit a VDYP first-growth curve when a unit has fewer than 2 unique old-enough source stands;
• the bad-curve audit now classifies missing curves against unique old-support stand counts rather than fragment-row counts;
• regenerated:
  • external/femic-mkrf-instance/data/model_input_bundle/first_growth_au_curves.csv
  • external/femic-mkrf-instance/data/model_input_bundle/first_growth_au_fit_diagnostics.csv
  • external/femic-mkrf-instance/data/model_input_bundle/bad_curve_audit_summary.csv
  • external/femic-mkrf-instance/data/model_input_bundle/bad_curve_audit_detail.csv

Current audit result:

• 14 flagged units out of 31 selected units
• flagged split is now:
  • 8 insufficient_source_stands
  • 5 no_first_growth_after_age_floor
  • 1 mixed_population

Most important concrete correction:

• cwh_vm_1_dr_hw no longer carries a bogus fitted first-growth curve;
• it now lands in insufficient_source_stands with old_support_stand_count = 1, which matches the actual builder support condition.

Validation:

• python -m pytest tests/test_mkrf_first_growth.py tests/test_mkrf_bad_curves.py tests/test_mkrf_plots.py
• python -m ruff check src/femic/pipeline/mkrf_first_growth.py src/femic/workflows/mkrf.py tests/test_mkrf_first_growth.py tests/test_mkrf_bad_curves.py
• real reruns:
  • femic instance mkrf-build-first-growth-curves ...
  • femic instance mkrf-audit-bad-curves ...

[gparadis]

P60.7b correction slice completed.

Commits:

• instance: ddebe54 P60.7b Merge MKRF insufficient-support curves
• parent: 3a7a7b7 P60.7b Merge MKRF insufficient-support curves

What changed:

• added a deterministic post-fit merge rule for insufficient_source_stands first-growth units;
• merge contract is now:
  • same BEC bucket required;
  • prefer more shared species;
  • then prefer larger covered area;
• regenerated:
  • external/femic-mkrf-instance/data/model_input_bundle/first_growth_au_curves.csv
  • external/femic-mkrf-instance/data/model_input_bundle/first_growth_au_fit_diagnostics.csv
  • external/femic-mkrf-instance/data/model_input_bundle/bad_curve_audit_summary.csv
  • external/femic-mkrf-instance/data/model_input_bundle/bad_curve_audit_detail.csv

Current audit result:

• 9 flagged units out of 31 selected units
• flagged split is now:
  • 2 insufficient_source_stands
  • 5 no_first_growth_after_age_floor
  • 2 mixed_population

Examples of merged units:

• cwh_vm_1_dr_hw -> cwh_vm_1_hw_cw
• cwh_vm_1_fdc_cw -> cwh_vm_1_hw_cw
• cwh_dm_x_dr_cw -> cwh_dm_x_hw_cw

Validation:

• python -m pytest tests/test_mkrf_first_growth.py tests/test_mkrf_bad_curves.py tests/test_mkrf_plots.py
• python -m ruff check src/femic/workflows/mkrf.py tests/test_mkrf_bad_curves.py
• real reruns:
  • femic instance mkrf-build-first-growth-curves ...
  • femic instance mkrf-audit-bad-curves ...

[gparadis]

Follow-up for P60.7: the managed comparison side has been rebuilt from expert-rule curves rather than the old 4-AU legacy bootstrap ceiling.

Relevant commits:

• parent: f06885b
• instance: f0a6b42

What changed on the comparison side:

• config/tipsy/tsamkrf.yaml is now the active managed-rule authority;
• the managed lane now covers all 31 selected AUs;
• stand_origin_assignment.csv publishes fire_origin / logging_origin from AGE_2020;
• the managed-vs-first-growth plot bundle was rerun against managed_au_curves.csv rebuilt from the expert-rule BTC lane.

Current plot counts after the rerun:

• 28 vdyp_lmh_tsamkrf-*.png
• 80 vdyp_fitdiag_tsamkrf-*.png
• 24 tipsy_vdyp_tsamkrf-*.png

This does not close P60.7. The bad-curve gate remains active, but it is now operating against the rebuilt managed comparison surface rather than the earlier 4-AU legacy bootstrap surface.

[gparadis]

P60.7b is complete.

Implemented and committed:

• parent: 37c02f8 P60.7b Clear MKRF bad-curve blockers
• instance: 3de8d9b P60.7b Publish MKRF corrected first-growth curves

What changed:

• enforced the AGE_2020 >= 80 first-growth floor in the canonical MKRF first-growth lane;
• reclassified zero-old-support units as managed-only after the age floor;
• merged insufficient-support units onto sane same-BEC neighbors using fragment-level old-support accounting aligned with the audit surface;
• tightened donor selection so low-terminal donor curves are rejected; and
• added a narrow observed_bin_tail_rescue fit path for severe right-tail underfit against observed 5-year VDYP medians.

Regenerated:

• external/femic-mkrf-instance/data/model_input_bundle/first_growth_au_curves.csv
• external/femic-mkrf-instance/data/model_input_bundle/first_growth_au_fit_diagnostics.csv
• external/femic-mkrf-instance/data/model_input_bundle/bad_curve_audit_summary.csv
• external/femic-mkrf-instance/data/model_input_bundle/bad_curve_audit_detail.csv

Result:

• bad-curve audit now reports 0 flagged AUs out of 31 selected AUs.
• the final surviving blocker cwh_vm_1_cw_hw now lands on selected_path = observed_bin_tail_rescue with terminal first-growth volume 279.922306.

Validation run in this slice:

• .venv\Scripts\python.exe -m pytest tests/test_mkrf_bad_curves.py tests/test_mkrf_first_growth.py tests/test_docs_contract.py::test_roadmap_is_a_compact_ordered_control_surface
• .venv\Scripts\python.exe -m ruff check src/femic/pipeline/mkrf_first_growth.py src/femic/workflows/mkrf.py tests/test_mkrf_bad_curves.py tests/test_mkrf_first_growth.py
• real reruns:
  • femic instance mkrf-build-first-growth-curves ...
  • femic instance mkrf-audit-bad-curves ...

Active edge moves to P60.7c: rerun the canonical diagnostic/comparison plots and record the curve-quality acceptance gate before any P60.8 runtime-package generation.

[gparadis]

P60.7c checkpoint landed.

What changed:

• locked canonical MKRF first-growth publication to AU-local curves only;
• removed sibling / same-BEC whole-curve borrowing from the bundle builder;
• kept the accepted first-growth curve family as strongly smoothed observed-bin PCHIP where AU-local support is adequate;
• fixed the L/M/H fitdiag overlay seam so diagnostics now use the same age-eligible stand set as the fitted curve.

Commits:

• parent: 53cef91 P60.7c Lock MKRF first-growth to local curves
• instance: 2e12afd P60.7c Publish MKRF local-only first-growth surfaces

Current rebuilt state:

• 20 AU-local smoothed_bin_pchip first-growth curves
• 11 insufficient_source_stands
• bad-curve audit: 8 flagged selected AUs, all insufficient_source_stands
• plot bundle:
  • 16 vdyp_lmh_tsamkrf-*.png
  • 37 vdyp_fitdiag_tsamkrf-*.png
  • 18 tipsy_vdyp_tsamkrf-*.png

Interpretation:

• the current surface is no longer masking weak first-growth evidence with borrowed curves;
• the remaining open question is now explicit policy for the insufficient-support AUs before P60.8 runtime generation.

[gparadis]

P60.7 is complete and the MKRF curve-quality gate is now explicitly recorded.

Closeout summary:

• the accepted direct first-growth curve family is smoothed_bin_pchip;
• canonical first-growth publication is AU-local only;
• whole-curve sibling / same-BEC borrowing is no longer allowed in first_growth_au_curves.csv;
• current accepted surface:
  • 20 AU-local first-growth curves
  • 11 insufficient_source_stands
  • 8 flagged selected AUs, all insufficient-support and all already covered by managed curves
  • 18 canonical managed-vs-first-growth comparison plots;
• runtime policy for the 8 flagged AUs is now explicit: treat them as managed-only runtime units and do not require, synthesize, or borrow first-growth curves for them.

Docs / planning surfaces updated:

• ROADMAP.md
• planning/mkrf_femic_native_rebuild.md
• CHANGE_LOG.md

Validation:

• python -m pytest tests/test_docs_contract.py::test_roadmap_is_a_compact_ordered_control_surface
• git diff --check

This closes #177 because its scope was to audit, correct, and formalize the curve-quality gate before runtime generation. The remaining work now belongs to P60.8+ under parent issue #173.