Feature/sedm student iep disabilities

[ryanaguilar]

feature/sedm: Student IEP disability collections staging, Ed-Fi 3 key alignment, and IDEA event field fixes

Description & motivation

This PR extends SEDM staging to surface student IEP disability collections from Ed-Fi 3: a base model on student_iep_disability_collections and a staging model that keys to the parent IEP, deduplicates at the IEP grain, flattens the embedded disabilities array, and exposes disability-related descriptors for downstream use.

It also aligns gen_skey() definitions for k_student_iep and k_idea_event with the camelCase JSON property names used on Ed-Fi 3 studentIEPReference / IDEA event payloads (consistent with other keys in gen_skey.sql that read from VARIANT references). base_sedm__idea_events is updated to read begin/end dates from beginDate and endDate instead of eventBeginDate and eventEndDate, matching the API shape.

Finally, stg_sedm__student_ieps surrogate key inputs are reordered to mirror the IEP natural key ordering, and dbt_utils is allowed up through 1.3.3 in packages.yml.

Breaking changes introduced by this PR:

Potentially breaking: If gen_skey('k_student_iep') or gen_skey('k_idea_event') previously produced hashes that downstream models persisted or joined on, recomputing those keys with corrected JSON paths may change key values for some rows. Downstream should reconcile joins and any materialized keys after refresh.

Potentially breaking: base_sedm__idea_events now maps event_begin_date / event_end_date from v:beginDate and v:endDate. Consumers that depended on the old eventBeginDate / eventEndDate paths may see different populated dates if the source payload uses the canonical fields.

Not breaking by default: New models are gated by var('src:domain:sedm:enabled', False) like other SEDM models.

PR Merge Priority:

• Low
• Medium
• High

Changes to existing files:

• macros/gen_skey.sql: For k_idea_event and k_student_iep, updated col_list entries to use Ed-Fi 3 JSON keys (educationOrganizationId, iepFinalizedDate, studentIEPAssociationId, studentUniqueId, and IDEA event equivalents) so keys are derived consistently from reference VARIANTs.

• models/staging/sedm/base/base_sedm__idea_events.sql: Renamed VARIANT accessors for event window fields from v:eventBeginDate / v:eventEndDate to v:beginDate / v:endDate for event_begin_date and event_end_date.

• models/staging/sedm/stage/stg_sedm__student_ieps.sql: Reordered fields in dbt_utils.generate_surrogate_key() for k_student_iep to align with IEP identifier ordering (ed_org_id, iep_finalized_date, then association and student ids).

• models/staging/sedm/stage/_sedm_stage.yml: Registered stg_sedm__student_iep_disability_collections with SEDM tags and the same src:domain:sedm:enabled gate as other SEDM stage models.

New files created:

• models/staging/sedm/base/base_sedm__student_iep_disability_collections.sql: Base layer on source_edfi3('student_iep_disability_collections'); extracts tenant/API metadata, studentIEPReference scalars and nested reference, v:disabilities as v_disabilities, and _ext for extensions.

• models/staging/sedm/stage/stg_sedm__student_iep_disability_collections.sql: Builds k_student_iep (same surrogate pattern as stg_sedm__student_ieps), adds school_year, applies extract_extension, deduplicates with partition_by='k_student_iep, ed_org_id' ordered by latest modification/pull, flattens v_disabilities via json_flatten, filters out deleted rows, and exposes disability descriptor fields (disability_descriptor, disability_determination_source_type_descriptor, disability_diagnosis, order_of_disability).

Tests and QC done:

• dbt run --target data_dev successful in stadium_boston dev and stadium_south_carolina dev for base and stg models for student_ieps, student_iep_disabilities, and idea_events

Submitter checklist

• Description of changes has been added to Unreleased section of CHANGELOG.md. Add under ## New Features for features, etc.
• Code has been tested/checked for Databricks and Snowflake compatibility - EA engineers see Databricks checklist here
• Reviewer confirms the grain of all tables are unchanged, OR any changes are expected, communicated, and this PR is flagged as a breaking change (not for patch release). Grain for 2 existing models did not change.
• If a new configuration xwalk was added: n/a
  • The code is written such that the xwalk is optional (preferred), and this behavior was tested, OR n/a
  • The code is written such that the xwalk is required, and the required xwalk is added to edu_project_template, and this PR is flagged as breaking change (not for patch release) n/a
  • A description for the new xwalk has been added to EDU documentation site here n/a
• If a new configuration variable was added:
  • The code is written such that the variable is optional (preferred), and this behavior was tested, OR
  • The code is written such that the variable is required, and a default value was added to edu_project_template, and this PR is flagged as breaking change (not for patch release)
  • A description for the new variable has been added to EDU documentation site here

edu_wh PR Review Checklist:

Make sure the following have been completed before approving this PR:

• Description of changes has been added to Unreleased section of CHANGELOG.md. Add under ## New Features for features, etc.
• Code has been tested/checked for Databricks and Snowflake compatibility - EA engineers see Databricks checklist here
• Reviewer confirms the grain of all tables are unchanged, OR any changes are expected, communicated, and this PR is flagged as a breaking change (not for patch release)
• If a new configuration xwalk was added:
  • The code is written such that the xwalk is optional (preferred), and this behavior was tested, OR
  • The code is written such that the xwalk is required, and the required xwalk is added to edu_project_template, and this PR is flagged as breaking change (not for patch release)
  • A description for the new xwalk has been added to EDU documentation site here
• If a new configuration variable was added:
  • The code is written such that the variable is optional (preferred), and this behavior was tested, OR
  • The code is written such that the variable is required, and a default value was added to edu_project_template, and this PR is flagged as breaking change (not for patch release)
  • A description for the new variable has been added to EDU documentation site here

[rlittle08]

i didn't review the disability collections models, bc i don't see it documented in ed-fi 6.1 docs - was it not part of that release?

[rlittle08]

based on API docs, I think we need ideaEventTypeDescriptor instead of ideaEvent

[rlittle08]

thanks for this catch!

[rlittle08]

the edfi handbook and api docs seem to call this StudentIEPIdentifier -- did that change?

[rlittle08]

i don't see this resource in the ed-fi handbook or api docs, is it not part of the 6.1 release?