Replace FailureInfo with flat failure_category/failure_subcategory fields

[dejanzele]

FailureInfo bundled four fields (exit_code, termination_message, categories, container_name) that were already present on ContainerError. Replacing it with two flat strings on Error cuts wire size, removes the duplicate termination-message storage, and gives the ingester two text columns to write instead of a jsonb blob.

Classifier returns a single (category, subcategory) with first-match-wins semantics instead of []string of all matches. Adds a default category and per-rule subcategory. Gated behind a new enableJobErrorCategorization executor flag, off by default, so existing deployments are untouched until they opt in.

Proto: delete FailureInfo, reserve field 15, add failure_category (16) and failure_subcategory (17) on Error. Same on public api.JobFailedEvent which swaps categories []string for the two scalars.

Lookout: migration 032 adds the columns, ingester writes them in place of failure_info. The old column stays put here and is dropped in a follow-up once readers are on the new columns.

Follow-ups on top of this PR:

• Display failure_category/failure_subcategory in Lookout UI #4853 switches the Lookout read path and UI to the new columns
• Add job_failure_category_total counter metric to executor #4854 adds an executor counter job_failure_category_total
• Drop failure_info column from job_run table #4855 drops the unused failure_info jsonb column

[greptile-apps]

Greptile Summary

This PR replaces the FailureInfo proto message with two flat string fields (failure_category, failure_subcategory) on Error, and updates the classifier from returning all matching categories to first-match-wins semantics. The ingester writes the new text columns in place of the old failure_info jsonb blob, gated behind a new enableJobErrorCategorization executor flag that defaults to off.

Confidence Score: 5/5

Safe to merge; flag-gating ensures existing deployments are untouched and proto changes are backward-compatible.

All remaining findings are P2. No logic bugs or data-loss paths found. Previous P1 concerns about the v1 read path gap and nil-classifier safety are acknowledged in the PR and prior review. Proto field reservation is correct.

internal/lookout/schema/migrations/032_add_failure_category_to_job_run.sql — missing index; internal/lookoutingester/lookoutdb/insertion.go — verify parameter ordering after new column additions.

Important Files Changed

Filename	Overview
internal/lookout/schema/migrations/032_add_failure_category_to_job_run.sql	Adds failure_category and failure_subcategory as nullable text columns; no index added, which may hurt future queries in follow-up PRs #4853/#4854.
internal/lookoutingester/lookoutdb/insertion.go	Temp table, batch UPDATE, and scalar UPDATE all swapped failure_info jsonb for failure_category/failure_subcategory text columns; parameter counts look correct but no index added for the new columns.
internal/lookoutingester/instructions/instructions.go	Removed EnableJobRunFailureInfoMap flag and failureInfoToMap; ingester now writes failure_category/failure_subcategory as plain strings only when non-empty and terminal.
internal/executor/categorizer/classifier.go	Classifier refactored from returning all matching categories to first-match-wins (ClassifyResult with Category+Subcategory); nil-receiver safety on Classify() preserved.
internal/executor/reporter/event.go	FailureInfo removed; CreateJobFailedEvent now takes flat failure_category/failure_subcategory strings; CreateSimpleJobFailedEvent signature cleaned up.
pkg/armadaevents/events.proto	FailureInfo message deleted, field 15 reserved; failure_category (16) and failure_subcategory (17) added as flat strings on Error; backward-compatible proto changes.
pkg/api/event.proto	JobFailedEvent.categories (field 15) reserved; failure_category (16) and failure_subcategory (17) added as scalars; correct proto reservation.
internal/lookoutingester/model/model.go	UpdateJobRunInstruction.FailureInfo map replaced with FailureCategory/*string and FailureSubcategory/*string; cleaner model matching the new DB schema.

Sequence Diagram

sequenceDiagram
    participant Pod as Kubernetes Pod
    participant Executor as Executor
    participant Classifier as Classifier
    participant Event as armadaevents.Error
    participant Pulsar as Pulsar
    participant Ingester as LookoutIngester
    participant DB as job_run (Postgres)

    Pod->>Executor: PodFailed
    Executor->>Classifier: Classify(pod)
    Note over Classifier: First-match-wins across categories and rules
    Classifier-->>Executor: ClassifyResult{Category, Subcategory}
    Executor->>Event: CreateJobFailedEvent(..., category, subcategory)
    Note over Event: Error.failure_category = category
    Event->>Pulsar: publish EventSequence
    Pulsar->>Ingester: consume
    Ingester->>DB: UPDATE job_run SET failure_category, failure_subcategory

Loading

_{Reviews (19): Last reviewed commit: "Replace FailureInfo with flat failure_ca..." | Re-trigger Greptile}

[dejanzele]

@greptileai

[dejanzele]

@greptileai

[dejanzele]

Closing in favour of mode modular PR