Fix DataFrame truthiness crash in _validate_kinfile; surface pdp3 failures - #34
Merged
Conversation
…lures _validate_kinfile used `if kin_df and not kin_df.empty`, which raises `ValueError: The truth value of a DataFrame is ambiguous` whenever an existing kin file parses into a DataFrame — so the cached-result check crashed instead of skipping re-processing. earthscope-sfg-workflows currently monkeypatches the method to work around this; with this fix that workaround can be dropped. _run_pdp3 now also logs an error when pdp3 exits non-zero (returncode + stderr tail) and when a run produces no kin output, so failed runs are visible in logs instead of only in the ProcessingResult fields. Regression tests cover the valid-kinfile path (raised before the fix), missing path, override=True, and unparseable files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
frigusgulo
pushed a commit
to EarthScope/earthscope-sfg-workflows
that referenced
this pull request
Jul 31, 2026
sv3_pipeline.py carried three monkey-patches applied at import time: write_config_file (inject 'ISB model' / 'AI Ambiguity validation' pdp3 keys), PrideProcessor._validate_kinfile (DataFrame truthiness crash), and TBDArray.write_df (promote 'time' column to index for tiledb.from_pandas). All three are now fixed at the source: - pride-ppp: config keys fixed in EarthScope/GNSSommelier#33 (merged), _validate_kinfile fixed in EarthScope/GNSSommelier#34 - earthscope-sfg-tools: write_df fixed in EarthScope/earthscope-sfg-tools#19 (which also makes the Go-binary wrappers raise on non-zero exit instead of failing silently) pyproject now pins pride-ppp to an explicit GNSSommelier rev (it was unpinned, resolving default-branch HEAD at lock time) and moves the sfg-tools pin from 0.2.0 to the fix commit; pixi.lock re-solved accordingly. Once the upstream PRs merge, both pins should move to merged-main revs (sfg-tools 0.2.1 tag) with a single re-lock. Removing the base-class write_df patch also ends qc_pipeline's silent dependence on sv3_pipeline being imported first for its qcKinPositionTDB writes to work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
process() logged 'Missing required products' and then ran pdp3 anyway with an incomplete config — a run guaranteed to fail downstream, and invisibly so for callers that only checked the log. It now raises MissingProductsError (exported from pride_ppp) unless a valid cached kin file short-circuits the run first. process_batch() stays batch-robust: files whose date is missing required products yield a failed ProcessingResult (returncode -1, stderr lists the missing specs) without dispatching pdp3, and other dates in the batch process normally. New tests cover _run_pdp3 via a fake pdp3 script on PATH (output moving/renaming, non-zero exit + no-kin-output error logging) and both missing-products paths, with a guard that pdp3 is never invoked for an unfulfilled date. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PrideProcessor._validate_kinfileusedif kin_df and not kin_df.empty, which raisesValueError: The truth value of a DataFrame is ambiguousfor any existing kin file that parses into a DataFrame. The cached-result check therefore crashed instead of skipping re-processing. Fixed toif kin_df is not None and not kin_df.empty._run_pdp3now logs an error when pdp3 exits non-zero (returncode + stderr tail) and when a run produces no kin output, so silent 0-KIN-file runs (cf. pdp3 config_file missing 'AI Ambiguity validation' key → every PPP run silently produces 0 KIN files #28) are visible in logs.process()no longer launches pdp3 when required products are missing — it previously loggedMissing required productsand ran a doomed pdp3 job anyway. It now raisesMissingProductsError(newly exported frompride_ppp).process_batch()stays batch-robust: affected files yield a failedProcessingResult(returncode == -1, missing specs instderr) without dispatching pdp3, and other dates process normally.earthscope-sfg-workflowscurrently monkeypatches_validate_kinfileat the top ofsv3_pipeline.pyto work around the truthiness bug — EarthScope/earthscope-sfg-workflows#23 drops that patch in favor of this fix.Test plan
TestValidateKinfileregression tests: valid kin file → True (raisedValueErrorbefore the fix — verified against the unfixed code), missing path → False,override=True→ False, unparseable file → False.TestRunPdp3tests drive_run_pdp3through a fakepdp3script on PATH: outputs are moved/renamed correctly, and non-zero exit + missing kin output are logged.TestMissingRequiredProductstests cover theprocess()raise and theprocess_batch()failed-result path, with a guard asserting pdp3 is never invoked for an unfulfilled date.uv run pytest packages/pride-ppp/tests/ -q→ 49 passed;ruff check/ruff format --checkclean.