Repository navigation
Paimon 1.0 full-compaction recovery can lose final changelog after APPEND #318
Description
Problem
Released Paimon 1.0.0 can acknowledge recovery of a full-compaction commit while permanently losing that commit's changelog. A batch scan still sees the newest value. A streaming consumer waiting for those final updates can therefore wait indefinitely.
This is reproduced with the released Paimon writer, commit and reader APIs, without a Flink planner, StreamFusion operators, or native kernels. It is independent of the writer cleanup defect fixed in #309.
Deterministic two-row reproducer
Create a one-bucket primary-key table (k INT, v INT) with changelog-producer=full-compaction and default deletion-vector settings:
- Write
(1,10), perform full compaction and commit. Start a stream scan and consume its initial snapshot. - Write
(1,20), perform full compaction, and prepare the final commit. Confirm its compact increment contains changelog files. - Fail one manifest read after the final APPEND snapshot is durable, before the separate COMPACT snapshot becomes durable.
- Recreate the commit object with the same commit user, and recover the original messages with
filterAndCommit. - Compare a fresh batch scan and the existing incremental stream scan.
| Scenario | Latest snapshot | Fresh batch values | Incremental changelog |
|---|---|---|---|
| No injected failure | COMPACT | [20] |
Includes 20 |
| Injected failure followed by recovery | APPEND | [20] |
[] |
The recovered case fails the unchanged requirement that the final changelog include 20. It does not merely hit a short deadline. The control passes and recovery fails on both JDK 11 and 17. Additional standalone runs remove every StreamFusion production class directory/JAR from the classpath and use only canonical released dependency JARs plus the diagnostic classes; they reproduce the same missing changelog. The earlier deletion-vector split-commit reproducer also still fails separately (20 expected, 10 observed).
Implementation evidence
The released FileStoreCommitImpl.commit writes APPEND and COMPACT separately under the same commit identifier. filterCommitted discards a pending identifier once any snapshot from that user has that identifier, so recovery after APPEND can discard the pending COMPACT and its changelog.
The current PR #315 CI failure is consistent with missing final changelog, but its exact cause is not established by this standalone reproducer:
- Paimon 1.0 parity job reports 241/242 upstream cases passed and
PrimaryKeyFileStoreTableITCase.testFullCompactionChangelogProducerStreamingRandomtimed out after 180 seconds. - Retained inventory shows one producer, two sink subtasks,
local-merge-buffer-size=5m, four buckets and partitions, 64 keys per partition, and injected filesystem failures. - The writer restores checkpoint 4 and later checkpoint 13, then finishes at 04:44:37 UTC. The consumer keeps completing checkpoints and waiting until the test timeout at 04:47:13 UTC.
- The artifact lacks the failed warehouse snapshots/manifests needed to prove the split-commit sequence for that invocation.
Proposed resolution
- Retain the failing CI warehouse and compare final APPEND/COMPACT snapshots, commit identifiers, manifests and changelog files. Replay both stock and native readers against the same retained state.
- Keep the deterministic two-row regression as an opt-in diagnostic; keep the original randomized upstream case and its assertions/deadline enabled.
- If the captured failure is this released SDK defect, the upstream recovery protocol must account for incomplete APPEND/COMPACT pairs rather than filtering the entire identifier. Validate retries, idempotence and crash points around both snapshots, then adopt a canonical verified upstream release.
- If native writing or scanning introduces an additional difference, address it separately with stock/native twins over identical snapshots and failure locations.
StreamFusion should retain released dependencies and exact stock semantics. Raising deadlines, weakening final-row assertions, editing upstream tests, or enabling unverified recovery behavior does not resolve missing durable changelog.
Activity
liuyongvs commentedon Oct 10, 2026
Revalidated on main
bf9837c00c3513c88a3739dcefab549f037da3fbwith the released dependencies. The split-commit recovery defect is still reproducible, and the released Paimon 2.0.0 line also loses final full-compaction changelog.PaimonSplitCommitRecoveryReproducerTest2010, expected20PaimonFullCompactionCommitRecoveryTest,-U 1In the Paimon 2.0.0 diagnostic, the fresh data scan still sees
20; the failing assertion is specifically the missing update changelog. Both diagnostics directly use released writer/committer/reader APIs and do not execute native operators.Reproduction commands from this commit:
Each diagnostic has one passing control and one failing injected-recovery invocation. This is separate from the normally enabled module suites: Paimon 1.x ran 95 records with its two opt-in diagnostic tests skipped; Paimon 2.x ran 697 records with its opt-in diagnostic class skipped. A green default module suite does not establish correctness at this split-commit failure boundary.
Please retain this issue as the recovery tracker across the affected released SDK lines. A verified canonical upstream release or an explicitly reviewed compatibility strategy is still needed; increasing consumer timeouts would not restore the missing durable changelog.
Establish lookup split-commit loss with released-only recovery evidence
liuyongvs commentedon Oct 12, 2026
PR #333's new Paimon 1.0 parity failure now retains enough durable state to identify the split-commit recovery boundary in an actual lookup invocation.
Head:
2f6295946034e09719ceadd21f5e8eee0125dd45, based on mainc048cfbe. The suite runs all 242 unchanged upstream tests; 241 pass andtestLookupChangelogProducerStreamingRandomtimes out after 180 seconds. This differs from the fetcher retirement exception addressed by #334.Table:
pt STRING, k INT, v1 BIGINT, v2 STRING, primary key(pt,k), partitioned bypt; four buckets,changelog-producer=lookup,lookup-wait=false, deletion vectors disabled, precommit compaction disabled. Two producers imply 512 terminal keys.The captured sequence is:
Long.MAX_VALUE.File deletion conflicts detected! Give up committing.after APPEND is durable. Recovery eventually finishes without any COMPACT snapshot for that final identifier.This is verified using the released stock reader with every StreamFusion production class/JAR excluded. Exact stock/native split replay also agrees on 73 splits and 2,290 changelog rows, including RowKinds and all four fields; the native replay executes its decoder over 83 files. The replay starts at snapshot 1 and is finite. It verifies captured durable state and reader parity, without reconstructing live scheduling or claiming that every concurrent native writer schedule matches stock.
Original evidence artifact contains the warehouse, snapshots, manifests, data/changelog files, Surefire report, inventory and diagnostics.
Follow-up commit
e97e9ea9in PR #333 extends the existing opt-in two-row released-SDK diagnostic tolookupas well asfull-compaction. On JDK 11 and 17, with StreamFusion production classes/JARs excluded:[20], changelog[][20], changelog[]The compact increment is checked to contain the final changelog before injecting the failure. The failure must occur after APPEND, and the batch assertion still requires the latest value. The missing changelog assertion is kept as a failure; this is not reported as a passing recovery test.
Reproduce the diagnostic with released dependencies:
mvn -Pflink-1.18,paimon1 -pl streamfusion-paimon1 -am \ -Dtest=PaimonFullCompactionRecoveryReproducerTest \ -Dsurefire.failIfNoSpecifiedTests=false \ -Dsf.paimon1.reproduceFullCompactionCommit=true testExpected current result: four invocations, two passing controls and two failing final-changelog assertions. The diagnostic remains opt-in; the original randomized upstream SQL test and its deadline remain enabled and unchanged.
This issue remains unresolved. The new evidence extends its scope to lookup and establishes the actual captured CI failure boundary; it does not repair the canonical released commit protocol. The connector documentation and
docs/benchmarks/results/paimon-lookup-split-commit-2026-10-12.jsonrecord the evidence and its limits.Reproduce released full-compaction recovery loss and retain failed state
liuyongvs commentedon Oct 13, 2026
PR #335 has another retained full-compaction failure at the same unresolved recovery boundary.
Paimon 1.0 compatibility job: 241/242 unchanged upstream cases pass;
PrimaryKeyFileStoreTableITCase.testFullCompactionChangelogProducerStreamingRandomtimes out after 180 seconds. The producer jobs eventually finish, but their captured final snapshots are incomplete:b8c9fb8c-5781-42ee-866f-0b5621f25c13Long.MAX_VALUE0068105e-d837-420f-b402-28b86b379a7cLong.MAX_VALUE0068105e-d837-420f-b402-28b86b379a7cLong.MAX_VALUEThere is no final COMPACT for the other commit user and no snapshot after 19. Each producer contributes 256 terminal keys, so the unchanged checker requires 512 terminal values.
Independent reads of the retained table use the released Paimon 1.0.0 APIs and released dependency JARs, with every StreamFusion production class directory/JAR excluded:
The streaming scan checkpoints snapshot 20 and has no further split. This establishes that the captured durable streaming state lacks half the terminal values even though the batch data contains them. Increasing the consumer deadline cannot create the missing final changelog.
This new read is a finite retained-state diagnostic. It does not replay every historical split from snapshot 1, reconstruct live writer scheduling, or establish full stock/native writer parity. The existing deterministic released-SDK two-row diagnostic and the earlier #333 historical replay remain separate evidence.
Main #336 changes string-window admission and does not change Paimon commit recovery. #335 has been rebased onto main
b2e6e629; its four Flink/JDK window-regression legs and Rust core tests pass. This rebase does not repair the released split-commit protocol. The original Paimon assertions, injected failures and 180-second deadline remain unchanged and enabled.The issue is currently closed, but the released recovery limitation remains reproducible. The remaining resolution still requires a verified canonical dependency release or an explicitly reviewed compatibility strategy that safely recovers incomplete APPEND/COMPACT pairs, including retries, idempotence and failures around both snapshots.
Document recovery that retains Paimon's snapshot protocol
Document Paimon recovery review and local workaround limits (#352)
jordepic commentedon Oct 15, 2026
Deferring StreamFusion-side work on this for now. This is an upstream Paimon recovery bug that also affects Paimon’s standard Flink connector, rather than a StreamFusion-specific defect; the released-SDK reproductions above demonstrate the underlying issue without StreamFusion.
We’ll revisit when a verified upstream fix is available in a canonical release. The proposed upstream fix is tracked in apache/paimon#10568.
Preserve released host behavior for upstream storage defects
jordepic commentedon Oct 15, 2026
Closing as not planned for StreamFusion, following the decision above. The incomplete APPEND/COMPACT recovery defect reproduces in released Paimon without StreamFusion and involves the core commit protocol also used by Paimon's stock Flink connector. The stock connector has lookup-specific recovery safeguards (apache/paimon#3512 and apache/paimon#5796), so this is not a claim that every stock Flink configuration fails.
We will not implement a StreamFusion-specific recovery workaround. Keep the released dependency and existing recovery diagnostics; reassess when a verified canonical upstream release is available. The proposed upstream fix remains apache/paimon#10568; closing this issue does not mean that defect is fixed.
Decision recorded in 3dfb0997.