feat(tracker): catch bypass paths via extra observers + reconciler cron (#1) - #10
Merged
Merged
Conversation
…on (#1) Products edited on the source could miss the change tracker and drift on the destination indefinitely. Two complementary fixes on the source module: 1. Observers for the bypass events that still move updated_at but skip catalog_product_save_after: - catalog_product_save_commit_after (some 3rd-party modules fire only this) - catalog_product_media_save_after / catalog_product_gallery_save_after (image/gallery saves) via a new trackProductMedia() that resolves the product id defensively from the varying event payload - cataloginventory_stock_item_save_commit_after The unique (entity_type, entity_id, sync_completed) index dedups, so the extra events are harmless when the standard event also fires. 2. A catch-all reconciler cron (every 10 min, reconcileChanges()) that INSERTs a pending tracker row for any product/category/customer whose source updated_at is newer than its latest tracker row. This is the only way to catch arbitrary bypass paths (direct SQL updated_at bumps, saveAttribute(), parent rollups, 3rd-party connectors) - they can't be enumerated, but divergence is always detectable after the fact. Idempotent via ON DUPLICATE KEY UPDATE. Stock is intentionally excluded from the reconciler: cataloginventory_stock_item has no updated_at in Magento 1. Stock writes are caught by trackStock(), and a parent rollup from a child stock change surfaces as a product updated_at bump. Source-side OpenMage/Magento 1 module (Varien_*/Zend_Log correct here). Version bumped 1.0.0 -> 1.1.0. Fixes #1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1.
Problem
Products (and other entities) edited on the source can fail to land in
datasync_change_tracker, so the destination incremental never re-syncs them and they drift indefinitely after the initial bulk load. The current observers miss several write paths that still bumpupdated_at.Fix (source module, two complementary parts)
1. Observers for the known bypass events
catalog_product_save_commit_after- some 3rd-party modules dispatch only the commit variantcatalog_product_media_save_after/catalog_product_gallery_save_after- image/gallery saves that skipproduct_save_after, handled by a newtrackProductMedia()that resolves the product id defensively from the (caller-dependent) event payloadcataloginventory_stock_item_save_commit_afterThe unique
(entity_type, entity_id, sync_completed)index dedups, so these firing alongside the existing events is harmless (ON DUPLICATE KEY UPDATE).2. Catch-all reconciler cron (
reconcileChanges(), every 10 min)Arbitrary bypass paths can't be enumerated (direct SQL
updated_atbumps,saveAttribute(), parent rollups, 3rd-party connectors), but divergence is always detectable after the fact. The cron INSERTs a pending tracker row for any product / category / customer whose sourceupdated_atis newer than its latest tracker row:Idempotent; logs flagged counts per entity to
datasync.log.Deliberate scope notes
cataloginventory_stock_itemhas noupdated_atin Magento 1, so it can't be reconciled by timestamp. Stock writes are caught bytrackStock()(now also on commit_after), and a parent rollup from a child stock change surfaces as acatalog_product_entity.updated_atbump, covered by the product reconcile.Varien_*/Zend_Logare intentional (not Maho).php -lclean,config.xmlvalidates.Test plan
updated_atvia direct SQL (no event), wait for the reconciler cron (or run the job), confirm a pending row appears indatasync_change_trackerand areconcile: product flagged Nline indatasync.log.catalog_product_media_save_after.