Skip to content

feat(tracker): catch bypass paths via extra observers + reconciler cron (#1) - #10

Merged
mageaustralia merged 1 commit into
mainfrom
feat/tracker-bypass-reconciler
Jun 3, 2026
Merged

feat(tracker): catch bypass paths via extra observers + reconciler cron (#1)#10
mageaustralia merged 1 commit into
mainfrom
feat/tracker-bypass-reconciler

Conversation

@mageaustralia

Copy link
Copy Markdown
Owner

Fixes #1.

Problem

Products (and other entities) edited on the source can fail to land in datasync_change_tracker, so the destination incremental never re-syncs them and they drift indefinitely after the initial bulk load. The current observers miss several write paths that still bump updated_at.

Fix (source module, two complementary parts)

1. Observers for the known bypass events

  • catalog_product_save_commit_after - some 3rd-party modules dispatch only the commit variant
  • catalog_product_media_save_after / catalog_product_gallery_save_after - image/gallery saves that skip product_save_after, handled by a new trackProductMedia() that resolves the product id defensively from the (caller-dependent) event payload
  • cataloginventory_stock_item_save_commit_after

The unique (entity_type, entity_id, sync_completed) index dedups, so these firing alongside the existing events is harmless (ON DUPLICATE KEY UPDATE).

2. Catch-all reconciler cron (reconcileChanges(), every 10 min)

Arbitrary bypass paths can't be enumerated (direct SQL updated_at bumps, saveAttribute(), parent rollups, 3rd-party connectors), but divergence is always detectable after the fact. The cron INSERTs a pending tracker row for any product / category / customer whose source updated_at is newer than its latest tracker row:

INSERT INTO datasync_change_tracker (entity_type, entity_id, action, created_at, sync_completed)
SELECT ?, s.entity_id, 'update', s.updated_at, 0
FROM <source> s
LEFT JOIN (SELECT entity_id, MAX(created_at) last_tracked
           FROM datasync_change_tracker WHERE entity_type=? GROUP BY entity_id) t
  ON t.entity_id = s.entity_id
WHERE s.updated_at > COALESCE(t.last_tracked, '1970-01-01 00:00:00')
ON DUPLICATE KEY UPDATE created_at = VALUES(created_at), sync_completed = 0;

Idempotent; logs flagged counts per entity to datasync.log.

Deliberate scope notes

  • Stock excluded from the reconciler: cataloginventory_stock_item has no updated_at in Magento 1, so it can't be reconciled by timestamp. Stock writes are caught by trackStock() (now also on commit_after), and a parent rollup from a child stock change surfaces as a catalog_product_entity.updated_at bump, covered by the product reconcile.
  • Source-side OpenMage/Magento 1 module, so Varien_* / Zend_Log are intentional (not Maho).
  • Version bumped 1.0.0 -> 1.1.0. php -l clean, config.xml validates.

Test plan

  • Bump a product's updated_at via direct SQL (no event), wait for the reconciler cron (or run the job), confirm a pending row appears in datasync_change_tracker and a reconcile: product flagged N line in datasync.log.
  • Save only a product image; confirm the product is flagged via catalog_product_media_save_after.
  • Re-run the reconciler immediately; confirm it's a no-op (no duplicate pending rows, dedup via the unique index).

…on (#1)

Products edited on the source could miss the change tracker and drift on
the destination indefinitely. Two complementary fixes on the source module:

1. Observers for the bypass events that still move updated_at but skip
   catalog_product_save_after:
   - catalog_product_save_commit_after (some 3rd-party modules fire only this)
   - catalog_product_media_save_after / catalog_product_gallery_save_after
     (image/gallery saves) via a new trackProductMedia() that resolves the
     product id defensively from the varying event payload
   - cataloginventory_stock_item_save_commit_after
   The unique (entity_type, entity_id, sync_completed) index dedups, so the
   extra events are harmless when the standard event also fires.

2. A catch-all reconciler cron (every 10 min, reconcileChanges()) that INSERTs
   a pending tracker row for any product/category/customer whose source
   updated_at is newer than its latest tracker row. This is the only way to
   catch arbitrary bypass paths (direct SQL updated_at bumps, saveAttribute(),
   parent rollups, 3rd-party connectors) - they can't be enumerated, but
   divergence is always detectable after the fact. Idempotent via
   ON DUPLICATE KEY UPDATE.

Stock is intentionally excluded from the reconciler: cataloginventory_stock_item
has no updated_at in Magento 1. Stock writes are caught by trackStock(), and a
parent rollup from a child stock change surfaces as a product updated_at bump.

Source-side OpenMage/Magento 1 module (Varien_*/Zend_Log correct here).
Version bumped 1.0.0 -> 1.1.0.

Fixes #1
@mageaustralia
mageaustralia merged commit d1169d7 into main Jun 3, 2026
1 check passed
@mageaustralia
mageaustralia deleted the feat/tracker-bypass-reconciler branch June 3, 2026 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tracker misses bypass paths → silently stale products

1 participant