fix: correct ShopifyQL schema discovery and timestamp handling#153
Open
naveen-meltano wants to merge 2 commits into
Open
fix: correct ShopifyQL schema discovery and timestamp handling#153naveen-meltano wants to merge 2 commits into
naveen-meltano wants to merge 2 commits into
Conversation
The sales_over_time stream (and any other config-driven ShopifyQLStream)
was silently broken end-to-end:
- schema discovery and row parsing both read
data.get("shopifyqlQuery") instead of unwrapping the standard GraphQL
"data" envelope, so every ShopifyQL response resolved to an empty
result with no error raised.
- primary_keys/replication_key were assigned before super().__init__(),
which unconditionally resets the backing _primary_keys/_replication_key
attributes, silently wiping the configured values.
- the discovered schema typed every column (including the replication
key) as a plain string, which fails singer_sdk's timestamp-based
bookmark logic.
- ShopifyQL returns date-only strings (e.g. "2026-07-03") for TIMESERIES
columns; normalize these to full timestamps in post_process so they
satisfy both the declared schema type and downstream TIMESTAMP-typed
destination columns.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
sales_over_timestream (and any other config-drivenShopifyQLStream) was silently broken end-to-end. Found and fixed while debugging a real pipeline run — extraction/discovery reported success with zero errors while the destination table stayed empty.dataenvelope unwrap:schemaandparse_responseboth readdata.get("shopifyqlQuery")instead of unwrapping the standard GraphQLdataenvelope ({"data": {"shopifyqlQuery": {...}}}). Every response silently resolved to{}, no error raised. This was a regression from an earlier PR review round that incorrectly claimed this endpoint has no envelope.primary_keys/replication_keyset beforesuper().__init__(): these are properties backed byself._primary_keys/self._replication_keyon the singer_sdk baseStreamclass, whose__init__unconditionally resets both to their defaults. Since the subclass set them before callingsuper().__init__(), the configured values were silently wiped every time.StringType: the discovered schema types every column asStringType(values are cast downstream in dbt), but the replication key column itself needs a real timestamp-typed schema (format: date-time) for singer_sdk's bookmark logic — otherwise it raisesInvalidReplicationKeyException/ValueError: not of timestamp type."2026-07-03") forTIMESERIEScolumns. Once typed as a timestamp, this fails downstreamTIMESTAMP-typed destination columns (e.g. BigQuery rejects date-only strings forTIMESTAMPload). Normalized inpost_processby appendingT00:00:00Zwhen the value is a bare date.These four bugs compound in sequence — fixing one surfaces the next — which is why they went unnoticed until traced end-to-end against a live pipeline run.
Test plan
day) against the live Shopify APIreplication_key/primary_keysresolve correctly post-constructionmeltano run tap-shopify target-bigqueryend-to-end;sales_over_timeextracted and loaded correctly with no duplicate rows and correct incremental bookmarking🤖 Generated with Claude Code