Teradata support in dbt-expectations#38
Open
vs255034 wants to merge 11 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of Changes
This PR adds comprehensive Teradata adapter support to dbt-expectations, enabling all core data quality tests to run successfully on Teradata databases while maintaining backward compatibility with all existing adapters.
Why Do We Need These Changes
Teradata has several SQL syntax and functional differences compared to other databases that require adapter-specific implementations:
Key Teradata Limitations Addressed:
No nested CTEs: Teradata doesn't support WITH clauses inside other WITH clauses (Error 6926)
No boolean literals: Must use integers (1/0) instead of true/false (Error 5628)
Strict timestamp handling: Requires careful casting between DATE and TIMESTAMP types (Error 6760)
Limited date arithmetic: Doesn't support INTERVAL multiplication syntax (Error 3706)
DateTime operation restrictions: Hour-based dateadd operations in moving window calculations not supported (Error 5407)
FROM clause requirement: Literal SELECT statements require a FROM clause
Changes Implemented:
Added Teradata-specific implementations using adapter dispatch pattern ({% if target.type == 'teradata' %})
Modified macros in categories:
Table shape tests: Flattened CTEs, added FROM clauses
Column value tests: Boolean literal replacements, datatype adjustments
Distributional tests: Alternative statistical calculations
Date/time tests: Conditional casting logic for DATE vs TIMESTAMP handling
Applied adapter dispatch to test models: data_test.sql, data_test_factored.sql, data_text.sql, timeseries_*.sql, window_function_test.sql
Added conditional date handling in timeseries_hourly.sql (current datetime for Teradata, midnight for others)
Updated schema.yml to disable incompatible hour-based moving stdevs test for Teradata
expect_row_values_to_have_data_for_every_n_datepart: Created Teradata-specific version with flattened CTEs, proper date arithmetic, and conditional formatting for day vs hour dateparts
get_base_dates (teradata_utils): Added conditional casting based on datepart (day uses DATE intermediate, hour preserves TIMESTAMP)
Various utility macros: md5, datatypes, groupby adaptations
Testing Results:
All dbt-expectations tests now pass on Teradata while maintaining full compatibility with existing adapters (PostgreSQL, Snowflake, BigQuery, Redshift, Spark, etc.)
Reviewers
@GuruM