Skip to content

feat: add param _columns_to_include and _columns_to_exclude in the reverse_etl task#52

Merged
claudiazi merged 4 commits intomasterfrom
feature/new-param-for-reverse-etl
Apr 23, 2025
Merged

feat: add param _columns_to_include and _columns_to_exclude in the reverse_etl task#52
claudiazi merged 4 commits intomasterfrom
feature/new-param-for-reverse-etl

Conversation

@claudiazi
Copy link
Copy Markdown

@claudiazi claudiazi commented Apr 22, 2025

Add _input_table_columns_to_include and _input_table_columns_to_exclude to the reverse_etl task param, so that we can choose what fields to upload to the dynamo/glue

@claudiazi claudiazi requested a review from a team as a code owner April 22, 2025 13:34
parent_fields=["task_parameters"],
validator=str,
required=False,
comment='Optional comma-separated list of columns to include in the job. Example: \'column1,column2,column3\', if not provided, all columns will be included',
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you be more specific if this is the list of column names before the column mapping happening or after?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done! :)

parent_fields=["task_parameters"],
validator=str,
required=False,
comment='Optional comma-separated list of columns to exclude from the job. Example: \'column1,column2,column3\', if not provided, all columns will be included',
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same here

@claudiazi claudiazi requested a review from siklosid April 23, 2025 08:43
@claudiazi claudiazi merged commit 6b1d3b5 into master Apr 23, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants