Skip to content

scripts/notify-on-record-change: only support zst #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
joverlee521 merged 1 commit into from
Jun 9, 2026
Merged

scripts/notify-on-record-change: only support zst #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ Scripts for supporting workflow automation that don’t really belong in any of
- [notify-on-job-fail](scripts/notify-on-job-fail) - Send Slack message with details about failed workflow job on GitHub Actions and/or AWS Batch
- [notify-on-job-start](scripts/notify-on-job-start) - Send Slack message with details about workflow job on GitHub Actions and/or AWS Batch
- [notify-on-record-change](scripts/notify-on-record-change) - Send Slack message with details about line count changes for a file compared to an S3 object's metadata `recordcount`.
If the S3 object's metadata does not have `recordcount`, then will attempt to download S3 object to count lines locally, which only supports `xz` compressed S3 objects.
If the S3 object's metadata does not have `recordcount`, then will attempt to download S3 object to count lines locally, which only supports `zst` compressed S3 objects.
- [notify-slack](scripts/notify-slack) - Send message or file to Slack
- [s3-object-exists](scripts/s3-object-exists) - Used to prevent 404 errors during S3 file comparisons in the notify-* scripts
- [trigger](scripts/trigger) - Triggers downstream GitHub Actions via the GitHub API using repository_dispatch events.
Expand Down
2 changes: 1 addition & 1 deletion scripts/notify-on-record-change
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ dst_record_count="$(aws s3api head-object --bucket "$bucket" --key "$key" --quer
if [[ -z "$dst_record_count" ]]; then
# This object doesn't have the record count stored as metadata
# We have to download it and count the lines locally
dst_record_count="$(wc -l < <(aws s3 cp --no-progress "$dst" - | xz -T0 -dcfq))"
dst_record_count="$(wc -l < <(aws s3 cp --no-progress "$dst" - | zstd -T0 -dcq))"
fi

added_records="$(( src_record_count - dst_record_count ))"
Expand Down
Loading