Releases: colpal/dataeng-container-tools
v0.6.4
v0.6.3: JSON file read and Secret Parsing Stability Improvements
What's Changed
- Feature/print service by @haitiangk92 in #153
- Secret Stability; JSON readability by @mourtallah in #160
Full Changelog: v0.6.2...v0.6.3
v0.6.2
Fixing bq module error handling
v0.6.1
This release contains the following updates:
- Addition of the
BQmodule to handle communication with Google BigQuery. There is no longer a connection to bigquery from the gcs module. - New implementation of secret handling. Secrets are now passed in as dictionaries in the format
{ secret_identifier : path_to_secret }and are accessed as object attributes. Ex.
sf_secret = secret_locations.SF - New documentation regarding the new changes and updates
- multiple bug fixes
v0.6.0
- Updates encoding logic to force casting censored words as strings to avoid encoding issues.
- Updates parsing logic for gcs uri; splits and and then concatenates protocol prefix and breadcrumbs to enable references files in buckets that don't have intermediate paths.
- Updates command line arguments logic to separating clashing command line flags.
v0.5.9 : Update get methods for uri's
Fix for input and output uri methods
This release revises the get_input_uris() and get_output_uris() to handle the the gcs prefix separately from the rest of the uri. This should ensure that the prefix is not alters when the uri is trimmed of invalid slash combination that may arise when referencing a pathless files in a gcs bucket. - (PR #105)
Fix for command_line_arguments class methods
get_input_uris() and get_output_uris() returning uri strings with the gcs prefix missing a forward slash: (i.e gs:/ instead of gs://). New release ensures that full corrects gcs uri strings are returned when returning cla.get_input_uris() or cla.get_output_uris().
UPDATE: Adding revised release per @mourtallah PR #105 fix
Pandas Kwargs in build
Adding the pandas kwargs addition to the build version of the codebase.
Pandas Kwargs
Downloading and Uploading to and from pandas data frames can now access every possible argument for pandas functions. Arguments will be passed in as a dictionary where the key is the argument and the value is the argument value. "{argument : value}"
Changes made in the "cla" module:
init()
get_pandas_kwargs() new
Changes made in the "gcs" module:
__wildcard_download()
download_file_to_object()
download_file_to_objects()
upload_file_from_object()
upload_file_from_objects()