Skip to content

Releases: colpal/dataeng-container-tools

v0.6.4

16 Jul 14:52
289ad0f

Choose a tag to compare

What's Changed

Full Changelog: v0.6.3...v0.6.4

v0.6.3: JSON file read and Secret Parsing Stability Improvements

14 Feb 20:02
48f2c5d

Choose a tag to compare

What's Changed

Full Changelog: v0.6.2...v0.6.3

v0.6.2

19 May 21:07
0e5052f

Choose a tag to compare

Fixing bq module error handling

v0.6.1

13 May 21:14
82c3d6b

Choose a tag to compare

This release contains the following updates:

  • Addition of the BQ module to handle communication with Google BigQuery. There is no longer a connection to bigquery from the gcs module.
  • New implementation of secret handling. Secrets are now passed in as dictionaries in the format { secret_identifier : path_to_secret } and are accessed as object attributes. Ex.
sf_secret = secret_locations.SF 
  • New documentation regarding the new changes and updates
  • multiple bug fixes

v0.6.0

10 May 20:18
4672fe3

Choose a tag to compare

  • Updates encoding logic to force casting censored words as strings to avoid encoding issues.
  • Updates parsing logic for gcs uri; splits and and then concatenates protocol prefix and breadcrumbs to enable references files in buckets that don't have intermediate paths.
  • Updates command line arguments logic to separating clashing command line flags.

v0.5.9 : Update get methods for uri's

06 Jan 16:42
a6c1505

Choose a tag to compare

What's Changed

Full Changelog: 0.5.8...v0.5.9

Fix for input and output uri methods

05 Nov 16:02
97def9b

Choose a tag to compare

This release revises the get_input_uris() and get_output_uris() to handle the the gcs prefix separately from the rest of the uri. This should ensure that the prefix is not alters when the uri is trimmed of invalid slash combination that may arise when referencing a pathless files in a gcs bucket. - (PR #105)

Fix for command_line_arguments class methods

18 Oct 18:03
4e3a1e8

Choose a tag to compare

get_input_uris() and get_output_uris() returning uri strings with the gcs prefix missing a forward slash: (i.e gs:/ instead of gs://). New release ensures that full corrects gcs uri strings are returned when returning cla.get_input_uris() or cla.get_output_uris().

UPDATE: Adding revised release per @mourtallah PR #105 fix

Pandas Kwargs in build

07 Sep 14:51
bf42d00

Choose a tag to compare

Adding the pandas kwargs addition to the build version of the codebase.

Pandas Kwargs

03 Sep 00:58
14385bc

Choose a tag to compare

Downloading and Uploading to and from pandas data frames can now access every possible argument for pandas functions. Arguments will be passed in as a dictionary where the key is the argument and the value is the argument value. "{argument : value}"

Changes made in the "cla" module:

init()
get_pandas_kwargs() new

Changes made in the "gcs" module:

__wildcard_download()
download_file_to_object()
download_file_to_objects()
upload_file_from_object()
upload_file_from_objects()