Skip to content

fix(config): let an explicitly falsy value on the right-hand side win in merge_config - #4005

Open
Rodrigo-Palma wants to merge 1 commit into
apache:mainfrom
Rodrigo-Palma:fix/merge-config-falsy-override
Open

Rodrigo-Palma wants to merge 1 commit into
apache:mainfrom
Rodrigo-Palma:fix/merge-config-falsy-override

Conversation

@Rodrigo-Palma

Copy link
Copy Markdown
Contributor

Rationale

merge_config() says it takes "the non-null value, with precedence on rhs", but the implementation is:

new_config[rhs_key] = rhs_value or lhs_value

or falls back to the left-hand side for any falsy value, not just None. The right-hand side is the caller's own properties, so explicitly turning an option off is silently discarded when the configuration file sets it.

# ~/.pyiceberg.yaml
# catalog:
#   prod:
#     type: rest
#     uri: https://example.com
#     s3.path-style-access: "true"

from pyiceberg.catalog import load_catalog

catalog = load_catalog("prod", **{"s3.path-style-access": False})
# s3.path-style-access is still "true"

The same happens for "" and 0:

rhs value documented result actual result
False False "true" (from the file)
"" "" "true"
0 0 "true"
None value from the file value from the file (correct)

This contradicts what #45 set out to do. That PR swapped the merge order precisely so that "the passed in argument takes precedence" over configuration coming from the environment, and a caller passing False is passing an argument.

There is also a precedent inside this codebase: property_as_bool() in pyiceberg/utils/properties.py already distinguishes "not set" from "falsy", with if (value := properties.get(property_name)) not in (None, ""). After this change the two agree instead of disagreeing.

Change

new_config[rhs_key] = rhs_value if rhs_value is not None else lhs_value

None still means "not set" and still lets the left-hand value survive, which is what the _from_environment_variables path relies on.

Tests added to tests/utils/test_config.py:

  • test_merge_config_rhs_wins_for_falsy_values, parametrized over False, "" and 0.
  • test_merge_config_lhs_wins_when_rhs_is_none, pinning the None behavior so a future change cannot quietly turn None into an override.

Verification

  • make lint passes (ruff, ruff format, mypy, pydocstyle, codespell).
  • pytest tests/utils tests/catalog tests/cli passes: 1434 tests.
  • pytest tests/ -m "not integration and not s3 and not adls and not gcs" --ignore=tests/io --ignore=tests/benchmark: 3900 passed, 9 failed. The 9 failures are all in tests/avro/test_decoder.py for CythonBinaryDecoder and are pre-existing in my environment: the same 9 fail on the unmodified branch at the same commit (9 failed, 59 passed both with and without this patch), since the Cython extension is not built locally.

merge_config() documents 'take the non-null value, with precedence on
rhs', but it used `rhs_value or lhs_value`, which falls back to the
left-hand side for any falsy value, not just None.

load_catalog(name, **properties) merges the configuration file into the
caller's properties, so turning an option off explicitly was silently
ignored when the file set it:

    # .pyiceberg.yaml has s3.path-style-access: "true"
    load_catalog("prod", **{"s3.path-style-access": False})
    # -> 's3.path-style-access' stays 'true'

That contradicts the intent of apache#45, which made the passed-in argument
take precedence over the environment. property_as_bool() in the same
codebase already distinguishes 'not set' from falsy by testing against
(None, ""), so the two now agree.

None on the right-hand side still means 'not set' and keeps the
left-hand value.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant