fix(dbt): emit expr '1' for COUNT(*) metrics - #432
LukasSchwarzlmueller wants to merge 2 commits into
Conversation
ossie-to-msi turned COUNT(*) into a count metric with expr '*'. MetricFlow renders count metrics as SUM(CASE WHEN <expr> IS NOT NULL THEN 1 ELSE 0 END), so the generated query contained CASE WHEN * IS NOT NULL, which is invalid SQL. Use the constant 1 instead. It is never null, so every row is counted, which is COUNT(*) semantics. COUNT(dataset.*) is handled the same way.
|
|
||
| # COUNT(*) → count of the constant 1 (MetricFlow cannot render a bare * inside a count) | ||
| if isinstance(tree, exp.Count) and _is_star(tree.this): | ||
| return AggregationType.COUNT, "1", None, False |
There was a problem hiding this comment.
I believe this hands "1" back as a column name, and the caller resolves it like one.
_convert_metric passes bare_col to _find_dataset_for_col, which for "1" finds no qualifier, matches no field, and falls through to datasets[0].name.
Two ways that goes wrong:
- Doc with
datasets: [customers, orders]andorder_count = COUNT(*)producesmetric_aggregation_params.semantic_model == "customers". Before this PR, the emitted*made MetricFlow generateCASE WHEN * IS NOT NULLand the query failed to compile. Now it compiles and returns the customers row count under the nameorder_count. In a "ratio", it's worse:(SUM(amount)) / (COUNT(*))over those same two datasets gives numeratorordersand denominatorcustomers, so MetricFlow joins two unrelated models and the average is quietly wrong. - If any dataset has a field whose expression is the constant
1, the field expression scan matches it and everyCOUNT(*)in the document binds to that dataset.
1 is a constant, not a column, so it should not go through the column -> dataset resolver at all.
I suggest resolving the star case from the qualifier only, and refusing when it's ambiguous, which is that the Microsoft converter already does:
# _sql_to_dax.py:208-210
table = resolve_table()
if table is None:
return None, "'COUNT(*)' needs exactly one dataset to count rows of"Trading a compile error for a plausible-looking wrong number is the part I would like to avoid here. Wdyt?
There was a problem hiding this comment.
You're right, thanks for catching this. Fixed in 664e3a1.
One thing i was unsure of: it raises instead of skipping the metric, because the converter has no per-metric skip. I can switch to skip-with-a-warning if you prefer.
| return None | ||
|
|
||
| # COUNT(*) → count of the constant 1 (MetricFlow cannot render a bare * inside a count) | ||
| if isinstance(tree, exp.Count) and _is_star(tree.this): |
There was a problem hiding this comment.
The guard is placed after the DISTINCT branch and ignores extra arguments, so two star forms slip past it.
For instance:
COUNT(DISTINCT *)still emits*. sqlglot parses it, and it hitstree.thisisDistinct,cols == [Star],len(cols) == 1, so it returns(COUNT DISTINCT, '*'). MetricFlow rendersCOUNT(DISTINCT *)and DuckDB fails withSTAR expression is only allowed as the root element of an expression, the exact error this PR set out to remove. The same applies toCOUNT(DISTINCT orders.*).COUNT(t.*, x)is silently accepted. That parses toexp.Count(this=column(t.*), expressions=[x])._is_star(tree.this)is True, so the branch returns a plain row count and drops the second argument without a word.
Both fall out of folding the three exp.Count branches into one block that unwraps DISTINCT and reject multi-arg first. DAX converter is already using it:
if isinstance(tree, exp.Count):
if tree.args.get("expressions"):
return None # COUNT(t.*, x) → raw-expression fallback
argument, distinct = tree.this, False
if isinstance(argument, exp.Distinct):
operands = argument.args.get("expressions") or []
if len(operands) != 1:
return None
argument, distinct = operands[0], True
if _is_star(argument):
return None if distinct else (AggregationType.COUNT, "1", None, False)
...That also drops the trip isinstance(tree, exp.Count) test currently spread in different places.
|
|
||
| def _is_star(node: exp.Expression) -> bool: | ||
| """Return True for ``*`` and for a qualified ``dataset.*``.""" | ||
| return isinstance(node, exp.Star) or (isinstance(node, exp.Column) and isinstance(node.this, exp.Star)) |
There was a problem hiding this comment.
This accepts any multi-part qualified star, and the qualifier is then used verbatim as a semantic model name.
COUNT(db.orders, *) satisfies isinstance(node.this, exp.Star), so we take the star branch and _get_dataset_qualifier joins part[:-1] into "db.orders". The metric comes out with metric_aggregation_params.semantic_model == "db.orders" while the manifest's model is named orders, and MetricFlow can't resolve the metric owning model. Unlike the plain column path there is no bare column fallback to recover from it, because the star branch discards the node entirely.
I suggest to either restrict _is_star to a single part qualifier, or have the star path resolve the qualifier last segment against the actual dataset names and refuse when there is no match.
There was a problem hiding this comment.
Went with your second option. Fixed in 664e3a1.
- Restricting
_is_starwould senddb.orders.*down the column path and give*again. - Your literal
COUNT(db.orders, *)example is caught by the extra-argument check from the other thread.
|
|
||
| # COUNT(*) → count of the constant 1 (MetricFlow cannot render a bare * inside a count) | ||
| if isinstance(tree, exp.Count) and _is_star(tree.this): | ||
| return AggregationType.COUNT, "1", None, False |
There was a problem hiding this comment.
Also, I believe COUNT(dataset.*) doesn't survive a round trip.
With datasets: [customers, orders] and COUNT(orders.*):
- ossie -> msi gives
expr "1",semantic_model "orders",agg count: it's correct and that's whattest_qualified_count_star_uses_dataset_qualifierasserts. - on the trip msi -> ossie,
_qualify_colleaves"1"alone, it matches neither the bare identifier regex norCOUNT_CONVERSION_RE: so the Ossie expression comes back asSUM(1), with theordersbinding gone. - on the trip ossie -> msi again, it re-converts that to
expr "1",semantic_model "customers",agg sum.
The PR description "SUM(1) is valid SQL with the name result" holds only for a single dataset document.
With more than one dataset the round trip silently moves the metric to a different table. _qualify_col needs a case for the constant sentinel (re-emit COUNT(<semantic_model>.*) when agg == count and expr == "1"), and a round trip test over a two dataset doc would pin it.
There was a problem hiding this comment.
Reproduced. Fixed in 664e3a1, with one difference from your suggestion:
_qualify_colcan't checkagg == count, because the count is already aSUMby then.msi_to_ossierecords the row-count metrics before the transform instead.- Filtered counts are still not covered, since there's no
dataset.*form with a filter.
|
@LukasSchwarzlmueller thanks for the updates! I will do a new pass. Much appreciated 😄 |
Summary
ossie-to-msiturnedCOUNT(*)into acountmetric withtype_params.expr: '*'. MetricFlow renders everycountmetric asSUM(CASE WHEN <expr> IS NOT NULL THEN 1 ELSE 0 END), so the generated query containedCASE WHEN * IS NOT NULL, which is not valid SQL (DuckDB:STAR expression is only allowed as the root element of an expression)._extract_agg_infonow returns the constant"1"forCOUNT(*)andCOUNT(dataset.*).1is never null, so every row is counted, which isCOUNT(*)semantics.Checked end to end on DuckDB (
dbt run, thenmf query; the manifest was also patched foragg_time_dimension, the owning semantic model and the time spine, which are separate converter gaps this PR does not touch): before,mf query --metrics order_countfails with the binder error above; after, it returns 4 for four orders, and the per-segment and per-month counts, revenue and average match a plain SQL query on the same tables.Changes:
expression_utils.py: new_is_starhelper and aCOUNT(*)branch ahead of the genericCOUNT(col)branch; docstring updated.test_ossie_to_msi.py: tests forCOUNT(*),count( * )andCOUNT(orders.*), for the qualified form landing on the right semantic model, and forCOUNT(*)inside a ratio metric.README.md: one line under the Ossie → MSI conversion choices.The new tests fail without the fix and pass with it (5 cases fail on unchanged
main). The fullconverters/dbtsuite passes on Python 3.11, 3.12, 3.13 and 3.14 (111 tests).Not changed: on the way back,
msi_to_ossiegivesSUM(1)for this metric (MetricFlow's loader has already turned the count into a sum), notCOUNT(*). That is valid SQL with the same result, whereas today's output was invalid. Existing snapshots are unchanged.Related Issues
None yet.
Checklist
Converters
converters/is updated to reflect spec or ontology changesDocumentation
converters/dbt/README.mdis updated to reflect the user-facing changeTests
pytest/ CI green)Compliance
Specification, Ontology, Validation and Examples: not applicable.