Skip to content

assessment_quality.ipynb: NameError and a wrong-variable filter on the score-your-own-model path #389

Description

@nikhilsi

Found while running notebooks/pipeline/assessment_quality.ipynb at commit
7952236 with prediction_field = "prediction" (the score-your-own-model path,
rather than the default assessor-value path).

Two problems in the data-loading cell (the cell containing the
read_sales_univ fallback chain):

  1. NameError. The if prediction_field == "prediction": branch references
    sales_filters and univ_filters:
if len(sales_filters) > 0:
    df_sales = select_filter(df_sales, sales_filters)
...
if len(univ_filters) > 0:
    df_sales = select_filter(df_univ, univ_filters)

Neither variable is defined anywhere in the notebook (the configuration cell
defines other knobs but not these), so evaluating your own model's predictions
raises NameError before any study runs. The default path never touches the
branch, which is presumably why it has gone unnoticed.

  1. Wrong variable. The second snippet above assigns the filtered universe to
    df_sales and then sets sales_univ_pair.universe = df_univ, the unfiltered
    frame, so even with the variables defined, the universe filter result is
    discarded (and the sales frame is clobbered).

Suggested fix: define sales_filters = [] and univ_filters = [] in the
configuration cell, and change the second assignment to
df_univ = select_filter(df_univ, univ_filters).

(Separate from the two library-side crashes in this notebook already reported
in issue #359 and fixed by PR #357; this one is in the notebook itself.)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions