Using Filters in Tests

Filters can be used to specify a sub-selection of rows in Runs you would like to be tested.

For example, you might want to create a test that asserts that the absolute difference of means of the correct churn predictions is <= 0.2 between Baseline and Experiment Runs, only for rows where the loc column is NY.

Apply a Filter to a Test

UI

Navigate to the Create Test page and create the test with the filter specified on the baseline and experiment run.

Filter for the baseline Run:

equal_to({BASELINE}.loc, 'NY')

Filter for the experiment Run:

equal_to({EXPERIMENT}.loc, 'NY')

SDK

You can create a test with filters in the SDK via the create_test function:

import dbnl
dbnl.login()

proj = dbnl.get_or_create_project(name="My Project")

dbnl.experimental.create_test(
    test_spec_dict={
        "project_id": proj.id,
        "name": "abs diff of mean of correct churn preds of NY users is within 0.2",
        "statistic_name": "abs_diff_mean",
        "statistic_params": {},
        "assertion": {
            "name": "less_than_or_equal_to",
            "params": {
                "other": 0.2
            },
        },
        "statistic_inputs": [
            {
                "select_query_template": {
                    "select": "{BASELINE}.pred_correct",
                    "filter": "equal_to({BASELINE}.loc, 'NY')"
                }
            },
            {
                "select_query_template": {
                    "select": "{EXPERIMENT}.pred_correct",
                    "filter": "equal_to({EXPERIMENT}.loc, 'NY')"
                }
            },
        ],
    }
)

Once you've used one of the methods above, you can now see the new test in the Test Configuration tab of your Project.

When a Test Session is created, this test will use the defined filters to sub-select for the rows that have the loc column equal to NY.