Filters in Tests

Filters can also be used to specify a sub-selection of rows in runs you would like to include in the test computation.

For example, our goal could be to create a test that asserts that, for rows where the loc column is ‘NY’, the absolute difference of means of the correct churn predictions is <= 0.2 between baseline and experiment runs.

We will walk through how this can be accomplished:

1. Navigate to the Project Detail page and click on “Configure Tests”.

1. Click Add Test on the Test Configuration page. Don’t forget to also set a baseline run for automated test configuration.

1. Create the test with the filter specified on the baseline and experiment run.

Filter for the baseline Run:

equal_to({BASELINE}.loc, 'NY')

Filter for the experiment Run:

equal_to({EXPERIMENT}.loc, 'NY')

4. You can now see the new test in the Test Configuration Page. When new data is uploaded, this test will automatically run and compare the new run (as experiment) against the selected baseline run.

When new run data is uploaded, this test will run automatically and use the defined filters to sub-select for the rows that have the loc column equal to ‘NY’.

The full Test Spec in JSON format is shown below.

{
    "name": "abs diff of mean of correct churn preds of NY users is within 0.2",
    "statistic_name": "abs_diff_mean",
    "statistic_params": {},
    "assertions": [
        {
            "name": "less_than_or_equal_to",
            "params": {
                "other": 0.2
            },
        }
    ],
    "statistic_inputs": [
        {
            "select_query_template": {
                "select": "{BASELINE}.pred_correct",
                "filter": "equal_to({BASELINE}.loc, 'NY')"
            }
        },
        {
            "select_query_template": {
                "select": "{EXPERIMENT}.pred_correct",
                "filter": "equal_to({EXPERIMENT}.loc, 'NY')"
            }
        },
    ],
}