Using Filters in Tests
Filters can be used to specify a sub-selection of rows in Runs you would like to be tested.
For example, you might want to create a test that asserts that the absolute difference of means of the correct churn predictions is <= 0.2 between Baseline and Experiment Runs, only for rows where the loc column is NY.
Apply a Filter to a Test
Navigate to the Create Test page and create the test with the filter specified on the baseline and experiment run.
Filter for the baseline Run:
equal_to({BASELINE}.loc,'NY')Filter for the experiment Run:
equal_to({EXPERIMENT}.loc,'NY')You can create a test with filters in the SDK via the create_test function:
import dbnl
dbnl.login()
proj = dbnl.get_or_create_project(name="My Project")
dbnl.experimental.create_test(
test_spec_dict={
"project_id": proj.id,
"name": "abs diff of mean of correct churn preds of NY users is within 0.2",
"statistic_name": "abs_diff_mean",
"statistic_params": {},
"assertion": {
"name": "less_than_or_equal_to",
"params": {
"other": 0.2
},
},
"statistic_inputs": [
{
"select_query_template": {
"select": "{BASELINE}.pred_correct",
"filter": "equal_to({BASELINE}.loc, 'NY')"
}
},
{
"select_query_template": {
"select": "{EXPERIMENT}.pred_correct",
"filter": "equal_to({EXPERIMENT}.loc, 'NY')"
}
},
],
}
)Once you've used one of the methods above, you can now see the new test in the Test Configuration tab of your Project.
When a Test Session is created, this test will use the defined filters to sub-select for the rows that have the loc column equal to NY.
Was this helpful?