There are several ways for dbnl users to create tests, each of which may be useful in different circumstances and for different users within the same organization.
The base location at which a test can be created is the Test Configuration page, which is accessible from the Project Detail page.
From here, you can click Add Test to open the test creation page, which will enable you to define your test through the dropdown menu on the left side of the window.
The graphs available on the right side of the window can help guide test development as you choose the statistics you want to study and the thresholds which define acceptable behavior.
Tests can be programmatically created using the python SDK. Users must provide a JSON dictionary that adheres to the dbnl Test Spec and instantiates the key elements of a Test. The Testing Strategies section has sample content around creating tests using JSON. Additionally, the test creation page will soon also have the ability to convert a test from dropdowns to JSON, so that many tests of similar structure can be created in JSON after the first one is designed in the web UI.
You can also create tests by clicking on the + icon button which appears in several places :
Cells in the summary statistics tables (found on the Run Detail page, Compare pages)
Mini charts (clicking on the title and the column feature button, found on Run Detail page)
Above statistics found on Compare pages
At each of these locations, a test creation drawer will open on the right side of the page with several of the fields pre-populated based on the context of the button.
Test templates are macros for basic test patterns recommended by Distributional. It allows the user to quickly create tests from a builder in the UI. Distributional provides five classes of Test Templates.
Single Run: These are parametric statistics of a column.
Similarity of Statistics: These test if the absolute difference of a statistic of a column between two runs is less than a threshold.
Similarity of Distributions: These test if the column from two different runs are similarly distributed is using a nonparametric statistic.
Similarity of Results: These are tests on the row-wise absolute difference of result
Difference of Statistics: These tests the signed difference of a statistic of a column between two runs
From the Test Configuration page, click the Test Template dropdown under My Tests.
Select from one of the five options and click ADD TEST. A Test Creation drawer will appear and the user can edit the statistic, column, and assertion that they desire. Note that each Test Template has a limited set of statistics that it supports.
