# Creating Tests

As you become more familiar with the behavior of your application, you may want to build on the default App Similarity Index test with tests that you define yourself. Let's walk through that process.

## Designing a Test

The first step in coming up with a test is determining what behavior you're interested in. As described in [the section on Runs](https://docs.dbnl.com/v0.24.x/using-distributional/runs), each Run of your application reports its behavior via its results, which are organized into columns (and scalars). Once you've identified the column or scalar you'd like to test on, then you need to determine what [statistic](https://docs.dbnl.com/v0.24.x/using-distributional/tests/available-statistics-and-assertions#statistics) you'd like to apply to it and the [assertion](https://docs.dbnl.com/v0.24.x/using-distributional/tests/available-statistics-and-assertions#assertions) you'd like to make on that statistic.

This might seem like a lot, but DBNL has your back! While you can define tests [manually](#creating-tests-manually), DBNL has several ways of helping you identify what columns you might be interested in and letting you quickly define tests on them.

{% hint style="info" %}
When creating a test, you can specify tags to apply to it. You can use these tags to filter which tests you want to include or exclude later when [creating a Test Session](https://docs.dbnl.com/v0.24.x/using-distributional/running-tests#create-a-test-session). Some of the test creation shortcuts on the UI do not currently allow specifying tests, but you can edit the test and add tags after the fact.
{% endhint %}

## Context-Driven Test Creation

As you browse the DBNL UI, you will see "+" icons or "+ Add Test" buttons appear. These provide context-aware shortcuts for easily creating relevant tests.&#x20;

At each of these locations, a test creation drawer will open on the right side of the page with several of the fields pre-populated based on the context of the button, alongside a history of the statistic, if relevant. Here are some of the best places to look for DBNL-assisted test creation:

### Similarity Insights

When you're looking at a Test Session, DBNL will provide insights about which columns or metrics have have demonstrated the most drift. These are great candidates to define tests on if you want to be specifically alerted about their behavior. You can click the "Add Test" button to create a test on the [Similarity Index](https://docs.dbnl.com/v0.24.x/using-distributional/reviewing-tests#similarity-indexes) of the relevant column. The Similarity Index history graph can help guide you on choosing a threshold.

<figure><img src="https://content.gitbook.com/content/xVrDMviYOLuFCuRxVlSy/blobs/UFYiQS07v1UiujswDVeV/image.png" alt=""><figcaption><p>Test Session Summary Page</p></figcaption></figure>

### Column or Metric Details

When scrolling through the details of a column or metric from a Test Session (found by drilling into that column or metric from the Similarity Insights or Similarity Report), there are several "Add Test" buttons provided to allow you to quickly create a test on a relevant statistic or Similarity Index. The respective history graphs can help guide you on choosing a threshold.

<figure><img src="https://content.gitbook.com/content/xVrDMviYOLuFCuRxVlSy/blobs/VLJyN6zQyNACOvjAMzXA/image.png" alt=""><figcaption><p>Column or Metric Similarity Page</p></figcaption></figure>

### Summary Statistics Table

When viewing a Run, each entry in the summary statistics table can be used to seed creation of a test for that chosen statistic.

<figure><img src="https://content.gitbook.com/content/xVrDMviYOLuFCuRxVlSy/blobs/5u0BCCGoeSSchV8QTis6/image.png" alt=""><figcaption><p>Run Details Page</p></figcaption></figure>

### And More!

These shortcuts appear in several other places in the UI as well when you are inspecting your Runs and Test Sessions; keep an eye out for the "+"!

## Templated Tests

Test templates are macros for basic test patterns recommended by Distributional. It allows the user to quickly create tests from a builder in the UI. Distributional provides five classes of test templates:

* [Single Run](https://docs.dbnl.com/v0.24.x/to-be-deleted/testing-strategies/test-that-a-given-distribution-has-certain-properties): These are parametric statistics of a column.
* [Similarity of Statistics](https://docs.dbnl.com/v0.24.x/to-be-deleted/testing-strategies/test-that-distributions-have-the-same-statistics): These test if the absolute difference of a statistic of a column between two runs is less than a threshold.
* [Similarity of Distributions](https://docs.dbnl.com/v0.24.x/to-be-deleted/testing-strategies/test-that-columns-are-similarly-distributed): These test if the column from two different runs are similarly distributed is using a nonparametric statistic.
* [Similarity of Results](https://docs.dbnl.com/v0.24.x/to-be-deleted/testing-strategies/test-that-specific-results-have-matching-behavior): These are tests on the row-wise absolute difference of result
* [Difference of Statistics](https://docs.dbnl.com/v0.24.x/to-be-deleted/testing-strategies/test-that-distributions-are-not-the-same): These tests the signed difference of a statistic of a column between two runs

From the Test Configuration tab on your Project, click the dropdown next to "Add Test".

<figure><img src="https://content.gitbook.com/content/xVrDMviYOLuFCuRxVlSy/blobs/dJQ7hW7XK9gWtZqSTHSa/image.png" alt=""><figcaption><p>Test Configuration Tab</p></figcaption></figure>

Select from one of the five options. A Test Creation drawer will appear and the user can edit the statistic, column, and assertion that they desire. Note that each Test Template has a limited set of statistics that it supports.

<figure><img src="https://content.gitbook.com/content/xVrDMviYOLuFCuRxVlSy/blobs/ofl9RvWsFbC40YexdIZz/Screenshot%202024-11-15%20at%2011.18.47%E2%80%AFAM.png" alt="" width="375"><figcaption><p>Test Creation Drawer</p></figcaption></figure>

## Creating Tests Manually (Advanced)

{% hint style="info" %}
When creating a test manually, you can also specify filters to apply the test only to specific rows within your Runs. Check out [Using Filters in Tests](https://docs.dbnl.com/v0.24.x/using-distributional/tests/creating-tests/using-filters-in-tests) for more information.
{% endhint %}

If you have a good idea of what you want to test or just want to explore, you can create tests manually from either the UI or via the Python SDK.

Let's say you are building an Q\&A chatbot, and you have a column for the length of your bot's responses, `word_count`. Perhaps you want to ensure that your bot never outputs more than 100 words; in that case, you'd choose:

* The statistic `max`,
* The assertion `less than or equal to` ,
* and the threshold `100`.&#x20;

But what if you're not opinionated about the specific length? You just want to ensure that your app is behaving consistently as it runs and doesn't suddenly start being unusually wordy or terse. DBNL makes it easy to test that as well; you might go with:

* The statistic `absolute difference of mean`,
* The assertion `less than`,
* and the threshold `20`.

Now you're ready to go and create that test, either via the UI or the SDK:

{% tabs %}
{% tab title="UI" %}
From your Project, click the "Test Configuration" tab.

At the top, you can click "Add Test" to open the test creation page, which will enable you to define your test through the dropdown menu on the left side of the window.

<figure><img src="https://content.gitbook.com/content/xVrDMviYOLuFCuRxVlSy/blobs/qXH7q0qz1MaSVAvHG1MG/image.png" alt=""><figcaption><p>Create Test Page</p></figcaption></figure>

On the left side you can configure your test by choosing a statistic and assertion. Note that you can use our builder or build a test spec with raw JSON (you can see some example test spec JSONs [here](https://docs.dbnl.com/v0.24.x/using-distributional/tests/..#whats-in-a-test)). On the right, you can browse the data of recent Runs to help you figure out what statistics and thresholds are appropriate to define acceptable behavior.
{% endtab %}

{% tab title="SDK" %}
Tests can be [programmatically created](https://docs.dbnl.com/v0.24.x/reference/python-sdk/dbnl.experimental#create_test) using the python SDK. Users must provide a JSON dictionary that adheres to the dbnl Test Spec, which is described in the previous link and has an example provided below.

```python
import dbnl
dbnl.login()

proj = dbnl.get_or_create_project(name="My Project")

dbnl.experimental.create_test(
    test_spec_dict={
        "project_id": proj.id,
        "name": "Word count difference",
        "description": "Test the absolute difference of mean on word_count",
        "statistic_name": "abs_diff_mean",
        "statistic_params": {},
        "assertion": {
            "name": "less_than",
            "params": {
                "other": 20.0,
            },
        },
        "statistic_inputs": [
            {
                "select_query_template": {
                    "select": "{EXPERIMENT}.word_count"
                }
            },
            {
                "select_query_template": {
                    "select": "{BASELINE}.word_count"
                }
            },
        ],
    }
)
```

{% endtab %}
{% endtabs %}

You can see a full list with descriptions of available statistics and assertions [here](https://docs.dbnl.com/v0.24.x/using-distributional/tests/creating-tests/available-statistics-and-assertions).
