Tests are the core mechanism for asserting acceptable app behavior within Distributional. In this section, we introduce the necessary tools and explain how they can be used -- either with guidance from Distributional or by users based on unique needs.

At its core, a Test is a combination of a Statistic, derived from a run, which defines some behavior of the run, and an Assertion which enumerates the acceptable values of that statistic (and, thus, the acceptable behavior of a run). The run under consideration in a test is referred to as the Experiment run; often, there will also be a Baseline run used for comparison.

Test Tags can be created and applied to tests to group them together for, among other purposes, signaling the shared purpose of several tests (e.g., tests for text sentiment).

When a group of tests is executed on a chosen Experiment and Baseline Run, the output is a Test Session containing which assertions Pass and Fail.

While we expect everyone to use our automated Production test creation capabilities, we also recognize that many users bring significant expertise to their testing experience. Distributional provides you a suite of test creation tools and patterns that match your needs.

Most tests in dbnl are statistically-motivated to better measure and study fundamentally nondeterministic AI-powered apps. Means, percentiles, Kolmogorov-Smirnov, and other statistical entities are provided to allow you to study the behavior of your app as you would like. We provide templates to guide you through our suggested testing strategies.

The manual test creation process can be configured in any of three locations: in the main web UI test configuration page, through shortcuts to the test drawer scattered throughout the web UI, or through the SDK. This gives you the flexibility to systematically control test generation, or quickly respond to insights for which you would like to test in the future.

The core testing capability is supplemented by the ability to define filters on tests. These filters empower you to test for consistency within subsets of your user base. Filtering also allows you to test for bias in your app and help triage cases of undesired behavior.

Learn more about how to create your own tests here.

Production testing is the core tool in the Distributional toolkit – it provides the ability to continually state whether your app is performing as desired.

To make Production testing easy to start using, we have developed some automated Production test creation capabilities. Using sample data that you upload to our system, we can craft tests which help determine if there has been a shift in behavior for your app.

The Auto-Generate Tests button takes you to the modal below, which allows you to choose the columns on which you want to test for app consistency.

Setting a Baseline through a Run Query

You also have the opportunity to set a Baseline Run Query, which gives you the freedom to define how far into the past you want to look to define “consistency”. After setting the Baseline, your tests will utilize it by default; you can override it at test creation time. Over time, your project will graphically render the health of your app, as shown below.

Recalibrating tests

When you start work with Distributional (dbnl), you should focus on creating and executing to ask and answer the question "Is my AI-powered app behaving as desired?" As you gain more confidence using dbnl, the full pattern of standard dbnl usage looks as follows:

1. Execute Production testing at a regular interval (e.g., nightly) on recent app usages

2. Execute Deployment testing at a regular interval (e.g., weekly) on a fixed dataset

3. Review Production test sessions to start triage of any concerning app behavior

   1. This could be triggered by test failures, DBNL-automated guidance, or manual investigation

4. As concerning app behavior is identified, trigger Deployment tests to help diagnose any app component nonstationarity

   1. If nonstationarity is present, start Development testing on the affected 3rd party components

   2. If the components are stationary, start Development testing on the observed app examples which show unsatisfactory behavior

5. Once suitable app behavior is recorded in Development testing, record a new baseline for future Deployment testing on the fixed dataset

6. Push the new app version into production and update the Deployment + Production testing process