This guide walks you through using the Distributional SDK to create your first project, submit two runs and create a test session to compare the behavior of your AI application over time.

pip install dbnl

export DBNL_API_URL="YOUR_DBNL_API_URL"
export DBNL_API_TOKEN="YOUR_PERSONAL_ACCESS_TOKEN"

import random
from datetime import datetime

import dbnl
import pandas as pd

# Login to dbnl.
dbnl.login()

# Create a new project
now = datetime.now().isoformat()
project = dbnl.get_or_create_project(name=f"quickstart-{now}")

# Submit a first run (baseline)
run1 = dbnl.report_run_with_results(
    project=project,
    display_name="run1",
    column_data=pd.DataFrame([
        {
            "question": f"Is {i} an even or odd number?",
            "answer": random.choice(["even", "odd"]),
        }
        for i in range(20)
    ]).astype({"question": "string", "answer": "category"}),
)

# Submit a second run (experiment)
run2 = dbnl.report_run_with_results(
    project=project,
    display_name="run2",
    column_data=pd.DataFrame([
        {
            "question": f"Is {i} an even or odd number?",
            "answer": random.choice(["even", "odd"]),
        }
        for i in range(20)
    ]).astype({"question": "string", "answer": "category"}),
)

# Create a test session.
dbnl.set_run_as_baseline(run=run1)
dbnl.create_test_session(experiment_run=run2)

Next Steps

Distributional is an adaptive testing platform purpose-built for AI applications. It enables you to test AI application data at scale to define, understand, and improve your definition of AI behavior to ensure consistency and stability over time.

Adaptive Testing Workflow

Define Desired Behavior Automatically create a behavioral fingerprint from the app’s runtime logs and any existing development metrics, and generate associated tests to detect changes in that behavior over time.

Integrating Distributional with Other Tools

Adaptive testing requires a very different approach than traditional software testing. The goal of adaptive testing is to enable teams to define a steady baseline state for any AI application, and through testing, confirm that it maintains steady state, and where it deviates, figure out what needs to evolve or be fixed to reach steady state once again. This process needs to be discoverable, logged, organized, consistent, integrated and scalable.

Testing AI applications needs to be fundamentally reimagined to include statistical tests on distributions of quantities to detect meaningful shifts that warrant deeper investigation.

• Distributions > Summary Statistics: Instead of only looking at summary statistics (e.g. mean, median, P90), we need to analyze distributions of metrics, over time. This accounts for the inherent variability in AI systems while maintaining statistical rigor.

Why is this useful? Imagine you have an application that contains an LLM and you want to make sure that the latency of the LLM remains low and consistent across different types of queries. With a traditional monitoring tool, you might be able to easily monitor P90 and P50 values for latency. P50 represents the latency value below which 50% of the requests fall and will give you a sense of the typical (median) response time that users can expect from the system. However, the P50 value for a normal distribution and bimodal distribution can be the same value, even though the shape of the distribution is meaningfully different. This can hide significant (usage-based or system-based) changes in the application that affect the distribution of the latency scores. If you don’t examine the distribution, these changes go unseen.

Consider a scenario where the distribution of LLM latency started with a normal distribution, but due to changes in a third-party data API that your app uses to inform the response of the LLM, the latency distribution becomes bimodal, though with the same median (and P90 values) as before. What could cause this? Here’s a practical example of how something like this could happen. The engineering team of the data API organization made an optimization to their API which allows them to return faster responses for a specific subset of high value queries, and routes the remainder of the API calls to a different server which has a slower response rate.

The effect that this has on your application is that now half of your users are experiencing an improvement in latency, and now a large number of users are experiencing “too much” latency and there’s an inconsistent performance experience among users. Solutions to this particular example include modifying the prompt, switching the data provider to a different source, format the information that you send to the API differently or a number of other engineering solutions. If you are not concerned about the shift and can accept the new steady state of the application, you can also choose to not make changes and declare a new acceptable baseline for the latency P50 value.

Installing Distributional

1. Latest Stable Release

To install the latest stable release of the dbnl package:

2. Specific Release

To install a specific version (e.g., version 0.22.0):

3. Installing with the eval Extra

The dbnl.eval extra includes additional features and requires an external spaCy model.

3.1. Install the Required spaCy Model

To install the required en_core_web_sm pretrained English-language NLP model model for spaCy:

3.2. Install dbnl with the eval Extra

To install dbnl with evaluation extras:

If you need a specific version with evaluation extras (e.g., version 0.22.0):

4. Accessing the Distributional UI and API token

We recommend setting your API token as an environment variable, see below.

5. Environment Variables

DBNL has three reserved environment variables that it reads in before execution.

Set up for various deployment types

Version Matching Requirements

To check your SDK version:

To check your API server version:

• Logging into the web app

• Clicking the hamburger menu (☰) on the top-left corner

• Viewing the version number listed in the footer

What are Metrics?

Metrics are measurable properties that help quantify specific characteristics of your data. Metrics can be user-defined, by providing a numeric column computed from your source data alongside your application data.

Alternatively, the Distributional SDK offers a comprehensive set of metrics for evaluating various aspects of text and LLM outputs. Using Distributional's methods for computing metrics will enable better data-exploration and application stability monitoring capabilities.

Using Metrics

The SDK provides convenient functions for computing metrics from your data and reporting the results to Distributional:

import dbnl
import dbnl.eval
import pandas as pd

# login to dbnl
dbnl.login()
project = dbnl.create_project(name="Metrics Project")

df = pd.DataFrame(
    {
        "id": [1, 2, 3],
        "question": [
            "What is the meaning of life?",
            "What is the airspeed velocity of an unladen swallow?",
            "What is the capital of Assyria?",
        ],
        "answer": [
            "To be happy and fulfilled.",
            "It's a question of aerodynamics.",
            "Nineveh was the capital of Assyria.",
        ],
        "expected_answer": [
            "42",
            "It's a question of aerodynamics.",
            "Nineveh was the capital of Assyria.",
        ],
    }
)

# Create individual metrics
metrics = [
    dbnl.eval.metrics.token_count("question"),
    dbnl.eval.metrics.word_count("question"),
    dbnl.eval.metrics.rouge1("answer", "expected_answer"),
]

# Compute metrics and report results to Distributional
run = dbnl.eval.report_run_with_results(
    project=project, column_data=df, metrics=metrics
)

Convenience Functions

The SDK includes helper functions for creating common groups of related metrics based on consistent inputs.

import dbnl
import dbnl.eval
import pandas as pd

# login to dbnl
dbnl.login()
project = dbnl.create_project(name="Metrics Project")

df = pd.DataFrame(
    {
        "id": [1, 2, 3],
        "question": [
            "What is the meaning of life?",
            "What is the airspeed velocity of an unladen swallow?",
            "What is the capital of Assyria?",
        ],
        "answer": [
            "To be happy and fulfilled.",
            "It's a question of aerodynamics.",
            "Nineveh was the capital of Assyria.",
        ],
        "expected_answer": [
            "42",
            "It's a question of aerodynamics.",
            "Nineveh was the capital of Assyria.",
        ],
    }
)

# Get standard text evaluation metrics
text_eval_metrics = dbnl.eval.metrics.text_metrics(
    prediction="answer", target="expected_answer"
)

# Get comprehensive QA evaluation metrics
qa_metrics = dbnl.eval.metrics.question_and_answer_metrics(
    prediction="answer",
    target="expected_answer",
    input="question",
)

# Compute metrics and report results to Distributional
run = dbnl.eval.report_run_with_results(
    project=project, column_data=df, metrics=(text_eval_metrics + qa_metrics)
)

The Run is the core object for recording an application's behavior; when you upload a dataset from usage of your app, it takes the shape of a Run. As such, you can think of a Run as the results from a batch of uses or from a standard example set of uses of your application. When exploring or testing your app's behavior, you will look at the Run in dbnl either in isolation or in comparison to another Run.

What's in a Run?

A Run contains the following:

• a table of results where each row holds the data related to a single app usage (e.g. a single input and output along related metadata),

• a set of Run-level values, also known as scalars,

• structural information about the components of the app and how they relate, and

• user-defined metadata for remembering the context of a run.

Your Project will contain many Runs. As you report Runs into your Project, dbnl will build a picture of how your application is behaving, and you will utilize tests to verify that its behavior is appropriate and consistent. Some common usage patterns would be reporting a Run daily for regular checkpoints or reporting a Run each time you deploy a change to your application.

The structure of a Run is defined by its schema. This informs dbnl about what information will be stored in each result (the columns), what Run-level data will be reported (the scalars), and how the application is organized (the components).

Baseline and Experiment Runs

Similarity Index is a single number between 0 and 100 that quantifies how much your application’s behavior has changed between two runs – a Baseline and an Experiment run. It is Distributional’s core signal for measuring application drift, automatically calculated and available in every Test Session.

A lower score indicates a greater behavioral change in your AI application. Each Similarity Index has accompanying Key Insights with a description to help users understand and act on the behavioral drift that Distributional has detected.

Where You’ll See It in the UI

• Test Session Summary Page — App-level Similarity Index, results of failed tests, and Key Insights

• Similarity Report Tab — Breakdown of Similarity Indexes by column and metric

• Column Details View — Histograms and statistical comparison for specific metrics

• Tests View — History of Similarity Index-based test pass/fail over time

Why It Matters

When model behavior changes, you need:

1. A clear signal that drift occurred

2. An explanation of what changed

3. A workflow to debug, test, and act

Similarity Index + Key Insights provides all three.

Example:

• An app’s Similarity Index drops from 93 → 46

• Key Insight: “Answer similarity has decreased sharply”

• Metric: levenshtein__generated_answer__expected_answer

Result: Investigate histograms, set test thresholds, adjust model

Hierarchical Structure

Similarity Index operates at three levels:

• Application Level — Aggregates all lower-level scores

• Column Level — Individual column-level drift

• Metric Level — Fine-grained metric change (e.g., readability, latency, BLEU score)

Each level rolls up into the one above it. You can sort by Similarity Index to find the most impacted parts of your app.

Test Sessions and Thresholds

By default, a new DBNL project comes with an Application-level Similarity Index test:

• Threshold: ≥ 80

• Failure: Indicates meaningful application behavior change

In the UI:

• Passed tests are shown in green

• Failed tests are shown in red with diagnostic details

All past test runs can be reviewed in the test history.

Key Insights

Key Insights are human-readable interpretations of Similarity Index changes. They answer:

“What changed, and does it matter?”

Each Key Insight includes:

• A plain-language summary: “Distribution substantially drifted to the right”

• The associated column/metric

• The Similarity Index for that metric

• Option to add a test on the spot

Example:

Distribution substantially drifted to the right.

→ Metric: levenshtein__generated_answer__expected_answer

→ Similarity Index: 46

→ Add Test

Insights are prioritized and ordered by impact, helping you triage quickly.

Deep Dive: Column Similarity Details

Clicking into a Key Insight opens a detailed view:

• Histogram overlays for experiment vs. baseline

• Summary statistics (mean, median, percentile, std dev)

• Absolute difference of statistics between runs

• Links to add similarity or statistical tests on specific metrics

This helps pinpoint whether drift was due to longer answers, slower responses, or changes in generation fidelity.

Frequently Asked Questions

Example Workflow

1. Run a test session

2. Similarity Index < 80 → test fails

3. Review top-level Key Insights

4. Click into a metric (e.g., levenshtein__generated_answer__expected_answer)

5. View distribution shift and statistical breakdown

6. Add targeted test thresholds to monitor ongoing behavior

7. Adjust model, prompt, or infrastructure as needed

Distributional uses data generated by your AI-powered app to study its behavior and alert you to valuable insights or worrisome trends. The diagram below gives a quick summary of this process:

• Each app usage involves input(s), the resulting output(s), and context about that usage

  • Example: Input is a question from a user; Output is your app’s answer to that question; Context is the time/day that the question was asked.

• As the app is used, you record and store the usage in a data store for later review

  • Example: At 2am every morning, an Airflow job parses all of the previous day’s app usages and sends that info to a data store.

• When data is moved to your data store, it is also submitted to DBNL for testing.

  • Example: The 2am Airflow job is amended to include data augmentation by DBNL Eval and uploading of the resulting Run to trigger automatic app testing.

A Run usually contains many (e.g., dozens or hundreds) rows of inputs + outputs + context, where each row was generated by an app usage. Our insights are statistically derived from the distributions estimated by these rows.

Users

A user is an individual who can log into a dbnl organization.

Permissions

Permissions are settings that control access to operations on resources within a dbnl organization. Permissions are made up of two components.

• Resource: Defines which resource is being controlled by this permission (e.g. projects, users).

• Verb: Defines which operations are being controlled by this permission (e.g. read, write).

For example, the projects.read permission controls access to the read operations on the projects resource. It is required to be able to list and view projects.

Roles

A role consists in a set of permissions. Assigning a role to a user gives the user all the permissions associated with the role.

Roles can be assigned at the organization or namespace level. Assigning roles at the namespace level allows for giving users granular access to projects and their related data.

Org Roles

An org role is a role that can be assigned to a user within an organization. Org role permissions apply to resources across all namespaces.

There are two default org roles defined in every organization.

Org admin

The org admin role has read and write permissions for all org level resources making it possible to perform organization management operations such as creating namespaces and assigning users roles.

Org reader

The org reader role has read-only permissions to org level resources making it possible to navigate the organization by listing users and namespaces.

Assigning a User an Org Role

To assign a user an org role, go to ☰ > Settings > Admin > Users, scroll to the relevant user and select the an org role from the dropdown in the Org Role column.

Namespace Roles

A namespace role is a role that can be assigned to a user within a namespace. Namespace role permissions only apply to resources defined within the namespace in which the role is assigned.

There are three default namespace roles defined in every organization.

Namespace admin

The namespace admin role has read and write permissions for all namespace level resources within a namespace making it possible to perform namespace management operations such as assigning users roles within a namespace.

Namespace writer

The namespace admin role has read and write permissions for all namespace level resources within a namespace except for those resources and operations related to namespace management such as namespace role assignments.

(Experimental) Namespace reader

The namespace reader role has read-only permissions for all namespace level resources within a namespace.

Assigning a User a Namespace Role

To assign a user a namespace role within a namespace, go to ☰ > Settings > Admin > Namespaces, scroll and click on the relevant namespace and then click + Add User.

There are two main options to deploy the dbnl platform as a self-hosted deployment:

• Helm chart: The dbnl platform can be deployed using a Helm chart to existing infrastructure provisioned by the customer.

• Terraform module: The dbnl platform can be deployed using a Terraform module on infrastructure provisioned by the module alongside the platform. This is options is supported on AWS and GCP.

Which option to choose depends on your situation. The Helm chart provides maximum flexibility, allowing users to provision their infrastructure using their own processes, while the Terraform module provides maximum simplicity, reducing the installation to single Terraform command.

Adaptive testing for AI applications requires more information than standard deterministic testing. This is because:

• AI applications are multi-component systems where changes in one part can affect others in unexpected ways. For instance, a change in your vector database could affect your LLM's responses, or updates to a feature pipeline could impact your machine learning model's predictions.

• AI applications are non-stationary, meaning their behavior changes over time even if you don't change the code. This happens because the world they interact with changes - new data comes in, language patterns evolve, and third-party models get updated. A test that passes today might fail tomorrow, not because of a bug, but because the underlying conditions have shifted.

• AI applications are non-deterministic. Even with the exact same input, they might produce different outputs each time. Think of asking an LLM the same question twice - you might get two different, but equally valid, responses. This makes it impossible to write traditional tests that expect exact matches.

To account for this, each time you want to measure the behavior of the AI application, you will need to:

1. Record outcomes at all of the app’s components, and

2. Push a distribution of inputs through the app to study behavior across the full spectrum of possible app usage.

The inputs, outputs, and outcomes associated with a single app usage are grouped in a Result, with each value in a result described as a Column. The group of results that are used to measure app behavior is called a Run. To determine if an app is behaving as expected, you create a Test, which involves statistical analysis on one or more runs. When you apply your tests to the runs that you want to study, you create a Test Session, which is a permanent record of the behavior of an app at a given time.

Personal Access Tokens

A personal access token is a token that can be used for programmatic access to the dbnl platform through the SDK.

Permissions

Token permissions are resolved at use time, not creation time. As such, changing the user permissions after creating a personal access token will change the permissions of the personal access token.

Create a Personal Access Token

To create a new personal access token, go to ☰ > Personal Access Tokens and click Create Token.

Dynamic Baseline (Run Queries)

Setting a Default Baseline Run

You can set a default Baseline Run to be used in all Test Sessions either via the UI or the SDK. Additionally, you can create a Run Query to make your Baseline Run dynamic for each Test Session.

What's in a Project?

Creating a Project

From Scratch

You can create a Project either via the UI or the SDK:

import dbnl
dbnl.login()

project = dbnl.create_project(
    name="My Project",
    description="This is a very important project."
)

Copying a Project

You can quickly copy an existing Project to get up and running with a new one. This will copy the following items into your new Project:

• Test specifications

• Test tags

• Notification rules

There are a couple of ways to copy a Project.

Exporting and Importing

Any Project can be exported to a JSON file; that JSON file can then be adjusted to your liking and imported as a new Project. This is doable both via the UI and the SDK:

import dbnl
dbnl.login()

# Export
project_1 = dbnl.get_or_create_project(name="Existing Project")
export_json = dbnl.export_project_as_json(project=proj1)
# Adjust the project values as you'd like. You will need to change the name.
# An example of the JSON structure is in the collapsible section below
export_json["project"]["name"] = "New Project"

# Import
project_2 = dbnl.import_project_from_json(params=export_json)

{
    "project": {
        "name": "My Project",
        "description": "This is my project."
    },
    "notification_rules": [
        {
            "conditions": [
                {
                    "assertion_name": "less_than",
                    "assertion_params": { "other": 0.85 },
                    "query_name": "test_status_percentage_query",
                    "query_params": {
                        "exclude_tag_ids": [],
                        "include_tag_ids": [],
                        "require_tag_ids": [],
                        "statuses": ["PASSED"]
                    }
                }
            ],
            "name": "Alert if passed tests are less than 85%",
            "notification_integration_names": ["Notification channel"],
            "status": "ENABLED",
            "trigger": "test_session.failed"
        }
    ],
    "tags": [
        {
            "name": "my-tag",
            "description" :"This is my tag."
        }
    ],
    "test_specs": [
        {
            "assertion": { "name": "less_than", "params": { "other": 0.5 } },
            "description": "Testing the difference in the example statistic",
            "name": "Gr.0: Non Parametric Difference: Example_Statistic",
            "statistic_inputs": [
                {
                    "select_query_template": {
                        "filter": null,
                        "select": "{EXPERIMENT}.Example_Statistic"
                    }
                },
                {
                    "select_query_template": {
                        "filter": null,
                        "select": "{BASELINE}.Example_Statistic"
                    }
                }
            ],
            "statistic_name": "my_stat",
            "statistic_params": {},
            "tag_names": ["my-tag"]
        }
    ]
}

Copying

You can also just directly copy a given Project. Again, this can be done via the UI or the SDK:

import dbnl
dbnl.login()


project_1 = dbnl.get_or_create_project(name="Existing Project")
project_2 = dbnl.copy_project(project=project_1, name="New Project")

The full process of reporting a Run ultimately breaks down into three steps:

1. Creating the Run, which includes defining its structure and any relevant metadata

2. Reporting the results of the Run, which include columnar data and scalars

3. Closing the Run to mark it as complete once reporting is finished

Creating a Run

The important parts of creating a run are providing identifying information — in the form of a name and metadata — and defining the structure of the data you'll be reporting to it. As mentioned in the previous section, this structure is called the Run Schema.

Run Schema

A Run schema defines four aspects of the Run's structure:

• Columns (the data each row in your results will contain)

• Scalars (any Run-level data you want to report)

• Index (which column or columns uniquely identify rows in your results)

• Components (functional groups to organize the reported results in the form of a graph)

Columns

Columns are the only required part of a schema and are core to reporting Runs, as they define the shape your results will take. You report your column schema as a list of objects, which contain the following fields:

• name: The name of the column

• description: A descriptive blurb about what the column is

• component: Which part of your application the column belongs to (see Components below)

[
    { 
        "name": "error_type",
        "type": "category",
        "component": "classifier"
    },
    {
        "name": "email",
        "type": "string",
        "description": "raw email text content from source",
        "component": "input"
    },
    { 
        "name": "spam-pred",
        "type": "boolean",
        "component": "classifier"
    },
    {
        "name": "email_id",
        "type": "string",
        "description": "unique id for each email"
    }
]

Scalars

[
    {
        "name": "model_F1",
        "type": "float",
        "description": "F1 Score",
        "component": "classifier"
    },
    { 
        "name": "model_recall",
        "type": "float",
        "description": "Model Recall",
        "component": "classifier"
    }
]

Index

Using the index field within the schema, you have the ability to designate Unique Identifiers – specific columns which uniquely identify matching results between Runs. Adding this information facilitates more direct comparisons when testing your application's behavior and makes it easier to explore your data.

["email_id"]

Components

// Each key defines a component, and the corresponding list defines the
// components downstream from it in your DAG
{
    "input": ["classifier"]
    "classifier": [],
}

Reporting Run Results

Once you've defined the structure of your run, you can upload data to dbnl to report the results of that run. As mentioned above, there are two kinds of results from your run:

• The row-level column results (these each represent the data of a single "usage" of your application)

• The Run-level scalar results (these represent data that apply to all usages in your Run as a whole)

dbnl expects you to upload your results data in the form of a pandas DataFrame. Note that scalars can be uploaded as a single-row DataFrame or as a dictionary of values.

import pandas as pd

column_results = pd.DataFrame({
    "error_type": ["none", "none", "none", "none"],
    "email": [
        "Hello, I am interested in your product. Please send me more information.",
        "Congratulations! You've won a lottery. Click here to claim your prize.",
        "Hi, can we schedule a meeting for next week?",
        "Don't miss out on this limited time offer! Buy now and save 50%."
    ],
    "spam-pred": [False, True, False, True],
    "email_id": ["1", "2", "3", "4"]
})

scalar_results = pd.DataFrame({
    "model_F1": [0.8],
    "model_recall": [0.74]
})
# Above is equalent to:
scalar_results = {
    "model_F1": 0.8,
    "model_recall": 0.74 
}

Closing a Run

Putting it All Together

Now that you understand each step, you can easily integrate all of this into your codebase with a few simple function calls via our SDK:

import dbnl
import pandas as pd
dbnl.login()


proj = dbnl.get_or_create_project(name="My Project")
run_schema = dbnl.create_run_schema(
    columns=[
        {"name": "error_type", "type": "category", "component": "classifier"},
        {"name": "email", "type": "string", "description": "raw email text content from source", "component": "input"},
        {"name": "spam-pred", "type": "boolean", "component": "classifier"},
        {"name": "email_id", "type": "string", "description": "unique id for each email"},
    ],
    scalars=[
        {
            "name": "model_F1",
            "type": "float",
            "description": "F1 Score",
            "component": "classifier"
        },
        { 
            "name": "model_recall",
            "type": "float",
            "description": "Model Recall"
        }
    ],
    index=["email_id"],
    components_dag={
        "input": ["classifier"]
        "classifier": [],
    }
)
# Creates the run, reports results, and closes the run.
run = dbnl.report_run_with_results(
    project=proj,
    display_name="Run 1 of Email Classifier"
    run_schema=run_schema,
    column_data=pd.DataFrame({
        "error_type": ["none", "none", "none", "none"],
        "email": [
            "Hello, I am interested in your product. Please send me more information.",
            "Congratulations! You've won a lottery. Click here to claim your prize.",
            "Hi, can we schedule a meeting for next week?",
            "Don't miss out on this limited time offer! Buy now and save 50%."
        ],
        "spam-pred": [False, True, False, True],
        "email_id": ["1", "2", "3", "4"]
    }),
    scalar_data={
        "model_F1": 0.8,
        "model_recall": 0.74 
    }
)

The dbnl platform architecture consists of a set of services packaged as Docker images and a set of standard infrastructure components.

Infrastructure

The dbnl platform requires the following infrastructure:

• A Kubernetes cluster to host the dbnl platform services.

• A PostgreSQL database to store metadata.

• An object store bucket to store raw data.

• A Redis database to serve as a messaging queue.

• A load balancer to route traffic to the API or UI service.

Services

The dbnl platform consists in three core services:

• The API service (api-srv) serves the dbnl API and orchestrates work across the dbnl platform.

• The worker service (worker-srv) processes async jobs scheduled by the API service.

• The UI service (ui-srv) serves the dbnl UI assets.

Tests are the key tool within dbnl for asserting the behavior and consistency of Runs. Possible goals during testing can include:

• Asserting that your application, holistically or for a chosen column, behaves consistently compared to a baseline.

• Asserting that a chosen column meets its minimum desired behavior (e.g., inference throughput);

• Asserting that a chosen column has a distribution that roughly matches a baseline reference;

What's in a Test?

At a high level, a Test is a statistic and an assertion. Generally, the statistic aggregates the data in a column or columns, and the assertion tests some truth about that aggregation. This assertion may check the values from a single Run, or it may check how the values in a Run have changed compared to a baseline. Some basic examples:

1. Assert the 95th percentile of app_latency_ms is less than or equal to 180

{
    "name": "p95_app_latency_ms",
    "description": "Test the 95th percentile of latency in miliseconds",
    "statistic_name": "percentile",
    "statistic_params": {"percentage": 0.95},
    "assertion": {
        "name": "less_than_or_equal_to",
        "params": {
            "other": 180.0,
        },
    },
    "statistic_inputs": [
        {
            "select_query_template": {
                "select": "{EXPERIMENT}.app_latency_ms"
            }
        },
    ],
}

1. Assert the absolute difference of median of positive_sentiment_score against the baseline is close to 0

{
    "name": "median_sentiment_similar",
    "description": "Test the absolute difference of median on sentiment",
    "statistic_name": "abs_diff_median",
    "statistic_params": {},
    "assertion": {
        "name": "close_to",
        "params": {
            "other": 0.0,
            "tolerance": 0.01,
        },
    },
    "statistic_inputs": [
        {
            "select_query_template": {
                "select": "{EXPERIMENT}.positive_sentiment_score"
            }
        },
        {
            "select_query_template": {
                "select": "{BASELINE}.positive_sentiment_score"
            }
        },
    ],
}

In the next sections, we will explore the objects required for testing alongside the methods for creating tests, running tests, reviewing/analyzing tests, and some best practices.

The Terraform module option provides maximum simplicity. It provisions all the required infrastructure and permissions in your cloud provider of choice before deploying the dbnl platform Helm chart, removing the need to provision any infrastructure or permission separately.

Prerequisites

The following prerequisite steps are required before starting the Terraform module installation.

Configuration

To configure the Terraform module, you will need:

• A domain name to host the dbnl platform (e.g. dbnl.example.com).

An RSA key pair can be generated with:

Requirements

On the environment from which you are planning to install the module, you will need to:

Infrastructure

At a minimum, the user performing the installation needs to be able to provision the following infrastructure:

Installation

Steps

The steps to install the Terraform module using the Terraform CLI are as follows:

Options

For more details on all the installation options, see the Terraform module README file and examples folder.

Configuration

OIDC can be configured using the following options in the dbnl Helm chart or Terraform module:

• audience

• clientId

• issuer

• scopes

Instructions on how to get those options for each provider can be found below.

As you become more familiar with the behavior of your application, you may want to build on the default App Similarity Index test with tests that you define yourself. Let's walk through that process.

Designing a Test

Context-Driven Test Creation

As you browse the dbnl UI, you will see "+" icons or "+ Add Test" buttons appear. These provide context-aware shortcuts for easily creating relevant tests.

At each of these locations, a test creation drawer will open on the right side of the page with several of the fields pre-populated based on the context of the button, alongside a history of the statistic, if relevant. Here are some of the best places to look for dbnl-assisted test creation:

Key Insights

Column or Metric Details

When inspecting the details of a column or metric from a Test Session, there are several "Add Test" buttons provided to allow you to quickly create a test on a relevant statistic. The Statistic History graph can help guide you on choosing a threshold.

Summary Statistics Table

When viewing a Run, each entry in the summary statistics table can be used to seed creation of a test for that chosen statistic.

And More!

These shortcuts appear in several other places in the UI as well when you are inspecting your Runs and Test Sessions; keep an eye out for the "+"!

Templated Tests

Test templates are macros for basic test patterns recommended by Distributional. It allows the user to quickly create tests from a builder in the UI. Distributional provides five classes of test templates:

From the Test Configuration tab on your Project, click the dropdown next to "Add Test".

Select from one of the five options. A Test Creation drawer will appear and the user can edit the statistic, column, and assertion that they desire. Note that each Test Template has a limited set of statistics that it supports.

Creating Tests Manually (Advanced)

If you have a good idea of what you want to test or just want to explore, you can create tests manually from either the UI or via the Python SDK.

Let's say you are building an Q&A chatbot, and you have a column for the length of your bot's responses, word_count. Perhaps you want to ensure that your bot never outputs more than 100 words; in that case, you'd choose:

• The statistic max,

• The assertion less than or equal to ,

• and the threshold 100.

But what if you're not opinionated about the specific length? You just want to ensure that your app is behaving consistently as it runs and doesn't suddenly start being unusually wordy or terse. dbnl makes it easy to test that as well; you might go with:

• The statistic absolute difference of mean,

• The assertion less than,

• and the threshold 20.

Now you're ready to go and create that test, either via the UI or the SDK:

Ingress

Requirements

The dbnl platform needs to be hosted on a domain or subdomain (e.g. dbnl-example.com or dbnl.example.com). It cannot be hosted on a subpath.

HTTPS/SSL

It is recommended that the dbnl platform be served over HTTPS. Support for SSL termination at the load balancer is included.

Egress

Requirements

Currently, the dbnl platform cannot run in an air-gapped environment and requires a few URLs to be accessible via egress.

Artifacts Registry

Required to fetch the dbnl paltform artifacts such as the Helm chart and Docker images.

• https://us-docker.pkg.dev/dbnlai/

Object Store

Required for services to access the object store.

• https://{BUCKET}.s3.amazonaws.com/​ (if using S3)

• https://storage.googleapis.com/{BUCKET} (if using GCS)

OIDC

Required to validate OIDC tokens.

• https://login.microsoftonline.com/{APP_ID}/v2.0/ (if using Microsoft EntraID)

• https://{ACCOUNT}.okta.com/ (if using Okta)

Integrations

Required to use some integrations.

• https://events.pagerduty.com/v2/enqueue​ (if using PagerDuty)

• https://hooks.slack.com/services/ (if using Slack)

abs

Returns the absolute value of the input.

Syntax

abs(expr)

add

Adds the two inputs.

Syntax

add(expr1, expr2)

and

Logical and operation of two or more boolean columns.

Syntax

and(expr1, expr2)

automated_readability_index

Returns the ARI (Automated Readability Index) which outputs a number that approximates the grade level needed to comprehend the text. For example if the ARI is 6.5, then the grade level to comprehend the text is 6th to 7th grade.

Syntax

automated_readability_index(expr)

bleu

Computes the BLEU score between two columns.

Syntax

bleu(expr1, expr2)

character_count

Returns the number of characters in a text column.

Syntax

character_count(expr)

Aliases

• num_chars

divide

Divides the two inputs.

Syntax

divide(expr1, expr2)

equal_to

Computes the element-wise equal to comparison of two columns.

Syntax

equal_to(expr1, expr2)

Aliases

• eq

filter

Filters a column using another column as a mask.

Syntax

filter(expr1, expr2)

flesch_kincaid_grade

Returns the Flesch-Kincaid Grade of the given text. This is a grade formula in that a score of 9.3 means that a ninth grader would be able to read the document.

Syntax

flesch_kincaid_grade(expr)

greater_than

Computes the element-wise greater than comparison of two columns. input1 > input2

Syntax

greater_than(expr1, expr2)

Aliases

• gt

greater_than_or_equal_to

Computes the element-wise greater than or equal to comparison of two columns. input1 >= input2

Syntax

greater_than_or_equal_to(expr1, expr2)

Aliases

• gte

is_valid_json

Returns true if the input string is valid json.

Syntax

is_valid_json(expr)

less_than

Computes the element-wise less than comparison of two columns. input1 < input2

Syntax

less_than(expr1, expr2)

Aliases

• lt

less_than_or_equal_to

Computes the element-wise less than or equal to comparison of two columns. input1 <= input2

Syntax

less_than_or_equal_to(expr1, expr2)

Aliases

• lte

levenshtein

Returns Damerau-Levenshtein distance between two strings.

Syntax

levenshtein(expr1, expr2)

list_has_duplicate

Returns True if the list has duplicated items.

Syntax

list_has_duplicate(expr)

list_len

Returns the length of lists in a list column.

Syntax

list_len(expr)

list_most_common

Most common item in list.

Syntax

list_most_common(expr)

multiply

Multiplies the two inputs.

Syntax

multiply(expr1, expr2)

negate

Returns the negation of the input.

Syntax

negate(expr)

not

Logical not operation of a boolean column.

Syntax

not(expr)

not_equal_to

Computes the element-wise not equal to comparison of two columns.

Syntax

not_equal_to(expr1, expr2)

Aliases

• neq

or

Logical or operation of two or more boolean columns.

Syntax

or(expr1, expr2)

rouge1

Returns the rouge1 score between two columns.

Syntax

rouge1(expr1, expr2)

rouge2

Returns the rouge2 score between two columns.

Syntax

rouge2(expr1, expr2)

rougeL

Returns the rougeL score between two columns.

Syntax

rougeL(expr1, expr2)

rougeLsum

Returns the rougeLsum score between two columns.

Syntax

rougeLsum(expr1, expr2)

sentence_count

Returns the number of sentences in a text column.

Syntax

sentence_count(expr)

Aliases

• num_sentences

subtract

Subtracts the two inputs.

Syntax

subtract(expr1, expr2)

token_count

Returns the number of tokens in a text column.

Syntax

token_count(expr)

word_count

Returns the number of words in a text column.

Syntax

word_count(expr)

Aliases

• num_words

The dbnl sandbox deployment bundles all of the dbnl services and dependencies into a single self-contained Docker container. This container replicates a full scale dbnl deployment by creating a Kubernetes cluster in the container and using Helm to deploy the dbnl platform and its dependencies (postgresql, redis and minio).

Requirements

• The sandbox container needs access to the following two registries to pull the containers for the dbnl platform and its dependencies.

  • us-docker.pkg.dev

  • docker.io

• The sandbox container needs sufficient memory and disk space to schedule the k3d cluster and the containers for the dbnl platform and its dependencies.

Registry Credentials

Usage

Although the sandbox image can be deployed manually using Docker, we recommend using the dbnl CLI to manage the sandbox container. For more details on the sandbox CLI options, run:

$ dbnl sandbox --help

Start the Sandbox

To start the dbnl Sandbox, run:

$ dbnl sandbox start -p ${REGISTRY_PASSWORD}

This will start the sandbox in a Docker container named dbnl-sandbox. It will also create a Docker volume of the same name to persist data beyond the lifetime of the sandbox container.

Stop the Sandbox

To stop the dbnl sandbox, run:

$ dbnl sandbox stop

This will stop and remove the sandbox container. It does not remove the Docker volume and the next time the sandbox is started, it will remount the existing volume, persisting the data beyond the lifetime of the Sandbox container.

Get Sandbox Status

To get the status of the dbnl sandbox, run:

$ dbnl sandbox status

Get Sandbox Logs

To tail the dbnl sandbox logs, run:

$ dbnl sandbox logs

Execute Command in Sandbox

To execute a command in the dbnl sandbox, run:

$ dbnl sandbox exec [COMMAND]

This will execute COMMAND within the dbnl sandbox container. This is a useful tool for debugging the state of the containers running within the sandbox containers. For example:

To get a list of all Kubernetes resources, run:

$ dbnl sandbox exec kubectl get all

To get the logs for a particular pod, run:

$ dbnl sandbox exec kubectl logs [POD]

Delete Sandbox Data

To delete the sandbox data, run:

$ dbnl sandbox delete

Authentication

The sandbox deployment uses username and password authentication with a single user. The user credentials are:

• Username: admin

• Password: password

Storage

The sandbox persists data in a Docker volume named dbnl-sandbox. This volume is persisted even if the sandbox is stopped, making it possible to later resume the sandbox without losing data.

Remote Sandbox

If deploying and hosting the sandbox on a remote host, such as on EC2 or Compute Engine, the sandbox --base-url option needs to be set on start.

For example, if hosting the sandbox on http://example.com:8080, the sandbox needs to be started with:

$ dbnl sandbox start --base-url http://example.com:8080

The Test Sessions section in your Project is a record of all the Test Sessions you've created. You can view a line chart of the pass rate of your Test Sessions over time or view a table with each row representing a Test Session You can click on a point in the chart or a row in the table to navigate to the corresponding Test Session's detail page to dig into what happened within that session.

Test Session Details

When you first open a Test Session's page, you will land on the Summary tab. This tab provides you with summary information about the session such as the App Similarity Index, which tests have failed, and key insights about the session. There are also tabs to see the Similarity Report (more information below) or to view all the test results within the session.

Similarity Indexes

Key Insights

On the Summary tab, you'll notice a list of key insights that dbnl has discovered about your Test Session. The key insights will tell you at a glance which columns or metrics have had the most significant change in your Experiment Run when compared to the baseline. If you are particularly interested in the column or metric going forward, you can quickly add a test for its Similarity Index.

Expanding one of these will allow you to view some additional information such as a history of the Similarity Index for the related column or metric; if you are viewing a metric, it will also tell you the lineage of which columns the metric is derived from.

Similarity Report

The Similarity Report gives you an overview of all the columns your Experiment Run, providing the relevant Similarity Indexes, the ability to quickly create tests from them, and the option to deep-dive into a column. Expanding one of the rows for a column for show you all the metrics calculated for that column, with their own respective Similarity Indexes and details.

If you click on the "See Details" link on any of these rows (or from the Key Insights view), you'll be taken to a view that lets you explore the respective column or metric in detail.

From this view, you can easily compare the changes in the column/metric with graphs and summary statistics. Expanding of the comparison statistics will give you even more information to dig into! Click "Add Test" to quickly create a test on the related statistic.

You can run any tests you've created (or just the default App Similarity Index test) to investigate the behavior of your application.

Running Your Tests

When you run a Test Session, you are running your tests against a given Experiment Run.

Choose a Baseline Run

Create a Test Session

Tests are run within the context of a Test Session, which is effectively just a collection of tests run against an Experiment Run with a Baseline Run. You can create a Test Session, which will immediately run the tests, via the UI or the SDK:

Statistics

Assertions

The dbnl Query Language is a SQL-like language that allows for querying data in runs for the purpose of drawing visualizations, defining metrics or evaluating tests.

Expressions

An expression is a combination of literals, values, operators, and functions. Expressions can evaluate to scalar or columnar values depending on their types and inputs. There are three types of expressions that can be composed into arbitrarily complex expressions.

Literal Expressions

Literal expressions are constant-valued expressions.

Column and Scalar Expressions

Column and scalar expressions are references to columns or scalar values in a run. They use dot-notation to reference a column or scalar within a run.

For example, a column named score in a run with id run_1234 can be referenced with the expression:

Function Expressions

Function expressions are functions evaluated over zero or more other expressions. They make it possible to compose simple expressions into arbitrarily complex expressions.

For example, the word_count function can be used to compute the word count of the text column in a run with id run_1234 with the expression:

Operators

Operators are aliases for function expressions that enhance readability and ease of use. Operator precedence is the same as that of most SQL dialect.

Arithmetic operators

Arithmetic operators provide support for basic arithmetic operations.

Comparison operators

Comparison operators provide support for common comparison operations.

Logical operators

Logical operators provide support for boolean comparisons.

Null Semantics

The dbnl Query Language follows the null semantics of most SQL dialect. With a few exception, when a null value is used as an input to a function or operator, the result is null.

One exception to this is boolean functions and operators where ternary logic is used similar to most SQL dialects.

The primary mechanism for submitting data to Distributional is through our Python SDK. This section servers as a reference for the various functionalities available for interacting with dbnl via the SDK.

Example Usage

Filters can be used to specify a sub-selection of rows in Runs you would like to be tested.

For example, you might want to create a test that asserts that the absolute difference of means of the correct churn predictions is <= 0.2 between Baseline and Experiment Runs, only for rows where the loc column is NY.

Apply a Filter to a Test

Once you've used one of the methods above, you can now see the new test in the Test Configuration tab of your Project.

When a Test Session is created, this test will use the defined filters to sub-select for the rows that have the loc column equal to NY.

Notifications provide a way for users to be automatically notified about critical test events (e.g., failures or completions) via third-party tools like PagerDuty and Slack.

With Notifications you can:

• Add critical test failure alerting to your organization’s on-call

• Create custom notifications for specific feature tests

• Stay informed when a new test session has started

What's in a Notification?

A Notification is composed of two major elements:

• The Notification Channel — this contains the metadata for how and where a Notification will be sent

• The Notification Criteria — this defines the rules for when a Notification will be generated

Setting up a Notification Channel in your Namespace

Before setting up a Notification in your project, you must have a Notification Channel set up in your Namespace. A Notification Channel describes who will be notified and how. A Notification Channel in a Namespace can be used by Notifications across all Projects belonging to that Namespace.

1. In your desired Namespace, choose Notification Channels in the menu sidebar. Note: you must be a Namespace admin in order to do this.

2. Click the New Notification Channel button to navigate to the creation form.

3. Fill out the appropriate fields.

   1. Optional: If you’d like to test that your Notification Channel is set up correctly, click the Test button. If it is correctly set up, you should receive a notification through the integration you’ve selected.

4. Click the Create Notification Channel button. Your channel will now be available when setting up your Notification.

Supported Third-Party Notification Channels

Note: More coming up in the product roadmap!

Setting up a Notification in your Project

1. Navigate to your Project and click the Notifications tab.

1. Click the "New Notification" button to navigate to the creation form.

2. Click the "Create Notification" button. Your Notification will now notify you when your specified criteria are met!

Notification Criteria

Organization

An organization, or org for short, corresponds to a dbnl deployment.

Organization Resources

Some resources, such as users, are defined at the organization level. Those resources are sometimes referred to as organization resources or org resources.

Namespaces

A namespace is a unit of isolation within a dbnl organization.

Namespace Resources

Most resources, including projects and their related resources, are defined at the namespace level. Resources defined within a namespace are only accessible within that namespace providing isolation between namespaces.

Default Namespace

All organizations include a namespace named default. This namespace cannot be modified or deleted.

Switching Namespace

To switch namespace, use the namespace switcher in the navigation bar.

Creating a Namespace

To create a namespace, go to ☰ > Settings > Admin > Namespaces and click the + Create Namespace button.

Adding a User to a Namespace

The self-hosted deployment option allows you to deploy the dbnl platform directly in your cloud or on-premise environment.

get_column_schemas_from_dataframe

dbnl.util.get_column_schemas_from_dataframe(df: DataFrame) → list[RunSchemaColumnSchemaDict]

get_default_components_dag_from_column_schemas

dbnl.util.get_default_components_dag_from_column_schemas(column_schemas: Sequence[ColumnSchemaDict]) → dict[str, list[str]] | None

Gets the unconnected components DAG from a list of column schemas. If there are no components, returns None. The default components dag is of the form {

“component1”: [], “component2”: [], …}

• Parameters:column_schemas – list of column schemas

• Returns: dictionary of components DAG or None

get_run_config_column_schemas_from_dataframe

dbnl.util.get_run_config_column_schemas_from_dataframe(df: DataFrame) → list[RunConfigPrimitiveColumnSchemaDict | RunConfigContainerColumnSchemaDict]

get_run_config_scalar_schemas_from_dataframe

dbnl.util.get_run_config_scalar_schemas_from_dataframe(df: DataFrame) → list[RunConfigPrimitiveScalarSchemaDict | RunConfigContainerScalarSchemaDict]

get_run_schema_columns_from_dataframe

dbnl.util.get_run_schema_columns_from_dataframe(df: DataFrame) → list[RunSchemaColumnSchema]

get_run_schema_scalars_from_dataframe

dbnl.util.get_run_schema_scalars_from_dataframe(df: DataFrame) → list[RunSchemaScalarSchema]

get_scalar_schemas_from_dataframe

dbnl.util.get_scalar_schemas_from_dataframe(df: DataFrame) → list[RunSchemaScalarSchemaDict]

make_test_session_input

dbnl.util.make_test_session_input(*, run: Run | None = None, run_query: RunQuery | None = None, run_alias: str = 'EXPERIMENT') → TestSessionInput

Create a TestSessionInput object from a Run or a RunQuery. Useful for creating TestSessions right after closing a Run.

• Parameters:

  • run – The Run to create the TestSessionInput from

  • run_query – The RunQuery to create the TestSessionInput from

  • run_alias – Alias for the Run, must be ‘EXPERIMENT’ or ‘BASELINE’, defaults to “EXPERIMENT”

• Raises:DBNLInputValidationError – If both run and run_query are None

• Returns: TestSessionInput object

The Helm chart option separates the infrastructure and permission provisioning process from the dbnl platform deployment process, allowing you to manage the infrastructure, permissions and Helm chart using their existing processes.

Prerequisites

The following prerequisite steps are required before starting the Helm chart installation.

Infrastructure

To successfully deploy the dbnl Helm chart, you will need the following infrastructure:

Configuration

To configure the dbnl Helm chart, you will need:

• A hostname to host the dbnl platform (e.g. dbnl.example.com).

• A set of dbnl registry credentials to pull the dbnl artifacts (e.g. Docker images, Helm chart).

An RSA key pair can be generated with:

openssl genrsa -out dbnl_dev_token_key.pem 2048

Requirements

To install the dbnl Helm chart, you will need:

Permissions

For the services deployed by the Helm chart to work as expected, they will need the following permissions and network accesses:

• api-srv

  • Network access to the database.

  • Network access to the Redis database.

  • Permission to read, write and generate pre-signed URLs on the object store bucket.

• worker-srv

  • Network access to the database.

  • Network access to the Redis database.

  • Permission to read and write to the object store bucket.

Installation

Steps

The steps to install the Helm chart using the Helm CLI are as follows:

1. Create an image pull secret with the your dbnl registry credentials.

kubectl create secret docker-registry dbnl-docker-cfg \
    --docker-server="us-docker.pkg.dev/dbnlai/images" \
    --docker-username="${DBNL_REGISTRY_USERNAME}" \
    --docker-password="${DBNL_REGISTRY_PASSWORD}"

1. Create a minimal values.yaml file.

imagePullSecrets:
  - name: dbnl-docker-cfg
  
auth:
  # For more details on OIDC options, see OIDC Authentication section.
  oidc:
    enabled:   true
    issuer:    oidc.example.com
    audience:  xxxxxxxx
    clientId:  xxxxxxxx
    scopes:    "openid email profile"

db:
  host: db.example.com
  port: 5432
  username: user
  password: password
  database: database

redis:
  host: redis.example.com
  port: 6379
  username: user
  password: password

ingress:
  enabled: true
  api:
    host: dbnl.example.com
  ui:
    host: dbnl.example.com

storage:
  s3:
    enabled: true
    region: us-east-1
    bucket: example-bucket

1. Log into the dbnl Helm registry.

helm login \
    --username ${DBNL_REGISTRY_USERNAME} \
    --password ${DBNL_REGISTRY_PASSWORD} \
    us-docker.pkg.dev/dbnlai/charts

1. Install the Helm chart.

helm upgrade \
    --install \
    -f values.yaml \
    dbnl oci://us-docker.pkg.dev/dbnlai/charts/dbnl

Options

For more details on all the installation options, see the Helm chart README and values.yaml files. The chart can be inspected with:

helm show all oci://us-docker.pkg.dev/dbnlai/charts

Data for a run is split between the object store (e.g. S3, GCS) and the database.

• Metadata (e.g. name, schema) and aggregate data (e.g. summary statistics, histograms) are stored in the database.

• Raw data is stored in the object store.

Database

Database access is always done through the API with the API enforcing access controls to ensure users only access data for which they have permission.

Object Store

When uploading or downloading data for a run, the SDK first sends a request for a pre-signed upload or download URL to the API. The API enforces access controls, returning an error if the user is missing the necessary permissions. Otherwise, it returns a pre-signed URL which the SDK then uses to upload or download the data.

class dbnl.eval.metrics.Metric

column_schema() → RunSchemaColumnSchemaDict

Returns the column schema for the metric to be used in a run config.

• Returns: _description_

component() → str | None

description() → str | None

Returns the description of the metric.

• Returns: Description of the metric.

abstract evaluate(df: pd.DataFrame) → pd.Series[Any]

Evaluates the metric over the provided dataframe.

• Parameters:df – Input data from which to compute metric.

• Returns: Metric values.

abstract expression() → str

Returns the expression representing the metric (e.g. rouge1(prediction, target)).

• Returns: Metric expression.

greater_is_better() → bool | None

If true, larger values are assumed to be directionally better than smaller once. If false, smaller values are assumged to be directionally better than larger one. If None, assumes nothing.

• Returns: True if greater is better, False if smaller is better, otherwise None.

abstract inputs() → list[str]

Returns the input column names required to compute the metric. :return: Input column names.

abstract metric() → str

Returns the metric name (e.g. rouge1). :return: Metric name.

abstract name() → str

Returns the fully qualified name of the metric (e.g. rouge1__prediction__target).

• Returns: Metric name.

run_schema_column() → RunSchemaColumnSchema

Returns the column schema for the metric to be used in a run config.

• Returns: _description_

abstract type() → Literal['boolean', 'int', 'long', 'float', 'double', 'string', 'category']

Returns the type of the metric (e.g. float)

• Returns: Metric type.

class dbnl.eval.metrics.RougeScoreType(value)

An enumeration.

FMEASURE = 'fmeasure'

PRECISION = 'precision'

RECALL = 'recall'

answer_quality_llm_accuracy

Computes the accuracy of the answer by evaluating the accuracy score of the answer using a language model.

This metric is generated by an LLM using a specific specific prompt named llm_accuracy available in dbnl.eval.metrics.prompts.

• Parameters:

  • input – input column name

  • context – context column name

  • prediction – prediction column name

  • eval_llm_client – eval_llm_client

• Returns: accuracy metric

answer_quality_llm_answer_correctness

Returns answer correctness metric.

This metric is generated by an LLM using a specific specific prompt named llm_answer_correctness available in dbnl.eval.metrics.prompts.

• Parameters:

  • input – input column name

  • prediction – prediction column name

  • target – target column name

  • eval_llm_client – eval_llm_client

• Returns: answer correctness metric

answer_quality_llm_answer_similarity

Returns answer similarity metric.

This metric is generated by an LLM using a specific specific prompt named llm_answer_similarity available in dbnl.eval.metrics.prompts.

• Parameters:

  • input – input column name

  • prediction – prediction column name

  • target – target column name

  • eval_llm_client – eval_llm_client

• Returns: answer similarity metric