LogoLogo
AboutBlogLaunch app ↗
v0.22.x
v0.22.x
  • Get Started
  • Overview
  • Getting Access to Distributional
  • Install the Python SDK
  • Quickstart
  • Learning about Distributional
    • Distributional Concepts
    • Why We Test Data Distributions
    • The Flow of Data
  • Using Distributional
    • Projects
    • Runs
      • Reporting Runs
      • Setting a Baseline Run
    • Metrics
    • Tests
      • Creating Tests
        • Using Filters in Tests
        • Available Statistics and Assertions
      • Running Tests
      • Reviewing Tests
        • What Is a Similarity Index?
    • Notifications
    • Access Controls
      • Organization and Namespaces
      • Users and Permissions
      • Tokens
  • Platform
    • Sandbox
    • Self-hosted
      • Architecture
      • Deployment
        • Helm Chart
        • Terraform Module
      • Networking
      • OIDC Authentication
      • Data Security
  • Reference
    • Query Language
      • Functions
    • Python SDK
      • dbnl
      • dbnl.util
      • dbnl.experimental
      • Classes
      • Eval Module
        • Quick Start
        • dbnl.eval
        • dbnl.eval.metrics
        • Application Metric Sets
        • How-To / FAQ
        • LLM-as-judge and Embedding Metrics
        • RAG / Question Answer Example
    • CLI
  • Versions
    • Release Notes
Powered by GitBook

© 2025 Distributional, Inc. All Rights Reserved.

On this page
  • create_test
  • create_test_generation_session
  • create_test_recalibration_session
  • get_or_create_tag
  • get_test_sessions
  • get_tests
  • prepare_incomplete_test_spec_payload
  • wait_for_test_generation_session
  • wait_for_test_recalibration_session
  • wait_for_test_session

Was this helpful?

Export as PDF
  1. Reference
  2. Python SDK

dbnl.experimental

create_test

dbnl.experimental.create_test(*, test_spec_dict: TestSpecDict) → dict[str, Any]

Create a new Test Spec

  • Parameters:test_spec_dict – A dictionary containing the Test Spec schema.

  • Raises:

    • DBNLNotLoggedInError – dbnl SDK is not logged in.

    • DBNLAPIValidationError – Test Spec does not conform to expected format.

    • DBNLDuplicateError – Test Spec with the same name already exists in the Project.

  • Returns: The JSON dict of the created Test Spec object. The return JSON will contain the id of the Test Spec.

Test Spec JSON Structure

{
    "project_id": string,

    # Test data
    "name": string, // must be unique to Project
    "description": string | null,
    "statistic_name": string,
    "statistic_params": map[string, any],
    "statistic_inputs": list[
        {
            "select_query_template": {
                "select": string, // a column or a function on column(s)
                "filter": string | null
            }
        }
    ],
    "assertion": {
        "name": string,
        "params": map[string, any]
    },
    "tag_ids": string[] | null
}

create_test_generation_session

dbnl.experimental.create_test_generation_session(*, run: Run, columns: list[str | dict[Literal['name'], str]] | None = None) → TestGenerationSession

Create a Test Generation Session

  • Parameters:

    • run – The Run to use when generating tests.

    • columns – List of columns in the Run to generate tests for. If None, all columns in the Run will be used, defaults to None. If a list of strings, each string is a column name. If a list of dictionaries, each dictionary must have a ‘name’ key, and the value is the column name.

  • Raises:

    • DBNLNotLoggedInError – dbnl SDK is not logged in.

    • DBNLInputValidationError – arguments do not conform to expected format.

  • Returns: The TestGenerationSession that was created.

Examples:

import dbnl
dbnl.login()


run = dbnl.get_run(run_id="run_0000000")
dbnl.experimental.create_test_generation_session(
    run=run,
    columns=["col1", "col4"],
)

create_test_recalibration_session

dbnl.experimental.create_test_recalibration_session(*, test_session: TestSession, feedback: str, test_ids: list[str] | None = None) → TestRecalibrationSession

Create a Test Recalibration Session by redefining the expected output for tests in a Test Session

  • Parameters:

    • test_session – Test Session to recalibrate

    • feedback – Feedback for the recalibration. Can be ‘PASS’ or ‘FAIL’.

    • test_ids – List of test IDs to recalibrate, defaults to None. If None, all tests in the Test Session will be recalibrated.

  • Raises:

    • DBNLNotLoggedInError – dbnl SDK is not logged in.

    • DBNLInputValidationError – arguments do not conform to expected format.

  • Returns: Test Recalibration Session

IMPORTANT

If some generated Tests failed when they should have passed and some passed when they should have failed, you will need to submit 2 separate calls, one for each feedback result.

get_or_create_tag

dbnl.experimental.get_or_create_tag(*, project_id: str, name: str, description: str | None = None) → dict[str, Any]

Get the specified Test Tag or create a new one if it does not exist

  • Parameters:

    • project_id – The id of the Project that this Test Tag is associated with.

    • name – The name of the Test Tag to create or retrieve.

    • description – An optional description of the Test Tag. Limited to 255 characters.

  • Returns: The dictionary containing the Test Tag

  • Raises:DBNLNotLoggedInError – dbnl SDK is not logged in.

Sample Test Tag JSON

{
    # Tag metadata
    "id": string,
    "org_id": string,
    "created_at": timestamp,
    "updated_at": timestamp,

    # Tag data
    "name": string,
    "author_id": string,
    "description": string?,
    "project_id": string,
}

get_test_sessions

dbnl.experimental.get_test_sessions(*, project: Project) → list[TestSession]

Get all Test Sessions in the given Project

  • Parameters:project – Project from which to retrieve Test Sessions

  • Returns: List of Test Sessions

  • Raises:DBNLNotLoggedInError – dbnl SDK is not logged in.

get_tests

dbnl.experimental.get_tests(*, test_session_id: str) → list[dict[str, Any]]

Get all Tests executed in the given Test Session

  • Parameters:test_session_id – Test Session ID

  • Returns: List of test JSONs

  • Raises:DBNLNotLoggedInError – dbnl SDK is not logged in.

Sample Test JSON

{
    # Test metadata
    "id": string,
    "org_id": string,
    "created_at": timestamp,
    "updated_at": timestamp,
    "test_session_id": string,

    # Test data
    "author_id": string,
    "value": any?,
    "failure": string?,
    "status": enum(PENDING, RUNNING, PASSED, FAILED),
    "started_at": timestamp?,
    "completed_at": timestamp?,

    # Test Spec data
    "test_spec_id": id,
    "name": string,
    "description": string?,
    "statistic_name": string,
    "statistic_params": map[string, any],
    "assertion": {
        "name": string,
        "params": map[string, any]
        "status": enum(...),
        "failure": string?
    },
    "statistic_inputs": list[
        {
        "select_query_template": {
            "select": string
        }
        }
    ],
    "tag_ids": string[]?,
    }

prepare_incomplete_test_spec_payload

dbnl.experimental.prepare_incomplete_test_spec_payload(*, test_spec_dict: IncompleteTestSpecDict, project_id: str | None = None) → TestSpecDict

Formats a Test Spec payload for the API. Add project_id if it is not present. Replace tag_names with tag_ids.

  • Parameters:

    • test_spec_dict – A dictionary containing the Test Spec schema.

    • project_id – The Project ID, defaults to None. If project_id does not exist in test_spec_dict, it is required as an argument.

  • Raises:DBNLInputValidationError – Input does not conform to expected format

  • Returns: The dictionary containing the newly formatted Test Spec payload.

wait_for_test_generation_session

dbnl.experimental.wait_for_test_generation_session(*, test_generation_session: TestGenerationSession, timeout_s: int = 180) → TestGenerationSession

Wait for a Test Generation Session to finish. Polls every 3 seconds until it is completed.

  • Parameters:

    • test_generation_session – The TestGenerationSession to wait for.

    • timeout_s – The total wait time (in seconds) for Test Generation Session to complete, defaults to 180.

  • Raises:

    • DBNLNotLoggedInError – dbnl SDK is not logged in.

    • DBNLError – Test Generation Session did not complete after waiting for the timeout_s seconds

  • Returns: The completed TestGenerationSession

wait_for_test_recalibration_session

dbnl.experimental.wait_for_test_recalibration_session(*, test_recalibration_session: TestRecalibrationSession, timeout_s: int = 180) → TestRecalibrationSession

Wait for a Test Recalibration Session to finish. Polls every 3 seconds until it is completed.

  • Parameters:

    • test_recalibration_session – The TestRecalibrationSession to wait for.

    • timeout_s – The total wait time (in seconds) for Test Recalibration Session to complete, defaults to 180.

  • Returns: The completed TestRecalibrationSession

  • Raises:

    • DBNLNotLoggedInError – dbnl SDK is not logged in.

    • DBNLError – Test Recalibration Session did not complete after waiting for the timeout_s seconds

wait_for_test_session

dbnl.experimental.wait_for_test_session(*, test_session: TestSession, timeout_s: int = 180) → TestSession

Wait for a Test Session to finish. Polls every 3 seconds until it is completed.

  • Parameters:

    • test_session – The TestSession to wait for

    • timeout_s – The total wait time (in seconds) for Test Session to complete, defaults to 180.

  • Returns: The completed TestSession

  • Raises:

    • DBNLNotLoggedInError – dbnl SDK is not logged in.

    • DBNLError – Test Session did not complete after waiting for the timeout_s seconds

Previousdbnl.utilNextClasses

Last updated 2 months ago

Was this helpful?