Setting a Baseline Run

The "Baseline Run" is a core concept in dbnl that, appropriately, refers to the Run used as a baseline when executing a Test Session. Conversely, the Run being tested is called the "Experiment Run". Any tests you've created that compare statistics will test the values in the experiment relative to the baseline.

Dynamic Baseline (Run Queries)

Depending on your use case, you may want to make your Baseline Run dynamic. You can use a Run Query for this. Currently, dbnl supports setting a Run Query that looks back a number of previous runs. For example, in a production testing use case, you may want to use the previous Run as the baseline for each Test Session, so you'd create a Run Query that looks back 1 run. See the UI example in the Setting a Default Baseline Run section for information on how to create a Run Query. You can also create a Run Query via the SDK.

Setting a Default Baseline Run

You can choose a Baseline Run at the time of Test Session creation. If you do not provide one, dbnl will use your Project's default Baseline Run. See Running Tests for more information.

You can set a default Baseline Run to be used in all Test Sessions either via the UI or the SDK. Additionally, you can create a Run Query to make your Baseline Run dynamic for each Test Session.

UI

From your Project, click the "Test Configuration" tab. Choose a Run or Run Query from the Baseline Run dropdown.

From your Project, click "Test Configuration" and select a Baseline Run in the dropdown.

SDK

You can set a Run as baseline via the set_run_as_baseline or set_run_query_as_baseline functions.

import dbnl

# Get a reference to a Run either by creating one or fetching by ID
run = dbnl.get_run(run_id="run_abc123") # or dbnl.report_result_with_results
dbnl.set_run_as_baseline(run=run)

# You can also use a Run Query for a dynamic baseline
project = dbnl.get_or_create_project(name="My Project")
run_query = dbnl.create_run_query(
  project=project,
  name="Look back 3 runs",
  query={
    "offset_from_now": 3,
  },
)
dbnl.set_run_query_as_baseline(run_query=run_query)