Our Privacy Policy can be found here.

The Data Pipeline is how DBNL converts raw production AI log data into actionable Insights and Dashboards for each Project and stores it for future analysis within the Data Model.

The Data Pipeline is invoked as production log data is ingested into your DBNL Deployment. This process kicks off at data ingestion if using SDK Log Ingestion and daily at UTC midnight for OTEL Trace Ingestion.

A DBNL Data Pipeline Run performs the following tasks:

1. Ingest: Raw production log data is flattened into Columns. By using the DBNL Semantic Convention certain Columns can have rich semantic meaning and allow for deeper Insights to be generated.

2. Enrich: are computed on the ingested log data, creating in each log line corresponding to each computed Metric.

3. Analyze: Various unsupervised learning techniques are applied to the enriched log data to discover behavioral signals corresponding to shifts, segments, or outliers in behavior as .

4. Publish: and updated charts are published to for consumption by the user.

A single log represents the captured behavior from a production AI product. Data from each log is flattened into multiple Columns, using the whenever possible. The required Columns for a given log are:

• input: The text input to the LLM.

• output: The text response from the LLM.

• timestamp: The UTC timecode associated with the LLM call.

Only columns defined in the are supported as top-level columns. To attach custom metadata, use span attributes via the .

are computed from Columns and appended as new Columns for each log.

represent filters on Columns of Logs.

Data Connections are how production AI log data is ingested into your DBNL Deployment as part of the Data Pipeline. Each Project has one ingestion method that is set at creation. If you need to change this later you can do this via the Project settings page.

DBNL supports two methods of data ingestion:

• OTEL Trace Ingestion: Publish OTEL traces directly to DBNL as the product runs.

• SDK Log Ingestion: Push data manually or as part of a daily orchestration job using the Python SDK.

From the landing page click on "Data Connections" on the left panel. On the Data Connections landing page "+ Add Data Connection" in the upper right. Provide a required name for the Data Connection and an optional description. All Data Connections will be available to any creating a in the .

The DBNL platform combines configurable infrastructure, secure data handling, and workspace administration so teams can deploy adaptive analytics in their own environments.

• – Options for running DBNL from quick sandboxes to fully managed clusters.

• – Service layout, data flow, and operational considerations.

• – Connectivity requirements for the platform and its integrations.

Segments are saved filters on data corresponding to a specific behavioral signal discovered manually or from an .

All Segments are computed and published to the as part of the and adapt future analytics by informing DBNL that the saved Segment is a meaningful bifurcation of the data.

Create a segment when you've identified a meaningful behavioral pattern you want to track over time, such as:

• Error conditions: Logs containing specific error types or failure patterns

• High-value interactions

Notification Connections allow you to integrate various publish/subscribe notification tools to be informed when specific actions are completed in your DBNL Deployment.

Supported notification channels include Email, Slack, and Pagerduty with more coming soon.

Supported notification events currently include:

• Data Run complete/error.

• Insights generated.

The Explorer enables rapid analysis and triage of by performing graphical and statistical comparison between different subsets of over different time windows and/or filters.

You can access the Explorer in three ways:

1. From the main navigation: Click "Explorer" in the left sidebar

2. From an Insight: Click the "View in Explorer" button on any Insight card

Our End User License Agreements (EULA) for deployments can be found here.

Our End User Service Agreement for SaaS can be found here.

Ingress

Requirements

The DBNL platform needs to be hosted on a domain or subdomain (e.g. dbnl-example.com or dbnl.example.com). It cannot be hosted on a subpath.

HTTPS/SSL

It is recommended that the DBNL platform be served over HTTPS. Support for SSL termination at the load balancer is included.

Egress

Requirements

Currently, the dbnl platform cannot run in an air-gapped environment and requires a few URLs to be accessible via egress.

Artifacts Registry

Required to fetch the DBNL platform artifacts such as the Helm chart and Docker images for installation and upgrades.

• https://ghcr.io/dbnlai/

An Internal Object Store

Required for services to access an object store, this data does not leave your environment.

• https://{BUCKET}.s3.amazonaws.com/​ (if using S3)

• https://storage.googleapis.com/{BUCKET} (if using GCS)

• https://{STORAGE_ACCOUNT}.blob.core.windows.net (if using Azure)

OIDC

Required to validate OIDC tokens, if using a 3rd party OIDC provider.

• https://login.microsoftonline.com/{APP_ID}/v2.0/ (if using Microsoft EntraID)

• https://{ACCOUNT}.okta.com/ (if using Okta)

The Status page shows you all ongoing and previous DBNL Data Pipeline runs for your project.

These runs represent the entire Data Pipeline, including:

• Data ingestion from the specified Data Connection for the Project

• Log enrichment by appending Metrics using the Model Connection

• Analysis and publishing of

You can view the current status of each run grouped by data date range, which time window DBNL was ingesting data for. If a Data Pipeline run has errored you can hover over the error status to view the exception and restart the run by clicking on the restart button in the actions column.

Typical pipeline run times depend on log volume and Model Connection latency:

Pipeline stages and their typical durations:

1. Ingest (10-30 seconds): Upload and validate data

2. Enrich (60-80% of total time): Compute metrics using Model Connection

3. Analyze (10-20% of total time): Run unsupervised learning algorithms

4. Publish (30-60 seconds): Update dashboards and generate insights

What is DBNL?

DBNL is an Adaptive Analytics platform designed to discover and track hidden behavioral signals in production AI logs and traces so that product owners can confidently know exactly where and how to improve their AI products over time. The platform gives a detailed snapshot of agent behavior - the interplay and correlations between users, context, tools, models, and metrics. Patterns in behavioral signals are automatically surfaced as Insights that can be investigated and tracked. This empowers AI teams to accelerate the AI data flywheel by pinpointing the signals and specific examples they can use to improve their products with confidence.

Why DBNL?

The AI data flywheel promises better agentic performance over time through post-training optimization on real production data, but not all data is created equal. DBNL helps AI product owners fill the critical gap between high level monitoring tools (focused on aggregate performance through evals, logging, and tracing) and low level debugging tools (focused on single-trace observability) to pinpoint hidden behavioral signals and relevant example data for post-training optimization. This allows AI product owners to better understand agent and user behavior to know exactly where and how to improve AI products in production.

Who is DBNL for?

Distributional is built for AI product teams looking to understand and improve their AI agents that have

• Scale: More than 1,000 traces or logs per day (too many to manually inspect)

• Data: Access to full spans from OTEL trace data or similarly rich data for analysis (See our )

• Value: Quantifiable business metrics to track and improve

• Understanding: You already monitor aggregate performance (but need richer analysis to know where and how to improve and fix your AI agents)

DBNL is openly distributed and free to within your cloud or on-premises environment, keeping your data safe, secure, and always under your control. Head over to our to get started right away.

DBNL integrates with your existing AI tools to easily and securely perform analytics for any AI product. The ingests, enriches, and analyzes production AI logs and traces, surfacing behavioral signals. These signals are published to and as , allowing users to discover, investigate, and track them as part of the . This gives you concrete signals and relevant data to power improvements to your agent as part of an AI data flywheel.

• Ready to start using DBNL? Head straight to our to get set up on the platform and start testing your AI products right away for free.

• Want to learn more about the workflow? Check out the .

• Want to understand more about the platform? Check out the , options, and other aspects of the .

Data does not leave your deployment. A DBNL Deployment is self contained and does not "call home" or send your data back to a hosted cloud service keeping your data safe, secure, and always under your control.

Location of Data

Data is split between Databases (e.g. postgres, redis, clickhouse) and an Object Store (e.g. S3, GCS).

• Databases contain:

  • Metadata (e.g. name, schema)

  • Aggregate data (e.g. summary statistics, histograms).

  • Raw traces (e.g. for )

• Object Store contains:

  • Raw data (e.g. enriched logs)

All data accesses are mediated by the API ensuring the enforcement of access controls. For more details on permissions, see .

Database access is always done through the API with the API enforcing access controls to ensure users only access data for which they have permission.

Direct object store access is required to upload or download raw Run data using the SDK. are used to provide limited direct access. This access is limited in both time and scope, ensuring only data for a specific Run is accessible and that it is only accessible for a limited time.

When uploading or downloading data for a Run, the SDK first sends a request for a pre-signed upload or download URL to the API. The API enforces access controls, returning an error if the user is missing the necessary permissions. Otherwise, it returns a pre-signed URL which the SDK then uses to upload or download the data.

An Insight is a detected behavioral signal generated from unsupervised analysis of enriched logs as part of the Data Pipeline.

Insights represent signals that the user can triage and refine through the Explorer or inspection of Logs and track as Metrics or Segments. They represent clusters of log data defined by filters on Columns corresponding to unique patterns of behavior. Insights can point to errors, issues, or changes within your agentic application that can be used to inform where and how to improve or fix your agent as part of an Analytics-Driven Data Flywheel. Insights can reveal new metrics to eval or incorporate into reward functions, they can also pinpoint specific segments of data that can be used for focued post-training optimization, whether that be fine tuning, reinforcement learning, hyperparameter optimization, prompt optimization, or any other method.

What to Expect

Typical insight volume: Most projects generate 5-20 new insights per week. Projects with stable, consistent behavior may generate fewer insights, while projects with volatile or rapidly changing behavior may generate more.

Insight Structure

• Summary: Human readable explaination of Insight with impact and severity

• Examples: Specific evidence of the discovered pattern from the logs

• Potential Fixes: How you could remediate the issue, ranked by effort. Used to complete the Analytics-Driven Data Flywheel by helping you fix or improve your agent.

• Suggested Segment: A filter on the logs that approximates the behavior observed by the Insight. Used for tracking the issue and that it is corrected by the chosen fix.

If you see no insights:

• DBNL requires at least 7 days of data to establish behavioral baselines

• Check the to ensure pipeline runs are completing successfully

• Verify sufficient log volume (insights are more meaningful with hundreds of logs per day)

• Very stable systems with little variation may naturally generate fewer insights

This section contains examples demonstrating how to get data into the DBNL platform using various methods all adhering to the DBNL Semantic Convention. Each example includes working code, detailed explanations, and guidance on when to use each approach.

Getting Started

If you're new to DBNL, start with the Quickstart which walks you through deploying a local sandbox and uploading your first data.

Data Input Examples

DBNL supports multiple ways to ingest data.

• Direct OTEL Ingestion: Stream traces in real-time from OTEL-instrumented applications

• SDK from JSON: Load trace data from JSONL files and upload via the Python SDK

• : Batch upload OpenTelemetry trace exports

• : Import traces exported from Langfuse

All example code is available in the GitHub repository.

Check out the and Walkthroughs to see DBNL in action for various end-to-end use cases.

This section contains examples demonstrating how to use DBNL in various scenarios using simulated data in real world scenarios.

Outing Agent Prompt Optimization Walkthrough

Projects are the main organizational tool in DBNL. Generally, you'll create one Project for every AI application that you'd like to analyze with DBNL. After a Project is created, you can start analyzing signals from your Production AI application using the .

A Project is initially defined by

• A : This is how the production AI log data is ingested into DBNL, one of or .

• A default : This is how DBNL creates LLM-as-judge metrics by default for the project. This is also how DBNL generates some of the insights as part of the unsupervised analytics in the Analyze step.

We’ve made it easy to get started exploring DBNL in a variety of ways:

1. . Start here if you want to start exploring the DBNL product with pre-populated data in a hosted environment. You won’t have to deploy anything but you also won’t see how data is ingested in the product.

2. . Start here to install the DBNL SDK and Sandbox locally to create your first project, submit log data to it, and start analyzing. Technical users that want to roll up their sleeves but don’t have project data to work with can start here.

The Adaptive Analytics Workflow is the core mechanism for discovering, investigating, and tracking hidden behavioral signals from your production AI log data.

1. Discover: New signals are displayed as and within a .

2. Investigate: Signals can be triaged and refined through population and temporal comparisons with the or dive directly into the evidence with the corresponding subset of enriched .

The Logs page allows the user to inspect specific logs with certain properties defined by

• A specific time window (default: last 7 full days of data)

• Specific filters on or (default: no filters)

Individual Logs can be viewed in a variety of ways:

This section contains examples demonstrating how to use DBNL in various scenarios. Each example includes , detailed explanations, and guidance on when to use each approach.

The provides a comprehensive walkthrough of building an end-to-end analytics pipeline:

• Generate OTEL traces from a Google ADK calculator agent

• Convert and augment trace data with computed metrics

Each DBNL deployment corresponds to a single Organization containing:

• All Namespaces

• All Users

A Namespace is a unit of isolation within an Organization containing:

Push data manually or as part of a daily orchestration job using our . This ingestion method allows for the most flexibility, but requires the most off-platform coding.

The following fields are required regardless of which ingestion method you are using:

• input: The text input to the LLM as a string.

• output

Dashboards are collections of histograms, time series and statistics of monitored Columns, tracked Segments, and generated Metrics for user-driven analysis.

There are three default dashboards for each Project:

• Monitoring Dashboard: Distributional recommended graphs and statistics built from required Columns and data from the DBNL Semantic Convention

• Segments Dashboard: Count graphs and statistics for all tracked Segments

• : Histograms, time series and statistics of generated

When investigating an issue, start with the time series to identify when it started, then use the histogram to understand what values are problematic, and finally check the logs page to see which specific logs exhibit the behavior.

Sankey charts show how agentic tool calls chain together in a trace:

• Nodes: Boxes representing specific tool calls (ie llm:gpt-4o-mini, tool:web_search, etc)

• Flows: Bands connecting nodes - width represents volume/quantity

• Direction: Left-to-right shows progression through tool calls

Example: A Sankey chart with many repeated nodes represents tool calls failing or needing to be retried many times, which may indicate an underlying bug in the agent or context.

Histograms show how frequently different values occur:

• X-axis: The metric value (e.g., token count, score from 1-5)

• Y-axis: Number of logs with that value

• Shape insights:

  • Normal (bell curve): Most values cluster around the average - typical, healthy distribution

Example: A token count histogram with two peaks (at 100 and 500 tokens) suggests two distinct conversation types.

Time series show how values change over time:

• X-axis: Date

• Y-axis: Metric value

• Lines: Typically shows average (mean) and P95 (95th percentile)

• Patterns to watch for:

Example: User frustration P95 suddenly spiking while mean stays flat suggests a subset of users are becoming frustrated.

Statistics give you quick numerical insights:

• Compare Max vs P95: If very different, you have extreme outliers worth investigating

• Compare Mean vs Median: If very different, your data is skewed (not normally distributed)

• Track P95 over P99: P95 is more stable and actionable for most use cases

• Use Min/Max: Identify best and worst case examples to investigate

DBNL is openly distributed and free to deploy within your cloud environment or on-premise, keeping your data safe, secure, and always under your control.

There are three options to deploy the DBNL platform as a self-hosted deployment:

• Sandbox: The DBNL sandbox deployment bundles all of the DBNL services and dependencies into a single self-contained Docker container for quick proof of concepts.

• Helm Chart: The full DBNL platform can be deployed using a Helm chart to existing infrastructure provisioned by the customer.

• : The full DBNL platform can be deployed using a Terraform module on infrastructure provisioned by the module alongside the platform. This option is supported on AWS, GCP, and Azure.

Personal Access Tokens are used for API authentication and are required for use of the .

To create a Personal Access Token click on your profile badge in the lower left of the UI, then click on "Personal Access Token." We recommend saving this as an environment variable like DBNL_API_TOKEN for future use.

The DBNL platform uses or OIDC for user authentication. OIDC providers that are known to work with DBNL include:

The dbnl CLI is installed as part of the SDK and allows for interacting with the dbnl platform from the command line.

To install the SDK, run:

The dbnl CLI.

• Options

  • --version - Show the version and exit.

The DBNL platform architecture consists of a set of packaged as Docker images and a set of standard components that are into your infrastructure (e.g. a VPC in AWS or GCP, or on-premise). The platform is scalable, modular, and self contained. It does not require an external connection to hosted Distributional services to operate.

The DBNL platform requires the following infrastructure:

• A Kubernetes cluster to host the DBNL platform services.

• A PostgreSQL database to store metadata.

Model Connections are how DBNL interfaces with LLMs, which is required for each step of the to function. It enables DBNL to

• Compute LLM-as-judge Metrics as part of the enrich step.

• Perform certain unsupervised analytics processes as part of the analysis step.

• Translate surfaced behavioral signals into human readable as part of the publish step.

The DBNL Query Language is a SQL-like language that allows for querying data in Runs for the purpose of drawing visualizations, defining metrics or evaluating tests.

Expressions

An expression is a combination of literals, values, operators, and functions. Expressions can evaluate to scalar or columnar values depending on their types and inputs. There are three types of expressions that can be composed into arbitrarily complex expressions.

Literal Expressions

Literal expressions are constant-valued expressions.

Column and scalar expressions are references to columns or scalar values in a Run. They use dot-notation to reference a column or scalar within a Run.

For example, a column named score in a Run can be referenced with the expression:

Function expressions are functions evaluated over zero or more other expressions. They make it possible to compose simple expressions into arbitrarily complex expressions.

For example, the word_count function can be used to compute the word count of the text column in a Run with the expression:

Operators are aliases for function expressions that enhance readability and ease of use. Operator precedence is the same as that of most SQL dialect.

Arithmetic operators

Arithmetic operators provide support for basic arithmetic operations.

Comparison operators

Comparison operators provide support for common comparison operations.

Logical operators

Logical operators provide support for boolean comparisons.

The DBNL Query Language follows the null semantics of most SQL dialect. With a few exception, when a null value is used as an input to a function or operator, the result is null.

One exception to this is boolean functions and operators where ternary logic is used similar to most SQL dialects.

The Helm chart option separates the infrastructure and permission provisioning process from the DBNL platform deployment process, allowing you to manage the infrastructure, permissions and Helm chart using your existing processes.

To get the Helm chart, see .

Jump straight to:

(OTEL) Trace Ingestion allows for the richest data to be uploaded to your Project, but requires some off-platform coding and does not support backfilling data. This guide provides comprehensive instructions for instrumenting your AI agent application to send OpenTelemetry (OTEL) traces to DBNL

DBNL Credentials: You'll need:

• DBNL API URL (e.g., http://localhost:8080/api)

• API Token (Bearer token for which can be generated at DBNL_API_URL/tokens

The DBNL sandbox deployment bundles all of the DBNL services and dependencies into a single self-contained Docker container. This container replicates a full DBNL deployment by creating a Kubernetes cluster in the container and using Helm to deploy the DBNL platform and its dependencies (e.g. postgresql, redis, and minio).

• Install .

• Install , the DBNL CLI and Python SDK.

Within the sandbox container, is used in conjunction with to schedule the containers for the DBNL platform and its dependencies.

The Python SDK can be used for programmatically creating projects and uploading data to them.

See for more information and examples on using the SDK to upload log data to your deployment.

To install the latest SDK, run: