DBNL turns raw trace data into actionable insights to continuously improve your agents
DBNL turns raw trace data into actionable insights to continuously improve your agents
DBNL Accelerates the AI Data Flywheel by pinpointing behavioral signals and relevant production log data that can be used to optimize the underlying agent.
Dive into the product right away in the Quickstart or Tutorials. The above is part of the Google ADK Calculator Tutorial.
Regardless of Ingestion method, the DBNL Data Pipeline ensures that all data is mapped to identical results tables and is treated the same for the purposes of the Analytics Workflow.
Data Connections are how Production AI log data is ingested into DBNL, kicking off the Analytics Workflow.
Externally managed service
(together.ai, OpenAI, etc)
Cloud managed service
(Bedrock, Vertex, etc)
Locally managed deployment
(NVIDIA NIMs in k8s cluster)
A Project is initialized with a Data Connection for log ingestion, a Model Connection for the analysis pipeline, and optional Notification Connections for reporting and alerting.
View all Projects in a Namespace from the Namespace landing page.
Navigate to a new Project or Namespace from the breadcrumb dropdown menu at the top of all project pages.
Default dashboard displaying recommended graphs and statistics for a specific time window (default: last 7 days)
Dashboard displaying all tracked [Segments](segments.md) as time series of daily counts for each [Segment](segments.md) within a specific time range (default: last 7 days)
Dashboard displaying all custom [Metrics](metrics.md) as histograms, time series, and statistics summaries for all logs within a specific time range (default: last 7 days)
The Single Segment Explorer page allows for quickly viewing all metrics for a filter within a time window.
The Segment Comparison Explorer page allows for quick comparison of two filters across a single time window.
The Temporal Comparison Explorer page allows for quick comparison of a single filter across two adjacent time windows.
topicllm_answer_groundednessllm_answer_refusalllm_answer_relevancyllm_context_relevancyllm_question_clarityllm_summarizationllm_text_frustrationllm_text_sentimentllm_text_similarityllm_text_toxicity
Quick, self contained proof of concept deployments
Fully customizable deployments within current infrastructure
Independent, full deployments in AWS, GCP, or Azure VPCs
auth:
# For more details on OIDC options, see OIDC Authentication section.
oidc:
enabled: true
issuer: oidc.example.com
audience: xxxxxxxx
clientId: xxxxxxxx
scopes: "openid email profile"
db:
host: db.example.com
port: 5432
username: user
password: password
database: database
redis:
host: redis.example.com
port: 6379
username: user
password: password
ingress:
enabled: true
api:
host: dbnl.example.com
ui:
host: dbnl.example.com
storage:
s3:
enabled: true
region: us-east-1
bucket: example-bucket
DBNL platform architecture
Data upload
Literal expression
Column expression
Function expression
Operators
| During Development | During Deployment | During Production |
|---|---|---|
|
|
|
Welcome to Distributional
Comparison of traditional software vs. AI workflows
AI SDLC
AI Reliability Stack: AI Testing spans the AI SDLC
Bimodal vs Normal distribution of Latency
The expected flow of app information to Distributional for run creation within a project
From this page, you can ask Distributional to create Tests and query Runs to define a baseline.
Select a Run from which to create Production tests,
and select columns that should be tested.
Oversee the behavior of your app on your tests from your Project's home page.
Menu of common testing strategies that are supported by templates in dbnl.
You can filter to only the failed tests to better understand how the app is violating test expectations and whether the tests should be recalibrated to pass in the future.
After subselecting tests, or selecting the full Test Session, dbnl provides a list of Notable Results that demonstrate the largest devioation from previously observed behavior.
At the Run Detail page, you can gain quick insights regarding your app's behavior and add tests on those insights as you desire.
The Compare Page exposes a significant drop in low-performing bleu scores when new documents are added to the vector database.
Applying a filter for low bleu score values allows you to identify which documents are being better retrieved with the extra 100 documents in the vector database.
Customize your notifications at the Project Detail page.
Runs in dbnl are created from data produced during the normal operation of your app, such as prompts (inputs) and responses (outputs).
This DAG suggests that failed tests in the TradeRecommender are likely attributed to the SentimentClassifier.
| Variable Name | Description |
|---|---|
DBNL_API_TOKEN | The API token used to authenticate your dbnl account. You can generate your API token at https://app.dbnl.com/tokens |
DBNL_API_URL | The base url of the Distributional API. For SaaS users, set this variable to api.dbnl.com. For other users, please contact your sys admin. |
DBNL_APP_URL | An optional base url of the Distributional app. If this variable is not set, the app url is inferred from the DBNL_API_URL variable. For on-prem users, please contact your sys admin if you cannot reach the Distributional UI. |
| Organization and Namespaces | Organization and Namespaces | |
| Users and Permissions | Users and Permissions | |
| Tokens | Tokens |
Writing a test for minimum performance using the Distributional app.
Writing a regression test using the distributional app.
Scalar data viewable from the Run page on the Distributional app.
Data upload
Creating a Test
The full test creation page, with the modal on the left side and helping graphs on the right.
Here, each entry in the summary statistics table (on the Run Detail page) can be used to seed creation of a test for that chosen statistic.
If an interesting statistical property is present for a given distribution, you may be alerted to that fact above the mini charts on the Run Detail page.
Expanding out this alert will generate a possible test of interest which can then be edited and saved at the test creation drawer.
When two distributions are being studied at the compare page, a suggested nonparametric statistic will be presented to guide potential test creation for asserting consistency of that distribution between runs.
Dropdown selector
Test Creation Drawer
Auto-Generate Tests Modal
Recalibration modal in Test Configuration page
{
"name": "p95_app_latency_ms",
"description": "Test the 95th percentile of latency in miliseconds",
"statistic_name": "percentile",
"statistic_params": {"percentage": 0.95},
"assertion": {
"name": "less_than_or_equal_to",
"params": {
"other": 180.0,
},
},
"statistic_inputs": [
{
"select_query_template": {
"select": "{EXPERIMENT}.app_latency_ms"
}
},
],
}
Example Test on 95th percentile of app_latency_ms
Example test on absolute difference of medians of positive_sentiment_score
Example test on discrepancy of distribution of coherence_score
Example Test on mean of absolute difference of toxicity_score
Example Test on signed difference of mean of toxicity_score
From the project details page, click configure tests.
Select the baseline run in the dropdown against which you would like new experiment runs to be compared.
A project detail page with two completed test sessions — the circled test session had two failed assertions.
| function name | aliases | description |
|---|---|---|
| and | Logical AND operation of two or more boolean columns | |
| or | Logical OR operation of two or more boolean columns | |
| not | Logical NOT operation of a boolean column | |
| less_than | ['lt'] | Computes the element-wise less than comparison of two columns. input1 < input2 |
| less_than_or_equal_to | ['lte'] | Computes the element-wise less than or equal to comparison of two columns. input1 <= input2 |
| greater_than | ['gt'] | Computes the element-wise greater than comparison of two columns. input1 > input2 |
| greater_than_or_equal_to | ['gte'] | Computes the element-wise greater than or equal to comparison of two columns. input1 >= input2 |
| equal_to | ['eq'] | Computes the element-wise greater than or equal to comparison of two columns |
