FAQ
Answers to frequently asked questions
Have a question that isn't in this FAQ? Send us an note at [email protected] and we'll get back to you with an answer right away!
General
What is DBNL?
DBNL is an Adaptive Analytics platform designed to discover and track hidden behavioral signals in production AI logs and traces over time. The platform gives a detailed snapshot of aggregate AI product behavior and surfaces insights as subsets of log data corresponding to patterns in behavioral signals that can be investigated and tracked over time. This empowers AI teams to better understand the behavior of their users and AI products so that they can fix and improve those products with confidence.
What is AI Behavior?
AI Behavior refers to the patterns and characteristics of how an AI system operates in a production environment. This includes the interplay between users, context, models, and the resulting outcomes. Distributional helps you define and understand your AI's behavior by creating a "behavioral fingerprint" from your data.
What is a Behavioral Signal?
A Behavioral Signal is a key insight or pattern extracted from your AI's production data that indicates a specific behavior. These signals can highlight daily shifts, clusters of similar behaviors, or outliers that deviate from the norm. By identifying these signals, you can better understand how your AI is performing and where it can be improved.
What is a Distributional/Behavioral Fingerprint?
A Distributional/Behavioral Fingerprint is a statistical profile that represents the expected behavior of your AI application. It is derived from the distributions of historical data for each attribute of your application. This fingerprint serves as a baseline to detect any deviations or changes in your AI's behavior over time.
What is the difference between Analytics and Monitoring?
While Monitoring typically involves tracking predefined metrics and alerting you when something goes wrong, Analytics, in the context of Distributional, goes a step further. It's about deeply understanding why things are happening by analyzing complex behavioral signals and providing context-rich insights, rather than just surface-level alerts.
What is Adaptive Behavioral Analytics? / What is the Adaptive Analytics Flywheel?
Adaptive Behavioral Analytics is a method of continuously analyzing and understanding the behavior of an AI system, where the definition of "normal" behavior is constantly updated and refined as new data becomes available. The Adaptive Analytics Flywheel represents the continuous cycle of this process: analyzing data, discovering behavioral signals, investigating them with context, and using those insights to improve the AI product, which in turn generates new data for further analysis.
How does Distributional help with model drift?
Distributional helps you detect model drift by continuously monitoring the behavioral signals of your AI. When the platform detects a significant deviation from the established behavioral fingerprint, it alerts you to the change. This allows you to quickly identify and address model drift before it negatively impacts your users or business goals.
Can the platform help us perform Root Cause Analysis (RCA) when an agent fails?
Absolutely. When an issue like a hallucination or task failure is detected, our platform allows you to drill down into the specific interaction traces and user segments involved. It automatically surfaces correlated patterns and anomalies, helping you move from what happened to why it happened in minutes, not days.
How does the platform help in identifying and mitigating AI risks like bias, toxicity, or hallucinations?
Our platform provides specialized LLM-as-Judge Metric Templates for AI safety and responsibility that you can further customize. You can define policies to automatically flag toxic language, measure demographic bias in agent responses, and track the frequency of model hallucinations, providing the critical insights needed to build safer and more trustworthy AI.
What kind of AI applications can I use Distributional with?
Distributional is designed to work with a wide variety of AI applications or agents, including those built with Large Language Models (LLMs), recommendation systems, fraud detection models, and more. Its flexible data ingestion and analysis capabilities make it adaptable to virtually any AI product that generates log data.
Do I need to be a data scientist to use Distributional?
While data scientists will find the platform's advanced analytical capabilities powerful, Distributional is designed to be accessible to a broader audience, including product managers and engineers. The platform translates complex data analysis into human-readable insights, making it easier for entire teams to understand and improve their AI products.
What are the data requirements to get started, and what formats are supported?
Getting started is simple. The platform primarily requires your production logs, which contain the interactions with your AI agent. We support structured data formats like JSON and Parquet, and our flexible ingestion APIs and SDKs make it easy to send data directly from your application or existing data infrastructure like Snowflake, Databricks, or S3.
How much data do I need?
The behavioral analytics that DBNL provides is most helpful when you have 1000s of logs or traces per day and can scale to many hundreds of thousands of logs or traces per day.
After a week of data has been collected the full DBNL Data Pipeline will run each day including topic modeling and other analytics that require a baseline.
Company
What is the pricing for DBNL? Why?
Distributional offers a free open-source version of their platform that you can deploy locally or in a Kubernetes cluster. For enterprise needs, they provide custom pricing. This approach allows for broad accessibility with the open-source option, while the enterprise plan provides dedicated support, enhanced security, and scalability for larger organizations. Contact us if you would like to learn more about our enterprise options or to join as a co-build partner.
How can I contact you for support?
You can contact us via our webform or by directly emailing [email protected]. If you are an enterprise willing to join us as a co-build partner we offer dedicated Slack channels and direct support options.
Metrics
What is the difference between performance and behavioral metrics?
Performance metrics typically measure the efficiency and effectiveness of a system in achieving a specific goal, such as accuracy, speed, or conversion rates. Behavioral metrics, on the other hand, focus on how the system and its users behave, capturing nuanced interactions and patterns that go beyond simple success or failure, like user engagement, error patterns, or unexpected model responses.
Can I bring my own metrics?
Yes, you can absolutely bring your own metrics. Distributional is designed to be extensible and allows you to integrate your own evaluation functions and metrics seamlessly into the platform. This flexibility ensures that you can tailor the analysis to the specific needs of your AI application.
What LLMs and providers do you support for LLM-as-judge metrics?
Our extensible Model Connections support externally managed APIs (OpenAI, together.ai), cloud-managed services (Bedrock, Vertex, Azure OpenAI, Gemini), and local clusters (NVIDIA NIMs).
Deployment
How is Distributional deployed?
DBNL is openly distributed and free to deploy in within your cloud environment or on-premise, keeping your data safe, secure, and always under your control. There are a variety of options for deploying DBNL within a Sandbox, Terraform Module, or Helm Chart. Learn more about these options and their tradeoffs in the Deployment documentation.
How much engineering effort is required for initial setup and ongoing maintenance?
The initial setup is designed to be lightweight, often taking less than an hour with our provided Python SDK, Sandbox deployment and Quickstart.
Is my data secure?
Yes, your data is secure with Distributional. The platform is designed with enterprise-grade security features, including Authentication, Administration, and robust Networking controls. When self-hosting, your data remains within your own environment, giving you full control over its security. Learn more in the Data Security documentation.
Was this helpful?