Insights surfaced elsewhere on Distributional

Test Sessions are not the only place to learn about your app

Distributional’s goal is to provide you the ability to ask and answer the question “Is my AI-powered app behaving as expected?” While testing is a key component of this, triage of a failed test and inspiration for new tests can come from many places in our web UI. Here, we show insights into app behavior which are uncovered with Distributional.

Run Detail page

Because each Run represents the recent behavior of your app, the Run Detail page is a useful source of insights about your app’s behavior. In the screenshot below, you can see:

• dbnl-generated alerts regarding highly correlated columns (in depth on a separate screen),

• Summary statistics for columns (along with shortcuts to create tests for any statistics of note), and

• Notable behavior for columns, such as a skewed or multimodal distribution.

At the Run Detail page, you can gain quick insights regarding your app's behavior and add tests on those insights as you desire.

Compare page

At the top of the Run Detail page, there are links to the Compare and Analyze pages, where you can conduct more in depth and customized analysis. You can drive your own analysis at these pages to uncover key insights about your app.

For example, after seeing a failed Test Session in a RAG (Q & A) application, you may visit the Compare page to understand the impact of adding new documents to your vector database. The image below shows a sample Compare page, which reveals a sizable decrease in the population of poorly-retrieved questions (drop in the low bleu value between Baseline and Experiment).

The Compare Page exposes a significant drop in low-performing bleu scores when new documents are added to the vector database.

Filtering for those columns (the screenshot below) gives a valuable insight about the impact of the extra documents. You see that, previously, the RAG app was incorrectly retrieving documents from “Liabilities and Contingencies” as well as “Asset Valuations.” Adding the new documents improved your app’s quality, and now you can confidently answer, “Yes, my app’s behavior has changed, and I am satisfied with its new behavior.”

Applying a filter for low bleu score values allows you to identify which documents are being better retrieved with the extra 100 documents in the vector database.