The Flow of Data

Your data + DBNL testing == insights about your app's behavior

Distributional uses data generated by your AI-powered app to study its behavior and alert you to valuable insights or worrisome trends. The diagram below gives a quick summary of this process:

• Each app usage involves input(s), the resulting output(s), and context about that usage

  • Example: Input is a question from a user; Output is your app’s answer to that question; Context is the time/day that the question was asked.

• As the app is used, you record and store the usage in a data store for later review

  • Example: At 2am every morning, an Airflow job parses all of the previous day’s app usages and sends that info to a data store.

• When data is moved to your data store, it is also submitted to DBNL for testing.

  • Example: The 2am Airflow job is amended to include data augmentation by DBNL Eval and uploading of the resulting Run to trigger automatic app testing.

Runs in DBNL are created from data produced during the normal operation of your app, such as prompts (inputs) and responses (outputs).

You can read more about the DBNL specific terms earlier in the documentation. Simply stated, a Run contains all of the data which DBNL will use to test the behavior of your app – insights about your app’s behavior will be derived from this data.

A Run usually contains many (e.g., dozens or hundreds) rows of inputs + outputs + context, where each row was generated by an app usage. Our insights are statistically derived from the distributions estimated by these rows.

DBNL Eval is our library that provides access to common, well-tested GenAI evaluation metrics. You can use DBNL Eval to augment data in your app, such as the inputs and outputs. You are also able to bring your own eval metrics and use them in conjunction with DBNL Eval or standalone. Doing so produces a broader range of tests that can be run, and it allows the platform to produce more powerful insights.