Comprehensive testing with Distributional

This is how you test when you are a dbnl expert

When you start work with Distributional (dbnl), you should focus on creating and executing Production tests to ask and answer the question "Is my AI-powered app behaving as desired?" As you gain more confidence using dbnl, the full pattern of standard dbnl usage looks as follows:

1. Execute Production testing at a regular interval (e.g., nightly) on recent app usages

2. Execute Deployment testing at a regular interval (e.g., weekly) on a fixed dataset

3. Review Production test sessions to start triage of any concerning app behavior

   1. This could be triggered by test failures, DBNL-automated guidance, or manual investigation

4. As concerning app behavior is identified, trigger Deployment tests to help diagnose any app component nonstationarity

   1. If nonstationarity is present, start Development testing on the affected 3rd party components

   2. If the components are stationary, start Development testing on the observed app examples which show unsatisfactory behavior

5. Once suitable app behavior is recorded in Development testing, record a new baseline for future Deployment testing on the fixed dataset

6. Push the new app version into production and update the Deployment + Production testing process