Testing

Tests are the key tool within dbnl for asserting performance and consistency of runs. Possible goals during testing can include:

• Asserting that a chosen column meets its minimum desired behavior (e.g., inference throughput);

• Asserting that a chosen column has a distribution that roughly matches the baseline reference;

• Asserting that no individual results have a severely divergent behavior from a baseline.

In this section, we explore the objects required for testing, methods for creating tests, suggested testing strategies, reviewing/analyzing tests, and best practices.