Defining Assertions

Assertions are the second half of test creation — defining what statistical values seem appropriate or aberrant

A test consists of statistical quantities that define the behavior of one or more runs and an associated threshold which determines whether those statistics define acceptable behavior. Essentially, a test passes or fails based on whether the quantities satisfy the thresholds. Common assertions include:

• equal_to – Used most often to confirm that two distributions are exactly identical

• close_to – Can be used to confirm that two quantities are near each other, e.g., that the 90th

• less_than – Can be used to confirm that nonparametric statistics are small enough to indicate that two distributions are suitably similar

A full list of assertions can be found at the test creation page.

Choosing a Threshold

Choosing a threshold can be simple, or it can be complicated. When the statistic under analysis has easily interpreted values, these thresholds can emerge naturally.

• Example: The 90th percentile of the app latency must be less than 123ms.

  • Here, the statistic is the percentile(EXPERIMENT.latency, 90), and the threshold is “less than 123”.

• Example: No result should have more than a 0.098 difference in probability of fraud prediction against the baseline.

  • Here, the statistic is max(matched_abs_diff(EXPERIMENT.prob_fraud, BASELINE.prob_fraud)) and the threshold is “less than 0.098”.

In situations where nonparametric quantities, like the scaled Kolmogorov-Smirnov statistic, are used to state whether two distributions are sufficiently similar, it can be difficult to identify the appropriate threshold. This is something that may require some trial and error, as well as some guidance from your dedicated Applied Customer Engineer. Please reach out as you would like our support developing thresholds for these statistics.

Research into Adaptive Testing Strategies

Additionally, we are hard at work developing new strategies for adaptively learning thresholds for our customers based on their preferences. If you would like to be a part of this development and beta testing process, please inform your Applied Customer Engineer.