Matched Results Between Runs

Special tests and comparisons are possible when the user provides a unique identifier to “match” result rows run over run. Users can write tests that compare stats on result columns considering only pairs of rows that have the same row_id.

To use this functionality, one or more row_id column names must be specified in the RunConfig. The results dataframe must contain those column(s) with and the row_ids must be unique within a single run's results.

To get started using this, the simplest strategy is to number each row 1, 2, 3, .... Those assignments must then be consistently applied to those results for all runs.