Executing Tests Via SDK

Tests can also be executed via the SDK after results data has been reported. This requires the following steps.

Report an initial run and set that as a baseline

From the project details page, click configure tests.

Select the baseline run in the dropdown against which you would like new experiment runs to be compared.

Setting baseline in SDK

Alternatively, you can also set a Run as baseline using the set_run_as_baseline function.

Close a new run in that project

Executing the close_run command for a new run in that same project will finalize the data, enabling it for use in Tests.

Create Test Session

Execute the create_test_session command, providing your new run as the "experiment". Tests will then use the previously-defined baseline for comparisons.