dbnl.eval

create_run_schema_from_results

dbnl.eval.create_run_schema_from_results(*, column_data: DataFrame, scalar_data: dict[str, Any] | DataFrame | None = None, index: list[str] | None = None, metrics: Sequence[[Metric](dbnl.eval.metrics.md#dbnl.eval.metrics.Metric)] | None = None) → RunSchema

Create a new RunSchema from column results, scalar results, and metrics.

This function assumes that the metrics have already been evaluated on the original, un-augmented data. In other words, the column data for the metrics should also be present in the column_data.

• Parameters:

  • column_data – DataFrame with the results for the columns

  • scalar_data – Dictionary or DataFrame with the results for the scalars, defaults to None

  • index – List of column names that are the unique identifier, defaults to None

  • metrics – List of metrics to report with the run, defaults to None

• Raises:DBNLInputValidationError – Input does not conform to expected format

• Returns: RunSchema with the desired schema for columns and scalars, if provided

evaluate

dbnl.eval.evaluate(*, df: DataFrame, metrics: Sequence[[Metric](dbnl.eval.metrics.md#dbnl.eval.metrics.Metric)], inplace: bool = False) → DataFrame

Evaluates a set of metrics on a dataframe, returning an augmented dataframe.

• Parameters:

  • df – input dataframe

  • metrics – metrics to compute

  • inplace – whether to modify the input dataframe in place

• Returns: input dataframe augmented with metrics

get_column_schemas_from_dataframe_and_metrics

dbnl.eval.get_column_schemas_from_dataframe_and_metrics(*, df: DataFrame, metrics: list[[Metric](dbnl.eval.metrics.md#dbnl.eval.metrics.Metric)]) → list[RunSchemaColumnSchemaDict]

Gets the run schema column schemas for a dataframe that was augmented with a list of metrics.

• Parameters:

  • df – Dataframe to get column schemas from

  • metrics – list of metrics added to the dataframe

• Returns: list of columns schemas for dataframe and metrics

get_column_schemas_from_metrics

dbnl.eval.get_column_schemas_from_metrics(metrics: list[[Metric](dbnl.eval.metrics.md#dbnl.eval.metrics.Metric)]) → list[RunSchemaColumnSchemaDict]

Gets the run schema column schemas from a list of metrics.

• Parameters:metrics – list of metrics to get column schemas from

• Returns: list of column schemas for metrics

get_run_schema_columns_from_metrics

dbnl.eval.get_run_schema_columns_from_metrics(metrics: list[[Metric](dbnl.eval.metrics.md#dbnl.eval.metrics.Metric)]) → list[RunSchemaColumnSchema]

Gets the run schema column schemas from a list of metrics.

• Parameters:metrics – list of metrics to get column schemas from

• Returns: list of column schemas for metrics

report_run_with_results

dbnl.eval.report_run_with_results(*, project: Project, column_data: DataFrame, scalar_data: dict[str, Any] | DataFrame | None = None, display_name: str | None = None, index: list[str] | None = None, run_schema: RunSchema | None = None, metadata: dict[str, str] | None = None, metrics: Sequence[[Metric](dbnl.eval.metrics.md#dbnl.eval.metrics.Metric)] | None = None, wait_for_close: bool = True) → Run

Create a new Run, report results to it, and close it.

If run_schema is not provided, a RunSchema will be created from the data. If a run_schema is provided, the results are validated against it.

If metrics are provided, they are evaluated on the column data before reporting.

• Parameters:

  • project – DBNL Project to create the Run for

  • column_data – DataFrame with the results for the columns

  • scalar_data – Dictionary or DataFrame with the results for the scalars, if any. Defaults to None

  • display_name – Display name for the Run, defaults to None.

  • index – List of column names that are the unique identifier, defaults to None. Only used when creating a new schema.

  • run_schema – RunSchema to use for the Run, defaults to None.

  • metadata – Additional key:value pairs user wants to track, defaults to None

  • metrics – List of metrics to report with the run, defaults to None

  • wait_for_close – If True, the function will block for up to 3 minutes until the Run is closed, defaults to True

• Raises:

  • DBNLNotLoggedInError – DBNL SDK is not logged in

  • DBNLInputValidationError – Input does not conform to expected format

• Returns: Run, after reporting results and closing it

report_run_with_results_and_start_test_session

dbnl.eval.report_run_with_results_and_start_test_session(*, project: Project, column_data: DataFrame, scalar_data: dict[str, Any] | DataFrame | None = None, display_name: str | None = None, index: list[str] | None = None, run_schema: RunSchema | None = None, metadata: dict[str, str] | None = None, baseline: Run | RunQuery | None = None, include_tags: list[str] | None = None, exclude_tags: list[str] | None = None, require_tags: list[str] | None = None, metrics: Sequence[[Metric](dbnl.eval.metrics.md#dbnl.eval.metrics.Metric)] | None = None) → Run

Create a new Run, report results to it, and close it. Start a TestSession with the given inputs. If metrics are provided, they are evaluated on the column data before reporting.

• Parameters:

  • project – DBNL Project to create the Run for

  • column_data – DataFrame with the results for the columns

  • scalar_data – Dictionary or DataFrame with the scalar results to report to DBNL, defaults to None.

  • display_name – Display name for the Run, defaults to None.

  • index – List of column names that are the unique identifier, defaults to None. Only used when creating a new schema.

  • run_schema – RunSchema to use for the Run, defaults to None.

  • metadata – Additional key:value pairs user wants to track, defaults to None

  • baseline – DBNL Run or RunQuery to use as the baseline run, defaults to None. If None, the baseline defined in the TestConfig is used.

  • include_tags – List of Test Tag names to include in the Test Session

  • exclude_tags – List of Test Tag names to exclude in the Test Session

  • require_tags – List of Test Tag names to require in the Test Session

  • metrics – List of metrics to report with the run, defaults to None

• Raises:

  • DBNLNotLoggedInError – DBNL SDK is not logged in

  • DBNLInputValidationError – Input does not conform to expected format

• Returns: Run, after reporting results and closing it