LLM Text Summarization

In this advanced tutorial, we demonstrate how to use dbnl to automatically evaluate the consistency of summarization output on a fixed set of documents.

The data files required for this tutorial are available in the following files.

9MB

summarization-2025-02-05-18-01-59.zip

archive

Open

Summarization Tutorial files

8MB

prompt-engineering-2024-11-18-18-01-59.zip

archive

Open

Prompt Engineering files

Applications

While summarization is the focus of this tutorial, the same principles can be applied to any task involving text generation. The goal is to evaluate the consistency of the generated text with the input text. Other tasks involving text generation are entity recognition, question answering, and machine translation.

Prerequisites

This tutorial assumes that you have already the following tutorials: Hello World (Sentiment Classifier) and ideally Trading Strategy.

Organization

This tutorial requires a good deal of preparation, so it has been divided into the following four sections:

• Defining the text summarization problem of interest, including the data source and the metrics,

• Creating a constrained optimization problem to govern the development of a text summarization app in dbnl,

• Managing the integration testing process for consistent testing after such an app has been created,

• Practical considerations which would arise when actually building an LLM summarization tool.