> For the complete documentation index, see [llms.txt](https://docs.dbnl.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.dbnl.com/v0.24.x/using-distributional/metrics/llm-models.md).

# LLM Models

Distributional can call a third-party or self-hosted LLM on your behalf, with your credentials, to generate results or create metrics based on text data inputs.

DBNL supports the following LLM providers:

* OpenAI
* Azure OpenAI
* AWS Bedrock
* AWS SageMaker
* Google Gemini
* Google VertexAI

### Providers

{% tabs %}
{% tab title="OpenAI" %}
The following arguments are supported for the OpenAI provider.

{% hint style="info" %}
Any OpenAI-compatible endpoint (such as a self-hosted LLM in your environment) can be accessed this way.
{% endhint %}

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>api_key</code></td><td>API key to access OpenAI</td></tr><tr><td><code>base_url</code></td><td><p>[Optional] Location of where to access LLMs.</p><p>If not provided, defaults to call OpenAI directly.<br>If provided, should be a valid URL that points to an OpenaAI-compatible provider, but not a specific model. For example: </p><pre><code>http://my.test.server.example.com:8083/v1
</code></pre></td></tr></tbody></table>

{% endtab %}

{% tab title="Azure OpenAI" %}
The following arguments are supported for the Azure OpenAI provider.

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>api_key</code></td><td>API key to access an Azure-hosted OpenAI-compatible model</td></tr><tr><td><code>azure_endpoint</code></td><td><p>Location of where models are hosted.<br>For example: </p><pre><code>http://my-model.openai.azure.com/
</code></pre></td></tr><tr><td><code>api_version</code></td><td>Version of Azure OpenAI REST API to use.<br>As of 2025-07-01, Distributional uses only the "Data plan - inference" REST API. See <a href="/pages/WoWHy90YJKkSzVkMev8R">official Azure OpenAI documentation</a> for further details.</td></tr></tbody></table>

{% endtab %}

{% tab title="AWS Bedrock" %}
The following arguments are supported for the AWS Bedrock provider.

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>aws_access_key_id</code></td><td>Must start with <code>AKIA</code>.<br>Note: session keys are not supported as they expire too quickly and Distributional expects long-lived keys, as the LLM will be called on all future Runs.</td></tr><tr><td><code>aws_secret_access_key</code></td><td>AWS Secret Key</td></tr><tr><td><code>aws_region</code></td><td>[Optional] AWS Region where the AWS Bedrock Model is available</td></tr><tr><td><code>aws_bedrock_runtime_endpoint</code></td><td>[Optional] Endpoint to interact with AWS Bedrock Runtime API.<br>Example: <code>bedrock-runtime.us-east-1.amazonaws.com</code>.</td></tr></tbody></table>

{% endtab %}

{% tab title="AWS SageMaker" %}
The following arguments are supported for the OpenAI provider.

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>aws_access_key_id</code></td><td>Must start with <code>AKIA</code>. Session keys are not supported as they expire too quickly and Distributional expects long-lived keys, as the LLM will be called on all future Runs.</td></tr><tr><td><code>aws_secret_access_key</code></td><td>AWS Secret Key</td></tr><tr><td><code>aws_region</code></td><td>AWS Region where the AWS SageMaker Endpoint is available</td></tr><tr><td><code>endpoint_name</code></td><td>[Optional] Name of AWS Endpoint that hosts the Model.<br>Can be left blank if the endpoint can be derived from the Model alone.</td></tr></tbody></table>
{% endtab %}

{% tab title="Google Gemini" %}
The following arguments are supported for the Google Gemini provider.

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>api_key</code></td><td>API key to access Google Gemini API</td></tr><tr><td><code>base_url</code></td><td><p>[Optional] Gemini API Base URL.<br>If provided, should be a valid URL that points to a provider, but not a specific model. For example: </p><pre><code>http://my.test.server.example.com:8083/v1
</code></pre></td></tr></tbody></table>

{% endtab %}

{% tab title="Google VertexAI" %}
The following arguments are supported for the OpenAI provider.

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>google_application_credentials_json</code></td><td>JSON string containing credentials for a service account.</td></tr><tr><td><code>region</code></td><td>[Optional] GCP Region for Vertex AI.</td></tr><tr><td><code>gcp_project_id</code></td><td>[Optional] GCP Project ID for Vertex AI.<br>If not provided, Distributional will try to infer from the provided credentials.</td></tr><tr><td><code>vertex_ai_endpoint</code></td><td>[Optional] Vertex AI Endpoint override, if desired.</td></tr></tbody></table>

{% endtab %}
{% endtabs %}

### Models

After setting your LLM Provider details, you must set the `model` you want Distributional to access on your behalf. Each provider has their own set of models that may be available. Distributional provides a *Validate Model* button to enable simple testing to confirm your model has been set up correctly.

{% tabs %}
{% tab title="OpenAI" %}
See [OpenAI's Model documentation](https://platform.openai.com/docs/models) to learn more about their available models.

{% hint style="info" %}
Model names must be provided exactly as the Azure OpenAI Models REST API requires.

For example, the [GPT-4.1 Model](https://platform.openai.com/docs/models/gpt-4.1) lists the appropriately-formatted names. Validation would fail for "GPT-4.1" but succeed for `gpt-4.1` or `gpt-4.1-2025-04-14`.<br>
{% endhint %}

<figure><img src="/files/gD3rnSSZE0rRD75PeGzx" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="Azure OpenAI" %}
See [Azure AI Foundry's documentation](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/models?tabs=global-standard%2Cstandard-chat-completions) to learn more about their available models.

{% hint style="info" %}
Model names must be provided exactly as the Azure OpenAI Models REST API requires.

For example, the [GPT-4.1 series](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/models?tabs=global-standard%2Cstandard-chat-completions#gpt-41-series) lists the appropriately-formatted model names. Validation would fail for "GPT-4.1" but succeed for `gpt-4.1` or `gpt-4.1-nano`.
{% endhint %}

<figure><img src="/files/kKEWenpWdN8unn7iRyQM" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="AWS Bedrock" %}
See [AWS Bedrock's Model documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html) to learn more about their available models.

{% hint style="info" %}
Models must be provided exactly as the AWS Bedrock API requires.

For example, the table at the above documentation link lists the appropriately-formatted Model IDs. Validation would fail for "Nova Lite" but succeed for `amazon.nova-lite-v1:0`.<br>
{% endhint %}

<figure><img src="/files/WijP0lQ3Pl3kPKoS65ur" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="AWS SageMaker" %}
See [AWS SageMaker's Model documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-latest.html) to learn more about their available models.

{% hint style="info" %}
Models must be provided exactly as the AWS SageMaker AI Studio API requires.

For example, picking a model from [AWS SageMaker JumpStart](https://aws.amazon.com/sagemaker-ai/jumpstart/getting-started/?sagemaker-jumpstart-cards.sort-by=item.additionalFields.priority\&sagemaker-jumpstart-cards.sort-order=asc\&awsf.sagemaker-jumpstart-filter-product-type=*all\&awsf.sagemaker-jumpstart-filter-text=*all\&awsf.sagemaker-jumpstart-filter-vision=*all\&awsf.sagemaker-jumpstart-filter-tabular=*all\&awsf.sagemaker-jumpstart-filter-audio-tasks=*all\&awsf.sagemaker-jumpstart-filter-multimodal=*all\&awsf.sagemaker-jumpstart-filter-RL=*all), the model name "Falcon 7B Instruct BF16" will fail validation, but it will succeed with `huggingface-llm-falcon-7b-instruct-bf16`.<br>
{% endhint %}

<figure><img src="/files/WijP0lQ3Pl3kPKoS65ur" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="Google Gemini" %}
See [Google Gemini's documentation](https://ai.google.dev/gemini-api/docs/models) to learn more about their available models.

{% hint style="info" %}
Models must be provided exactly as the Google Gemini API requires.

For example,  [Google Gemini 2.5 Pro](https://ai.google.dev/gemini-api/docs/models#gemini-2.5-pro), lists the appropriately-formatted model codes. Validation would fail for "Gemini 2.5 Pro", but succeed with `gemini-2.5-pro` or `gemini-2.5-pro-preview-06-05`.<br>
{% endhint %}

<figure><img src="/files/YY9TAKRNBKZWq5NaBp6x" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="Google VertexAI" %}
See [Google Vertex AI's documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/models) to learn more about their available models.

{% hint style="info" %}
Models must be provided exactly as the Google Gemini API requires.

For example,  [Google Gemini 2.5 Pro](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro), lists the appropriately-formatted model IDs. Validation would fail for "Gemini 2.5 Pro", but succeed with `gemini-2.5-pro` or `gemini-2.5-pro-preview-06-05`.<br>
{% endhint %}

<figure><img src="/files/G0R5HKqtiSdiplEJ7Rsk" alt=""><figcaption></figcaption></figure>
{% endtab %}
{% endtabs %}