# LLM Models

Distributional can call a third-party or self-hosted LLM on your behalf, with your credentials, to generate results or create metrics based on text data inputs.

DBNL supports the following LLM providers:

* OpenAI
* Azure OpenAI
* AWS Bedrock
* AWS SageMaker
* Google Gemini
* Google VertexAI

### Providers

{% tabs %}
{% tab title="OpenAI" %}
The following arguments are supported for the OpenAI provider.

{% hint style="info" %}
Any OpenAI-compatible endpoint (such as a self-hosted LLM in your environment) can be accessed this way.
{% endhint %}

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>api_key</code></td><td>API key to access OpenAI</td></tr><tr><td><code>base_url</code></td><td><p>[Optional] Location of where to access LLMs.</p><p>If not provided, defaults to call OpenAI directly.<br>If provided, should be a valid URL that points to an OpenaAI-compatible provider, but not a specific model. For example: </p><pre><code>http://my.test.server.example.com:8083/v1
</code></pre></td></tr></tbody></table>

{% endtab %}

{% tab title="Azure OpenAI" %}
The following arguments are supported for the Azure OpenAI provider.

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>api_key</code></td><td>API key to access an Azure-hosted OpenAI-compatible model</td></tr><tr><td><code>azure_endpoint</code></td><td><p>Location of where models are hosted.<br>For example: </p><pre><code>http://my-model.openai.azure.com/
</code></pre></td></tr><tr><td><code>api_version</code></td><td>Version of Azure OpenAI REST API to use.<br>As of 2025-07-01, Distributional uses only the "Data plan - inference" REST API. See <a href="/pages/WoWHy90YJKkSzVkMev8R">official Azure OpenAI documentation</a> for further details.</td></tr></tbody></table>

{% endtab %}

{% tab title="AWS Bedrock" %}
The following arguments are supported for the AWS Bedrock provider.

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>aws_access_key_id</code></td><td>Must start with <code>AKIA</code>.<br>Note: session keys are not supported as they expire too quickly and Distributional expects long-lived keys, as the LLM will be called on all future Runs.</td></tr><tr><td><code>aws_secret_access_key</code></td><td>AWS Secret Key</td></tr><tr><td><code>aws_region</code></td><td>[Optional] AWS Region where the AWS Bedrock Model is available</td></tr><tr><td><code>aws_bedrock_runtime_endpoint</code></td><td>[Optional] Endpoint to interact with AWS Bedrock Runtime API.<br>Example: <code>bedrock-runtime.us-east-1.amazonaws.com</code>.</td></tr></tbody></table>

{% endtab %}

{% tab title="AWS SageMaker" %}
The following arguments are supported for the OpenAI provider.

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>aws_access_key_id</code></td><td>Must start with <code>AKIA</code>. Session keys are not supported as they expire too quickly and Distributional expects long-lived keys, as the LLM will be called on all future Runs.</td></tr><tr><td><code>aws_secret_access_key</code></td><td>AWS Secret Key</td></tr><tr><td><code>aws_region</code></td><td>AWS Region where the AWS SageMaker Endpoint is available</td></tr><tr><td><code>endpoint_name</code></td><td>[Optional] Name of AWS Endpoint that hosts the Model.<br>Can be left blank if the endpoint can be derived from the Model alone.</td></tr></tbody></table>
{% endtab %}

{% tab title="Google Gemini" %}
The following arguments are supported for the Google Gemini provider.

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>api_key</code></td><td>API key to access Google Gemini API</td></tr><tr><td><code>base_url</code></td><td><p>[Optional] Gemini API Base URL.<br>If provided, should be a valid URL that points to a provider, but not a specific model. For example: </p><pre><code>http://my.test.server.example.com:8083/v1
</code></pre></td></tr></tbody></table>

{% endtab %}

{% tab title="Google VertexAI" %}
The following arguments are supported for the OpenAI provider.

<table><thead><tr><th width="225.10546875">Argument</th><th>Description</th></tr></thead><tbody><tr><td><code>google_application_credentials_json</code></td><td>JSON string containing credentials for a service account.</td></tr><tr><td><code>region</code></td><td>[Optional] GCP Region for Vertex AI.</td></tr><tr><td><code>gcp_project_id</code></td><td>[Optional] GCP Project ID for Vertex AI.<br>If not provided, Distributional will try to infer from the provided credentials.</td></tr><tr><td><code>vertex_ai_endpoint</code></td><td>[Optional] Vertex AI Endpoint override, if desired.</td></tr></tbody></table>

{% endtab %}
{% endtabs %}

### Models

After setting your LLM Provider details, you must set the `model` you want Distributional to access on your behalf. Each provider has their own set of models that may be available. Distributional provides a *Validate Model* button to enable simple testing to confirm your model has been set up correctly.

{% tabs %}
{% tab title="OpenAI" %}
See [OpenAI's Model documentation](https://platform.openai.com/docs/models) to learn more about their available models.

{% hint style="info" %}
Model names must be provided exactly as the Azure OpenAI Models REST API requires.

For example, the [GPT-4.1 Model](https://platform.openai.com/docs/models/gpt-4.1) lists the appropriately-formatted names. Validation would fail for "GPT-4.1" but succeed for `gpt-4.1` or `gpt-4.1-2025-04-14`.<br>
{% endhint %}

<figure><img src="/files/gD3rnSSZE0rRD75PeGzx" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="Azure OpenAI" %}
See [Azure AI Foundry's documentation](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/models?tabs=global-standard%2Cstandard-chat-completions) to learn more about their available models.

{% hint style="info" %}
Model names must be provided exactly as the Azure OpenAI Models REST API requires.

For example, the [GPT-4.1 series](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/models?tabs=global-standard%2Cstandard-chat-completions#gpt-41-series) lists the appropriately-formatted model names. Validation would fail for "GPT-4.1" but succeed for `gpt-4.1` or `gpt-4.1-nano`.
{% endhint %}

<figure><img src="/files/kKEWenpWdN8unn7iRyQM" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="AWS Bedrock" %}
See [AWS Bedrock's Model documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html) to learn more about their available models.

{% hint style="info" %}
Models must be provided exactly as the AWS Bedrock API requires.

For example, the table at the above documentation link lists the appropriately-formatted Model IDs. Validation would fail for "Nova Lite" but succeed for `amazon.nova-lite-v1:0`.<br>
{% endhint %}

<figure><img src="/files/WijP0lQ3Pl3kPKoS65ur" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="AWS SageMaker" %}
See [AWS SageMaker's Model documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-latest.html) to learn more about their available models.

{% hint style="info" %}
Models must be provided exactly as the AWS SageMaker AI Studio API requires.

For example, picking a model from [AWS SageMaker JumpStart](https://aws.amazon.com/sagemaker-ai/jumpstart/getting-started/?sagemaker-jumpstart-cards.sort-by=item.additionalFields.priority\&sagemaker-jumpstart-cards.sort-order=asc\&awsf.sagemaker-jumpstart-filter-product-type=*all\&awsf.sagemaker-jumpstart-filter-text=*all\&awsf.sagemaker-jumpstart-filter-vision=*all\&awsf.sagemaker-jumpstart-filter-tabular=*all\&awsf.sagemaker-jumpstart-filter-audio-tasks=*all\&awsf.sagemaker-jumpstart-filter-multimodal=*all\&awsf.sagemaker-jumpstart-filter-RL=*all), the model name "Falcon 7B Instruct BF16" will fail validation, but it will succeed with `huggingface-llm-falcon-7b-instruct-bf16`.<br>
{% endhint %}

<figure><img src="/files/WijP0lQ3Pl3kPKoS65ur" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="Google Gemini" %}
See [Google Gemini's documentation](https://ai.google.dev/gemini-api/docs/models) to learn more about their available models.

{% hint style="info" %}
Models must be provided exactly as the Google Gemini API requires.

For example,  [Google Gemini 2.5 Pro](https://ai.google.dev/gemini-api/docs/models#gemini-2.5-pro), lists the appropriately-formatted model codes. Validation would fail for "Gemini 2.5 Pro", but succeed with `gemini-2.5-pro` or `gemini-2.5-pro-preview-06-05`.<br>
{% endhint %}

<figure><img src="/files/YY9TAKRNBKZWq5NaBp6x" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="Google VertexAI" %}
See [Google Vertex AI's documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/models) to learn more about their available models.

{% hint style="info" %}
Models must be provided exactly as the Google Gemini API requires.

For example,  [Google Gemini 2.5 Pro](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro), lists the appropriately-formatted model IDs. Validation would fail for "Gemini 2.5 Pro", but succeed with `gemini-2.5-pro` or `gemini-2.5-pro-preview-06-05`.<br>
{% endhint %}

<figure><img src="/files/G0R5HKqtiSdiplEJ7Rsk" alt=""><figcaption></figcaption></figure>
{% endtab %}
{% endtabs %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.dbnl.com/v0.25.x/using-distributional/metrics/llm-models.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.