# Model Connections

### Why does DBNL require a Model Connection?

Model Connections are how DBNL interfaces with LLMs, which is required for each step of the [DBNL Data Pipeline](https://docs.dbnl.com/v0.27.x/configuration/data-pipeline) to function. It enables DBNL to

* Compute LLM-as-judge Metrics as part of the enrich step.
* Perform certain unsupervised analytics processes as part of the analysis step.
* Translate surfaced behavioral signals into human readable [Insights](https://docs.dbnl.com/v0.27.x/workflow/insights) as part of the publish step.

<figure><img src="https://content.gitbook.com/content/8N8zzLtIch6ZiTSwWtXD/blobs/InGhoEXW1vEr89AYTGF2/DBNL-LLMs.png" alt=""><figcaption></figcaption></figure>

### Types of Model Connections

Fundamentally a Model Connection needs to be able to expose a LLM chat completion interface that is accessible by your DBNL deployment. It can be

* An externally managed service (e.g. together.ai, OpenAI, etc)
* A cloud managed service that is part of your VPC (e.g. Bedrock, Vertex, etc)
* A locally managed deployment (e.g. a cluster of NVIDIA NIMs running in your DBNL k8s cluster as part of your deployment)

There are pros and cons to each of these approaches:

| Model Connection Type                                             | Pros                                                                                                                                                 | Cons                                                                                                                                 |
| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| <p>Externally managed service<br>(together.ai, OpenAI, etc)</p>   | <ul><li>Fast and easy to set up (just provide keys)</li><li>Model and scaling flexibility</li></ul>                                                  | <ul><li>Requires sending data outside of your cloud environment</li><li>Higher cost, on demand model</li></ul>                       |
| <p>Cloud managed service<br>(Bedrock, Vertex, etc)</p>            | <ul><li>Data stays within your cloud provider</li><li>Often managed by another team within the organization</li></ul>                                | <ul><li>Can be higher cost than locally running models</li><li>Usage, rate limits are typically shared across organization</li></ul> |
| <p>Locally managed deployment<br>(NVIDIA NIMs in k8s cluster)</p> | <ul><li>Data stays within your local deployment</li><li>Cheaper than a managed service</li><li>Maximum control of cost vs timing tradeoffs</li></ul> | <ul><li>Requires access to GPU resources</li><li>Can require local admin and debugging</li></ul>                                     |

### Creating a Model Connection

Model Connections are defined at the [Namespace](https://docs.dbnl.com/v0.27.x/platform/administration#namespaces) level of an [Organization](https://docs.dbnl.com/v0.27.x/platform/administration#organizations) and can be used by any [Projects](https://docs.dbnl.com/v0.27.x/workflow/projects) within the Namespace. For convenience, a new Model Connection can be created as part of the [Project Creation](https://docs.dbnl.com/v0.27.x/workflow/projects#creating-a-project) flow as well.

<figure><img src="https://content.gitbook.com/content/8N8zzLtIch6ZiTSwWtXD/blobs/rm8ETzRYpRjFr73Nphl0/image.png" alt=""><figcaption></figcaption></figure>

A Model Connection has the following attributes:

* **Name** (required): How the Model Connection is referenced when setting a default Model Connection for a project or LLM-as-judge [Metric](https://docs.dbnl.com/v0.27.x/workflow/metrics).
* **Description** (optional): Human readable description of the connection for reference.
* **Model** (required): The model name to be used as part of the API call (e.g. `gpt-3.5-turbo`, `gemini-2.0-flash-001`, etc). See the documentation for your model provider for more details.
* **Provider** (required): One of
  * [AWS Bedrock](https://aws.amazon.com/bedrock): Managed AWS service for foundation models.
  * [AWS Sagemaker](https://aws.amazon.com/sagemaker): Platform to build, train, and deploy machine learning models by AWS (not recommended for production DBNL deployments).
  * [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-foundry/models/openai): Microsoft service providing OpenAI models via Azure cloud.
  * [Google Gemini](https://gemini.google.com/): Google’s AI model for chat, code, and reasoning.
  * [Google Vertex AI](https://cloud.google.com/vertex-ai): Managed GCP service for building and deploying models.
  * [NVIDIA NIM](https://developer.nvidia.com/nim): NVIDIA microservices for deploying optimized AI models easily.
  * [OpenAI](https://openai.com/api/): Managed service for advanced language and reasoning models
  * OpenAI-compatible: Any provider that exposes an "OpenAI-like" API, like [together.ai](https://www.together.ai/)
* **Configuration Parameters** (required): Depending on the provider selected, you may need to provide additional required information like Access Key IDs, Secret Access Keys, preferred regions, endpoints/URLs, etc.

#### Configuration Parameters by Provider

Different providers require different configuration parameters:

{% tabs %}
{% tab title="AWS Bedrock" %}

* **AWS Access Key ID**: Your AWS IAM access key with Bedrock permissions
* **AWS Secret Access Key**: Corresponding secret key
* **AWS Region**: Region where Bedrock is available (e.g., `us-east-1`, `us-west-2`)
  {% endtab %}

{% tab title="AWS Sagemaker" %}

* **AWS Access Key ID**: Your AWS IAM access key
* **AWS Secret Access Key**: Corresponding secret key
* **Endpoint URL**: Your Sagemaker endpoint URL
* **AWS Region**: Region where your endpoint is deployed
  {% endtab %}

{% tab title="Azure OpenAI" %}

* **API Key**: Your Azure OpenAI resource key
* **Endpoint URL**: Your Azure OpenAI endpoint (e.g., `https://your-resource.openai.azure.com/`)
* **API Version**: Azure OpenAI API version (e.g., `2024-02-01`)
  {% endtab %}

{% tab title="Google Gemini" %}

* **API Key**: Your Google AI Studio API key
  {% endtab %}

{% tab title="Google Vertex AI" %}

* **Project ID**: Your GCP project ID
* **Region**: GCP region (e.g., `us-central1`)
* **Service Account JSON**: Path to service account credentials file (for authentication)
  {% endtab %}

{% tab title="NVIDIA NIM" %}

* **Endpoint URL**: URL where your NIM service is deployed (e.g., `http://nim-service.default.svc.cluster.local:8000`)
* **API Key**: (Optional) If authentication is enabled on your NIM deployment
  {% endtab %}

{% tab title="OpenAI" %}

* **API Key**: Your OpenAI API key from platform.openai.com
  {% endtab %}

{% tab title="OpenAI-compatible" %}

* **API Key**: API key from your provider
* **Base URL**: Provider's API endpoint (e.g., `https://api.together.xyz/v1` for together.ai)
  {% endtab %}
  {% endtabs %}

{% hint style="info" %}
**Finding your configuration values:**

* AWS credentials: [AWS IAM Console](https://console.aws.amazon.com/iam/)
* Azure keys: Azure Portal → Your OpenAI resource → Keys and Endpoint
* Google Gemini: [Google AI Studio](https://aistudio.google.com/apikey)
* OpenAI: [OpenAI Platform](https://platform.openai.com/api-keys)
  {% endhint %}

### Editing a Model Connection

A Model Connection can be edited or deleted by clicking on the "Model Connections" tab on the sidebar of the [Namespace](https://docs.dbnl.com/v0.27.x/platform/administration#namespaces) landing page.

### Debugging a Model Connection

A Model Connection can be tested by navigating to the specific Model Connection as above and clicking on the "Validate" button. This will send a simple request to the endpoint and inform you if it was able to complete the request.

### Next Steps

* **Ready to send data to your project?** Start ingesting data into your project using your defined [Data Connection](https://docs.dbnl.com/v0.27.x/configuration/data-connections) to kick off the [Adaptive Analytics Workflow](https://docs.dbnl.com/v0.27.x/workflow/adaptive-analytics-workflow).
* **Want to understand more about the platform?** Check out the [Architecture](https://docs.dbnl.com/v0.27.x/platform/architecture), [Deployment](https://docs.dbnl.com/v0.27.x/platform/deployment) options, and other aspects of the [Platform](https://docs.dbnl.com/v0.27.x/platform/platform).