> For the complete documentation index, see [llms.txt](https://docs.dbnl.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.dbnl.com/configuration/data-connections/otel-trace-ingestion.md).

# OTEL Trace Ingestion

{% hint style="info" %}
**Distributional is now** [**Talaria Scientific**](https://talariasci.com)**.** The DBNL product described in these docs has been sunset; this documentation is preserved for reference. Read [the announcement](https://distributional.com/blog/distributional-is-now-talaria).
{% endhint %}

[OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) (OTEL) Trace Ingestion allows for the richest data to be uploaded to your Project, but requires some off-platform coding and does not support backfilling data. This guide provides comprehensive instructions for instrumenting your AI agent application to send OpenTelemetry (OTEL) traces to DBNL.

## Prerequisites

**DBNL Credentials**: You'll need:

* DBNL API URL (e.g., `http://localhost:8080/api`)
* API Token (Bearer token for [authentication](https://github.com/dbnlAI/docs/tree/main/platform/authentication/README.md) which can be generated at `DBNL_API_URL/tokens`)
* Project ID (your DBNL project identifier, typically starts with `proj_` and is part of the URL for your project)

## Implementation

{% tabs %}
{% tab title="Python" %}

#### Quick Start (DBNL SDK)

The simplest way to get tracing working is with [`dbnl.init_tracing()`](/reference/python-sdk/sdk-functions.md#init_tracing), which handles provider setup, exporter configuration, and auto-instrumentation of supported libraries in a single call.

```bash
pip install 'dbnl[instrumentation]'
```

The `instrumentation` extra installs [OpenInference](https://github.com/Arize-ai/openinference) instrumentors for popular libraries (OpenAI, LangChain, Anthropic, etc.) so that `init_tracing()` can auto-instrument them. If you only need the exporter without auto-instrumentation, `pip install dbnl` is sufficient.

```python
import dbnl

dbnl.login()
dbnl.init_tracing(
    project_id="{PROJECT_ID}",
    service_name="my-agent",
)
```

`init_tracing()` registers a global `TracerProvider` with a `BatchSpanProcessor` pointing at your DBNL deployment, and automatically instruments any installed OpenInference-compatible libraries. Pass `auto_instrument=False` to disable this and instrument manually.

To tag traces with an application version or experiment variants, see [App Versioning & Experiments](/configuration/app-versioning-and-experiments.md).

#### Advanced: Raw OpenTelemetry Setup

If you need full control over the `TracerProvider` (for example to add additional exporters, custom resources, or a non-default span processor), you can manage your own provider while still using the SDK to handle exporter configuration.

Pass your own `TracerProvider` to [`init_tracing()`](/reference/python-sdk/sdk-functions.md#init_tracing) and it will attach the dbnl span processor and run auto-instrumentation on it, without replacing the global provider:

```python
import dbnl
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import Resource

dbnl.login()

resource = Resource.create({"service.name": "my-agent"})
tracer_provider = TracerProvider(resource=resource)

# Add your own processors/exporters here
# tracer_provider.add_span_processor(...)

dbnl.init_tracing(
    project_id="{PROJECT_ID}",
    tracer_provider=tracer_provider,
)

trace.set_tracer_provider(tracer_provider)
```

Pass `auto_instrument=False` if you want to skip auto-instrumentation and instrument libraries yourself.

For even more control, [`dbnl.get_dbnl_span_processor()`](/reference/python-sdk/sdk-functions.md#get_dbnl_span_processor) returns a pre-configured `BatchSpanProcessor` you can add to any provider directly, but note that it does not run auto-instrumentation.

**Fully Manual Setup (no SDK)**

If you prefer not to depend on the `dbnl` package at runtime, you can wire the OTLP exporter directly:

```bash
pip install 'opentelemetry-sdk>=1.20.0' 'opentelemetry-exporter-otlp>=1.20.0'
```

For LangChain applications, also install [OpenInference](https://arize-ai.github.io/openinference/) instrumentation:

```bash
pip install 'openinference-instrumentation-langchain>=0.1.0'
```

Create a telemetry initialization module (`telemetry.py`) in your application:

```python
import os
import logging
from typing import Optional
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def create_dbnl_exporter() -> Optional[OTLPSpanExporter]:
    """Create OTLP exporter for DBNL"""
    # Get configuration from environment
    api_url = os.environ.get("DBNL_API_URL", "").strip()
    api_token = os.environ.get("DBNL_API_TOKEN", "").strip()
    project_id = os.environ.get("DBNL_PROJECT_ID", "").strip()

    # Validate configuration
    if not all([api_url, api_token, project_id]):
        logger.info("DBNL configuration incomplete. Set DBNL_API_URL, DBNL_API_TOKEN, and DBNL_PROJECT_ID.")
        return None

    # Create headers
    headers = {
        "Authorization": f"Bearer {api_token}",
        "x-dbnl-project-id": project_id,
        "Content-Type": "application/x-protobuf",
    }

    # Create exporter pointing at the DBNL OTLP endpoint
    endpoint = f"{api_url}/otel/v1/traces"
    exporter = OTLPSpanExporter(endpoint=endpoint, headers=headers)
    logger.info(f"✅ DBNL exporter configured: {endpoint}")
    return exporter

def initialize_telemetry():
    """Initialize OpenTelemetry with DBNL exporter"""
    # Create tracer provider with resource attributes
    resource = Resource.create({
        "service.name": os.environ.get("OTEL_SERVICE_NAME", "my-agent"),
    })

    tracer_provider = TracerProvider(resource=resource)
    trace.set_tracer_provider(tracer_provider)

    # Add DBNL exporter
    dbnl_exporter = create_dbnl_exporter()
    if dbnl_exporter:
        tracer_provider.add_span_processor(BatchSpanProcessor(dbnl_exporter))
        logger.info("📊 DBNL OTEL tracing enabled")

    return tracer_provider

# Initialize on import
tracer_provider = initialize_telemetry()
tracer = trace.get_tracer(__name__)
```

**LangChain Integration**

For LangChain applications, add OpenInference instrumentation:

```python
from openinference.instrumentation.langchain import LangChainInstrumentor

def initialize_telemetry():
    """Initialize OpenTelemetry with DBNL exporter and LangChain instrumentation"""
    # ... (previous code) ...

    # Add LangChain instrumentation
    instrumentor = LangChainInstrumentor()
    instrumentor.instrument(tracer_provider=tracer_provider)

    return tracer_provider
```

**Application Integration**

Initialize telemetry early in your application startup:

**FastAPI Example:**

```python
from fastapi import FastAPI
from telemetry import initialize_telemetry

app = FastAPI()

@app.on_event("startup")
async def startup_event():
    initialize_telemetry()
```

**Standalone Script Example:**

```python
from telemetry import initialize_telemetry

if __name__ == "__main__":
    initialize_telemetry()
```

#### Message Content Capture

DBNL reads message content (prompts, completions, tool arguments) from **span attributes only**. There are two OTel instrumentor ecosystems for GenAI, and they handle content very differently:

* [**OpenInference**](https://github.com/Arize-ai/openinference) **instrumentors (`openinference-instrumentation-*`)** always place message content on span attributes. This is a core design choice in the OpenInference spec, so DBNL ingests their content with no extra configuration.
* [**OTel GenAI semantic convention**](https://opentelemetry.io/docs/specs/semconv/gen-ai/) **instrumentors (the `gen_ai.*` namespace)** are split: the spec allows content on either span attributes or separate OTel log records, and individual instrumentors pick different defaults. DBNL ingests traces, not logs, so any instrumentor that emits content as log events will produce traces with empty `input` and `output` columns.

When an OpenInference instrumentor exists for your framework, prefer it. If you must use an OTel GenAI instrumentor, the table below covers the common ones and how to ensure content lands on span attributes.

**OpenInference instrumentors (no configuration needed)**

| Framework       | Instrumentor                               |
| --------------- | ------------------------------------------ |
| LangChain       | `openinference-instrumentation-langchain`  |
| Google ADK      | `openinference-instrumentation-google-adk` |
| Anthropic       | `openinference-instrumentation-anthropic`  |
| OpenAI (direct) | `openinference-instrumentation-openai`     |

See the [OpenInference repo](https://github.com/Arize-ai/openinference) for the full list of supported frameworks.

**OTel GenAI instrumentors (configuration required)**

| Framework         | Instrumentor                                     | Default                      | How to get span attributes                                            |
| ----------------- | ------------------------------------------------ | ---------------------------- | --------------------------------------------------------------------- |
| Pydantic AI       | Native (`InstrumentationSettings`)               | Span attrs (recent releases) | Use `version=2` or higher; avoid `version=1`, which emits log events. |
| OpenAI Agents SDK | `opentelemetry-instrumentation-openai-agents-v2` | Off                          | Set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`.        |

If traces are appearing in DBNL but `input` / `output` are blank, the instrumentor is almost certainly emitting content as log events. Pick one of the configurations above.

**Pydantic AI**

```python
from pydantic_ai.agent import Agent
from pydantic_ai.models.instrumented import InstrumentationSettings

agent = Agent(
    "openai:gpt-5-nano",
    instrument=InstrumentationSettings(version=2),
)
```

**OpenAI Agents SDK**

```bash
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
```

```python
from opentelemetry.instrumentation.openai_agents import OpenAIAgentsInstrumentor

OpenAIAgentsInstrumentor().instrument()
```

{% endtab %}

{% tab title="Node.js" %}

#### Quick Start

Install the required OpenTelemetry packages:

```bash
npm install @opentelemetry/sdk-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto tsx
```

Create an `instrumentation.ts` file that configures the OTLP exporter to send traces to DBNL:

```typescript
import { NodeSDK } from "@opentelemetry/sdk-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";

const sdk = new NodeSDK({
  spanProcessors: [
    new BatchSpanProcessor(
      new OTLPTraceExporter({
        url: "{DBNL_API_URL}/otel/v1/traces",
        headers: {
          Authorization: "Bearer {API_TOKEN}",
          "x-dbnl-project-id": "{PROJECT_ID}",
        },
      })
    ),
  ],
});

sdk.start();
```

Use the `--import` flag to load instrumentation before your application code:

```bash
npx tsx --import ./instrumentation.ts app.ts
```

#### Auto-Instrumentation

For automatic tracing of LLM provider calls, add the relevant [OpenInference JS instrumentation](https://github.com/Arize-ai/openinference/tree/main/js) package and pass it to the `NodeSDK`. For example, to auto-instrument OpenAI calls:

```bash
npm install @arizeai/openinference-instrumentation-openai
```

```typescript
import { NodeSDK } from "@opentelemetry/sdk-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { OpenAIInstrumentation } from "@arizeai/openinference-instrumentation-openai";

const sdk = new NodeSDK({
  spanProcessors: [
    new BatchSpanProcessor(
      new OTLPTraceExporter({
        url: "{DBNL_API_URL}/otel/v1/traces",
        headers: {
          Authorization: "Bearer {API_TOKEN}",
          "x-dbnl-project-id": "{PROJECT_ID}",
        },
      })
    ),
  ],
  instrumentations: [new OpenAIInstrumentation()],
});

sdk.start();
```

Instrumentation packages are available for OpenAI, Anthropic, LangChain, Bedrock, and others. See the full list in the [OpenInference JS packages](https://github.com/Arize-ai/openinference/tree/main/js/packages).
{% endtab %}

{% tab title="OTel Collector" %}

#### Overview

If you already run an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/), you can forward traces to DBNL by adding an OTLP/HTTP exporter to your collector configuration. This lets you centralize trace routing without changing application code.

#### Configuration

Add the following blocks to your collector YAML config:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlphttp/dbnl:
    endpoint: "{DBNL_API_URL}/otel"
    headers:
      Authorization: "Bearer {API_TOKEN}"
      x-dbnl-project-id: "{PROJECT_ID}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/dbnl]
```

{% hint style="info" %}
The exporter endpoint is `{DBNL_API_URL}/otel` (without `/v1/traces`) because the `otlphttp` exporter appends `/v1/traces` automatically.
{% endhint %}

{% hint style="warning" %}
If you already have a collector config, merge these blocks into your existing `receivers`, `exporters`, and `service.pipelines` sections rather than replacing them. You can add `otlphttp/dbnl` alongside your other exporters in the `traces` pipeline.
{% endhint %}

#### Semantic Conventions

The collector forwards traces as-is; it does not transform span attributes. Your application's instrumentation must emit traces using [OpenInference Semantic Conventions](https://github.com/Arize-ai/openinference) so that DBNL can parse the required fields (`input`, `output`, etc.) from span attributes.
{% endtab %}
{% endtabs %}

## Required Trace Fields

The following fields are required regardless of which ingestion method you are using:

* **`input`**: The text input to the LLM as a `string`
* **`output`**: The text response from the LLM as a `string`
* **`timestamp`**: The UTC timecode associated with the LLM call as a `timestamptz`

You may choose to track other attributes such as `total_token_count` or `feedback_score` which are part of the [DBNL semantic convention](/configuration/dbnl-semantic-convention.md).

### Custom Attributes

Custom metadata should be added as span attributes using the [OpenInference semantic convention](https://github.com/Arize-ai/openinference). These attributes are available within the `spans` data for analysis. Note that only columns defined in the [DBNL Semantic Convention](/configuration/dbnl-semantic-convention.md) are supported as top-level columns; arbitrary custom columns are not ingested.

```python
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("agent_execution") as span:
    # Set semantic attributes
    span.set_attribute("input.value", user_query)
    span.set_attribute("output.value", agent_response)
    
    # Add custom metadata
    span.set_attribute("session.id", session_id)
    span.set_attribute("conversation.id", conversation_id)
    span.set_attribute("tool.name", "search_symbol")
    span.set_attribute("tool.success", True)
    span.set_attribute("deployment.type", "web-application")
```

## Advanced Configuration

### Batch Processing

DBNL uses `BatchSpanProcessor` by default for efficient trace export. This batches spans before sending, reducing network overhead:

```python
from opentelemetry.sdk.trace.export import BatchSpanProcessor

processor = BatchSpanProcessor(dbnl_exporter)
tracer_provider.add_span_processor(processor)
```

For immediate export (useful for debugging), use `SimpleSpanProcessor`:

```python
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

processor = SimpleSpanProcessor(dbnl_exporter)
tracer_provider.add_span_processor(processor)
```

## Verification

### Test Trace Export

Create a test span to verify traces are being sent:

```python
from opentelemetry import trace
from telemetry import tracer_provider

tracer = trace.get_tracer(__name__)

# Create a test span
with tracer.start_as_current_span("test_dbnl_export") as span:
    span.set_attribute("input.value", "test input")
    span.set_attribute("output.value", "test output")
    span.set_attribute("test", True)

# Force flush to ensure export
tracer_provider.force_flush()
print("✅ Test span exported to DBNL")
```

### View Traces in DBNL

After sending traces, verify they appear in your DBNL dashboard. By default, traces are processed into logs nightly so you will not see them right away.

1. Log into your DBNL deployment and go to your project
2. Check the Status page to confirm that they have been processed
3. Navigate to the Explorer or Logs section
4. Filter by your project ID or service name
5. Verify traces are appearing with the expected attributes

## Troubleshooting

### Traces Not Appearing in DBNL

1. **Check Environment Variables**: Verify all required variables are set:

   ```bash
   echo $DBNL_API_URL
   echo $DBNL_API_TOKEN
   echo $DBNL_PROJECT_ID
   ```
2. **Verify API Endpoint**: Test connectivity to DBNL:

   ```bash
   curl -H "Authorization: Bearer $DBNL_API_TOKEN" \
        -H "x-dbnl-project-id: $DBNL_PROJECT_ID" \
        $DBNL_API_URL/health
   ```
3. **Check Logs**: Look for DBNL exporter configuration messages:

   ```
   ✅ DBNL exporter configured: http://localhost:8080/otel/v1/traces
   📊 DBNL OTEL tracing enabled
   ```
4. **Verify URL Formatting**: Ensure the endpoint is correctly formatted:
   * Format: `{DBNL_API_URL}/otel/v1/traces`
   * Example: `http://localhost:8080/otel/v1/traces`

### Common Issues

**Issue: "DBNL configuration incomplete"**

* **Solution**: Ensure `DBNL_API_URL`, `DBNL_API_TOKEN`, and `DBNL_PROJECT_ID` are all set

**Issue: "Failed to configure DBNL exporter"**

* **Solution**: Check that the API URL is valid and the token has proper permissions

**Issue: Traces appear but missing attributes**

* **Solution**: Ensure you're using OpenInference semantic conventions or manually setting required attributes (`input`, `output`, `timestamp`)

**Issue: High latency or performance impact**

* **Solution**: Use `BatchSpanProcessor` (default) instead of `SimpleSpanProcessor` for better performance

For issues or questions:

1. Check the troubleshooting section above
2. Review DBNL documentation
3. Verify your DBNL deployment has OTEL Trace Ingestion enabled
4. Contact DBNL support at <support@distributional.com> with your project ID and API endpoint

## Best Practices

1. **Use Batch Processing**: Always use `BatchSpanProcessor` in production for better performance
2. **Use Semantic Conventions**: Follow OpenInference conventions for automatic attribute mapping
3. **Error Handling**: Wrap exporter creation in try-except blocks to prevent application failures
4. **Graceful Degradation**: Allow your application to function even if DBNL configuration is incomplete

## Additional Resources

* [DBNL Semantic Convention](https://docs.dbnl.com/configuration/data-pipeline/dbnl-semantic-convention) - Learn about semantic conventions for better analytics
* [OpenTelemetry Python Documentation](https://opentelemetry.io/docs/instrumentation/python/) - Official OpenTelemetry Python docs
* [OpenTelemetry JavaScript Documentation](https://opentelemetry.io/docs/instrumentation/js/) - Official OpenTelemetry JS docs
* [OpenTelemetry Collector Documentation](https://opentelemetry.io/docs/collector/) - Official OpenTelemetry Collector docs
* [OpenInference Documentation](https://github.com/Arize-ai/openinference) - OpenInference semantic conventions