Datadog Integration

Complete guide for integrating Autotel with Datadog using OpenTelemetry Protocol (OTLP).

Overview

Autotel provides first-class support for Datadog through the industry-standard OpenTelemetry Protocol (OTLP). This approach offers several advantages over vendor-specific integrations:

Benefits of OTLP Integration

• Unified Observability: Send traces, logs, and metrics through a single protocol
• Vendor Neutrality: Switch between Datadog, Honeycomb, New Relic without code changes
• Future-Proof: Built on OpenTelemetry, the CNCF industry standard
• Automatic Correlation: Traces and logs are automatically linked
• Simplified Setup: Single configuration for all observability signals
• Datadog Best Practices: Built-in support for Unified Service Tagging, hostname detection, and proper resource attributes

What You Get

When you integrate Autotel with Datadog, you get:

Signal	Destination	Features
Traces	Datadog APM	Distributed tracing, flame graphs, service maps, error tracking
Logs	Datadog Logs	Structured logs with automatic trace correlation
Metrics	Datadog Metrics	Custom business and application metrics
Infrastructure	Datadog Infrastructure (via Agent)	Host metrics, container metrics, process monitoring

Quick Start

1. Install Dependencies

# Core package
npm install autotel

# Backends (includes log export - no separate log packages needed)
npm install autotel-backends

2. Get Datadog API Key

1. Log in to Datadog
2. Go to Organization Settings > API Keys
3. Create or copy an existing API key
4. Note your Datadog site (e.g., datadoghq.com, datadoghq.eu)

3. Initialize Autotel

Simple Configuration (Traces + Metrics only):

import { init } from 'autotel';

init({
  service: 'my-app',
  environment: 'production',
  version: '1.0.0',
  endpoint: 'https://otlp.datadoghq.com',
  otlpHeaders: `dd-api-key=${process.env.DATADOG_API_KEY}`,
});

Recommended: Use the Datadog Preset (Traces + Logs + Metrics):

import { init } from 'autotel';
import { createDatadogConfig } from 'autotel-backends/datadog';

init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: 'my-app',
  environment: 'production',
  version: '1.0.0',
  site: 'datadoghq.com',
  enableLogs: true,
}));

Log libs are bundled in autotel-backends; no app-level install of @opentelemetry/sdk-logs or @opentelemetry/exporter-logs-otlp-http needed.

4. Instrument Your Code

import { trace } from 'autotel';
import { createBuiltinLogger } from 'autotel/logger';

const logger = createBuiltinLogger('my-app');

const processOrder = trace((ctx) => async (orderId: string) => {
  logger.info('Processing order', { orderId });

  ctx.setAttribute('order.id', orderId);
  ctx.setAttribute('order.status', 'processing');

  // Your business logic here

  return { success: true };
});

5. View Results in Datadog

• Traces: https://app.datadoghq.com/apm/traces
• Logs: https://app.datadoghq.com/logs
• Metrics: https://app.datadoghq.com/metric/explorer
• Service Catalog: https://app.datadoghq.com/services

Architecture Choices

Datadog supports two ingestion architectures. Choose based on your deployment type and requirements.

Direct Cloud Ingestion

How it works: Application sends OTLP data directly to Datadog cloud endpoints via HTTPS.

┌─────────────────┐
│   Application   │
│    (autotel)    │
└────────┬────────┘
         │ OTLP/HTTPS
         │ (API key in headers)
         ▼
┌─────────────────┐
│ Datadog Cloud   │
│ OTLP Intake     │
└─────────────────┘

Best for:

• Serverless (AWS Lambda, Google Cloud Functions, Azure Functions)
• Edge runtimes (Cloudflare Workers, Vercel Edge, Deno Deploy)
• Containerized apps without persistent infrastructure (AWS Fargate, Cloud Run)
• Development environments
• Getting started quickly

Pros:

• Zero infrastructure - no Agent to install/manage
• Works anywhere with HTTPS egress
• Simple configuration
• Perfect for ephemeral workloads

Cons:

• Higher egress costs (every instance sends directly)
• No local data aggregation/buffering
• Missing Agent-specific features (see below)

Configuration:

import { createDatadogConfig } from 'autotel-backends/datadog';

init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: 'my-lambda',
  site: 'datadoghq.com',
}));

Datadog Agent

How it works: Application sends OTLP to local Datadog Agent, which aggregates and forwards to Datadog cloud.

┌─────────────────┐
│   Application   │
│    (autotel)    │
└────────┬────────┘
         │ OTLP/HTTP (localhost)
         │ (no API key needed)
         ▼
┌─────────────────┐
│ Datadog Agent   │
│  (localhost)    │
│                 │
│ • Aggregates    │
│ • Enriches      │
│ • Scrubs PII    │
│ • Buffers       │
└────────┬────────┘
         │ Optimized protocol
         │ (API key in Agent)
         ▼
┌─────────────────┐
│ Datadog Cloud   │
└─────────────────┘

Best for:

• Production long-running services (Node.js APIs, web servers)
• Kubernetes/container orchestration
• On-premise/VM deployments
• High-volume applications
• Advanced monitoring needs

Pros:

• Lower costs: Agent batches/compresses locally, reducing egress
• 500+ integrations: Auto-collect infrastructure metrics (CPU, memory, disk, network, database stats)
• Advanced log processing: Multi-line log parsing (stack traces), PII scrubbing and sensitive data redaction, log enrichment with Kubernetes/container tags
• Trace-log correlation: Enhanced correlation in Agent
• Local buffering: Queues data during network issues
• DogStatsD: Ultra-low latency metrics via UDP
• Live debugging: Datadog Live Tail, Dynamic Instrumentation

Cons:

• Requires Agent installation/management
• Not available in serverless/edge environments
• Additional infrastructure dependency

Configuration:

import { createDatadogConfig } from 'autotel-backends/datadog';

init(createDatadogConfig({
  service: 'my-api',
  useAgent: true,
  agentHost: 'localhost', // Or Kubernetes service name
  agentPort: 4318,
}));

Decision Matrix

Factor	Direct Cloud Ingestion	Datadog Agent
Deployment Type	Serverless, Edge, Ephemeral	Long-running, VMs, Kubernetes
Infrastructure Required	None	Datadog Agent
API Key Location	Application	Agent (not in app)
Egress Costs	Higher (per instance)	Lower (Agent batches)
Infrastructure Metrics	No	Yes (500+ integrations)
Log Processing	Basic	Advanced (multi-line, PII scrubbing)
Setup Complexity	Low	Medium
Buffering	None	Yes (Agent queues)
DogStatsD	No	Yes
Best For	Serverless, development	Production, Kubernetes

Hybrid Approach (Recommended for large deployments):

• Serverless functions: Direct cloud ingestion
• Backend APIs/services: Datadog Agent
• Edge functions: Direct cloud ingestion
• Kubernetes workloads: Datadog Agent (DaemonSet)

Setup Instructions

Direct Cloud Ingestion Setup

1. Install Dependencies

npm install autotel

# Optional: for log export
npm install autotel-backends

2. Set Environment Variables

export DATADOG_API_KEY="your_api_key_here"
export DATADOG_SITE="datadoghq.com"  # or datadoghq.eu, us3.datadoghq.com, etc.

3. Configure Application

import { init } from 'autotel';
import { createDatadogConfig } from 'autotel-backends/datadog';

init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: 'my-app',
  environment: process.env.NODE_ENV || 'development',
  version: process.env.APP_VERSION,
  site: process.env.DATADOG_SITE as any || 'datadoghq.com',
  enableLogs: true, // Optional
}));

4. Deploy and Verify

After deploying, check Datadog:

• APM > Traces: Should see traces within 1-2 minutes
• Logs > Search: Filter by service:my-app
• Service Catalog: Your service should appear automatically

Datadog Agent Setup

1. Install Datadog Agent

On macOS:

DD_API_KEY="your_api_key" DD_SITE="datadoghq.com" bash -c "$(curl -L https://install.datadoghq.com/scripts/install_mac_os.sh)"

On Ubuntu/Debian:

DD_API_KEY="your_api_key" DD_SITE="datadoghq.com" bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

On Kubernetes (Helm):

helm repo add datadog https://helm.datadoghq.com
helm repo update

helm install datadog-agent datadog/datadog \
  --set datadog.apiKey=<YOUR_API_KEY> \
  --set datadog.site=datadoghq.com \
  --set datadog.otlp.receiver.protocols.http.enabled=true \
  --set datadog.otlp.receiver.protocols.grpc.enabled=true

On Docker:

docker run -d \
  --name datadog-agent \
  -e DD_API_KEY=<YOUR_API_KEY> \
  -e DD_SITE="datadoghq.com" \
  -e DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT="0.0.0.0:4318" \
  -e DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT="0.0.0.0:4317" \
  -p 4318:4318 \
  -p 4317:4317 \
  gcr.io/datadoghq/agent:latest

2. Enable OTLP Receiver

Edit /etc/datadog-agent/datadog.yaml:

# Enable OTLP receiver
otlp_config:
  receiver:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

# Optional: Enable debug logging
log_level: info

Restart the Agent:

sudo systemctl restart datadog-agent

Or with environment variables (Docker/Kubernetes):

DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT=0.0.0.0:4318
DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT=0.0.0.0:4317

3. Configure Application

import { init } from 'autotel';
import { createDatadogConfig } from 'autotel-backends/datadog';

init(createDatadogConfig({
  service: 'my-app',
  environment: 'production',
  version: '1.0.0',
  useAgent: true,
  agentHost: 'localhost', // Default
  agentPort: 4318,        // Default
}));

For Kubernetes: Use the Agent’s service hostname:

init(createDatadogConfig({
  service: 'my-app',
  useAgent: true,
  // Agent runs as DaemonSet, accessible via node-local or service
  agentHost: process.env.DD_AGENT_HOST || 'datadog-agent.default.svc.cluster.local',
}));

4. Verify Agent is Receiving Data

sudo datadog-agent status

Look for the OTLP section showing received spans/metrics/logs:

=========
OTLP
=========
  HTTP:
    Endpoint: 0.0.0.0:4318
    Spans: 1234 received
    Metrics: 567 received

  Status: OK

Using the Datadog Preset

The createDatadogConfig() preset helper simplifies configuration for both architectures.

Basic Usage

import { init } from 'autotel';
import { createDatadogConfig } from 'autotel-backends/datadog';

// Direct cloud ingestion
init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: 'my-app',
}));

// Local Agent
init(createDatadogConfig({
  service: 'my-app',
  useAgent: true,
}));

Full Configuration Options

interface DatadogPresetConfig {
  // Required
  service: string;

  // Cloud ingestion (required if useAgent: false)
  apiKey?: string;
  site?: 'datadoghq.com' | 'datadoghq.eu' | 'us3.datadoghq.com' | 'us5.datadoghq.com' | 'ap1.datadoghq.com';

  // Agent mode
  useAgent?: boolean;
  agentHost?: string;    // Default: 'localhost'
  agentPort?: number;    // Default: 4318

  // Optional (both modes)
  environment?: string;  // Default: DD_ENV || NODE_ENV || 'development'
  version?: string;      // Default: DD_VERSION || auto-detected
  enableLogs?: boolean;  // Default: false

  // Advanced
  logRecordProcessors?: LogRecordProcessor[]; // Custom log processors
}

Environment Variables

The preset respects Datadog standard environment variables:

DD_ENV=production           # Sets environment
DD_VERSION=1.2.3            # Sets version
DD_HOSTNAME=my-host         # Sets hostname
DD_AGENT_HOST=localhost     # Sets Agent host

Examples

Serverless (Lambda):

init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: 'order-processor',
  environment: 'production',
  site: 'datadoghq.com',
}));

Kubernetes with Agent:

init(createDatadogConfig({
  service: 'api-gateway',
  useAgent: true,
  agentHost: process.env.DD_AGENT_HOST || 'datadog-agent.monitoring.svc.cluster.local',
}));

Development:

init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: 'my-app-dev',
  environment: 'development',
  enableLogs: true, // More verbose in dev
}));

Migration from pino-datadog-transport

If you’re currently using pino-datadog-transport, Autotel provides a superior alternative with traces and automatic correlation.

Why Migrate?

Feature	pino-datadog-transport	Autotel + OTLP
Logs	Yes	Yes
Traces	No	Yes
Metrics	No	Yes
Trace-Log Correlation	Manual	Automatic
Protocol	Proprietary Datadog API	OTLP (industry standard)
Vendor Lock-in	Datadog-only	Vendor-neutral
Infrastructure Metrics	No	Yes (with Agent)

Migration Strategies

Strategy 1: Incremental (Recommended)

Keep pino-datadog-transport for logs, add Autotel for traces and metrics.

Before (logs only):

import pino from 'pino';

const logger = pino(
  pino.transport({
    target: 'pino-datadog-transport',
    options: {
      ddClientConf: {
        authMethods: { apiKeyAuth: process.env.DATADOG_API_KEY }
      },
      ddServerConf: { site: 'datadoghq.com' },
      service: 'my-app'
    }
  })
);

After (logs + traces + metrics):

import pino from 'pino';
import { init, trace } from 'autotel';

// Keep existing logger for now
const logger = pino(
  pino.transport({
    target: 'pino-datadog-transport',
    options: {
      ddClientConf: {
        authMethods: { apiKeyAuth: process.env.DATADOG_API_KEY }
      },
      ddServerConf: { site: 'datadoghq.com' },
      service: 'my-app'
    }
  })
);

// Add autotel for traces and metrics only
init({
  service: 'my-app',
  endpoint: 'https://otlp.datadoghq.com',
  otlpHeaders: `dd-api-key=${process.env.DATADOG_API_KEY}`,
  // No logRecordProcessors - keep existing log pipeline
});

// Now you have traces!
const processOrder = trace(async (orderId) => {
  logger.info({ orderId }, 'Processing order');
  // ... business logic
});

Benefits:

• Zero risk - no changes to existing log pipeline
• Immediately gain distributed tracing
• Can validate traces before migrating logs
• Only requires installing autotel

Strategy 2: Full Migration

Replace everything with Autotel OTLP.

Before:

import pino from 'pino';

const logger = pino(
  pino.transport({
    target: 'pino-datadog-transport',
    options: {
      ddClientConf: {
        authMethods: { apiKeyAuth: process.env.DATADOG_API_KEY }
      },
      ddServerConf: { site: 'datadoghq.eu' },
      service: 'my-app',
      ddsource: 'nodejs'
    }
  })
);

logger.info({ userId: '123' }, 'User created');

After:

import { init, trace } from 'autotel';
import { createBuiltinLogger } from 'autotel/logger';
import { createDatadogConfig } from 'autotel-backends/datadog';

const logger = createBuiltinLogger('my-app');

init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: 'my-app',
  site: 'datadoghq.eu',
  enableLogs: true,
}));

const createUser = trace((ctx) => async (userId: string) => {
  logger.info('User created', { userId });
  // Trace ID automatically included in logs!
  ctx.setAttribute('user.id', userId);
  // ... business logic
});

Benefits:

• Unified observability (traces + logs + metrics)
• Automatic trace-log correlation (no manual injection)
• OTLP standard (can switch to Honeycomb, New Relic later)
• Simpler configuration

Migration Steps:

1. Install: npm install autotel-backends
2. Replace pino setup with createBuiltinLogger()
3. Update init() to include logRecordProcessors
4. Test in development
5. Deploy to staging, validate logs appear in Datadog
6. Remove pino-datadog-transport dependency

Best Practices

1. Use Unified Service Tagging

Datadog’s Unified Service Tagging requires three tags on all telemetry:

init(createDatadogConfig({
  service: 'checkout-api',       // Required
  environment: 'production',      // Required
  version: '2.1.0',              // Required
}));

Why: Enables Deployment Tracking, Service Catalog, and proper correlation across signals.

2. Use Environment Variables

Don’t hardcode configuration:

init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: process.env.SERVICE_NAME!,
  environment: process.env.DD_ENV || process.env.NODE_ENV,
  version: process.env.DD_VERSION || require('./package.json').version,
  site: (process.env.DATADOG_SITE as any) || 'datadoghq.com',
}));

3. Use Adaptive Sampling in Production

Default sampling (10% baseline, 100% errors/slow) is good for most cases:

init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: 'my-app',
  // Default sampling is already adaptive - no config needed!
}));

For custom sampling:

import { AdaptiveSampler } from 'autotel/sampling';

init({
  service: 'my-app',
  endpoint: '...',
  otlpHeaders: '...',
  sampler: new AdaptiveSampler({
    baselineSampleRate: 0.05,  // 5% of normal traffic
    slowThresholdMs: 1000,      // Requests >1s are "slow"
    alwaysSampleErrors: true,   // Always capture errors
    alwaysSampleSlow: true,     // Always capture slow requests
  }),
});

4. Add Meaningful Attributes

Use semantic conventions where possible:

const processCheckout = trace((ctx) => async (userId, items) => {
  // Datadog recognizes these standard attributes
  ctx.setAttribute('user.id', userId);
  ctx.setAttribute('http.method', 'POST');
  ctx.setAttribute('http.route', '/checkout');

  // Custom business attributes
  ctx.setAttribute('checkout.items_count', items.length);
  ctx.setAttribute('checkout.total_amount', calculateTotal(items));
});

5. Use Structured Logging

Always log with context objects:

// Good
logger.info('Order processed', {
  orderId,
  amount,
  customerId,
  processingTime: duration
});

// Bad - harder to query in Datadog
logger.info(`Order ${orderId} processed for customer ${customerId}`);

6. Enable Log Correlation

Logs automatically include trace IDs for correlation:

const logger = createBuiltinLogger('my-app');

const processOrder = trace((ctx) => async (orderId) => {
  // This log automatically includes:
  // - traceId (hex)
  // - spanId (hex)
  // - dd.trace_id (decimal, for Datadog)
  // - dd.span_id (decimal, for Datadog)
  logger.info('Processing order', { orderId });
});

View correlated logs in Datadog:

1. Go to APM > Traces
2. Click any trace
3. See “Logs” tab with related log entries

7. Use Metrics for Business KPIs

Track important business metrics:

import { Metrics } from 'autotel/metrics';

const metrics = new Metric('checkout');

const processCheckout = trace(async (items) => {
  const total = calculateTotal(items);

  // Track business metrics
  metrics.trackEvent('checkout.completed', {
    payment_method: 'credit_card',
    currency: 'USD',
  });

  metrics.trackValue('checkout.amount', total, {
    currency: 'USD',
  });
});

8. Use Agent in Production

For production workloads on Kubernetes/VMs, use the Datadog Agent:

Benefits:

• Lower costs (Agent batches/compresses)
• Infrastructure metrics (CPU, memory, network)
• Advanced log features (multi-line parsing, PII scrubbing)
• Better reliability (local buffering)

Deployment Patterns

AWS Lambda

Challenge: Ephemeral, no persistent Agent Solution: Direct cloud ingestion

import { init } from 'autotel';
import { createDatadogConfig } from 'autotel-backends/datadog';

// Initialize once (outside handler for warm starts)
init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: 'order-processor',
  environment: process.env.ENVIRONMENT!,
  version: process.env.VERSION,
  site: 'datadoghq.com',
}));

export const handler = trace(async (event) => {
  // Your Lambda logic
  return { statusCode: 200, body: 'OK' };
});

Kubernetes

Challenge: Multiple pods, need infrastructure metrics Solution: Datadog Agent as DaemonSet

1. Install Agent:

helm install datadog-agent datadog/datadog \
  --set datadog.apiKey=$DD_API_KEY \
  --set datadog.otlp.receiver.protocols.http.enabled=true \
  --set datadog.logs.enabled=true \
  --set datadog.logs.containerCollectAll=true

2. Configure app:

init(createDatadogConfig({
  service: 'api-gateway',
  environment: 'production',
  useAgent: true,
  // Agent runs as DaemonSet on each node
  agentHost: process.env.DD_AGENT_HOST || 'localhost',
}));

3. Set pod environment:

apiVersion: v1
kind: Pod
metadata:
  name: api-gateway
spec:
  containers:
  - name: app
    image: myapp:1.0.0
    env:
    - name: DD_AGENT_HOST
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP  # DaemonSet agent on node

Docker Compose

Solution: Agent as sidecar service

version: '3.8'

services:
  app:
    image: myapp:latest
    environment:
      DD_AGENT_HOST: datadog-agent
      SERVICE_NAME: my-app
    depends_on:
      - datadog-agent

  datadog-agent:
    image: gcr.io/datadoghq/agent:latest
    environment:
      DD_API_KEY: ${DATADOG_API_KEY}
      DD_SITE: datadoghq.com
      DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT: "0.0.0.0:4318"
    ports:
      - "4318:4318"

Cloudflare Workers / Vercel Edge

Challenge: Edge runtime, no Node.js APIs Solution: Use autotel-edge (not regular autotel)

import { init, trace } from 'autotel-edge';

init({
  service: 'edge-api',
  endpoint: 'https://otlp.datadoghq.com',
  otlpHeaders: `dd-api-key=${DATADOG_API_KEY}`,
});

export default {
  fetch: trace(async (request) => {
    // Edge function logic
    return new Response('OK');
  }),
};

Multi-Region Deployments

Challenge: Different Datadog sites per region Solution: Configure site via environment variable

// Automatically picks correct site based on deployment region
init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: 'global-api',
  site: (process.env.DATADOG_SITE as any) || 'datadoghq.com',
  // US: datadoghq.com
  // EU: datadoghq.eu
  // AP: ap1.datadoghq.com
}));

Troubleshooting

Traces/Logs Not Appearing

1. Verify API Key

curl -X POST "https://http-intake.logs.datadoghq.com/api/v2/logs" \
  -H "dd-api-key: ${DATADOG_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"message":"test","ddsource":"test"}'

Expected: HTTP 202 Accepted If 403 Forbidden: Invalid API key

2. Check Endpoint URL

Verify the site matches your Datadog region:

• US1: datadoghq.com
• EU: datadoghq.eu
• US3: us3.datadoghq.com

3. Check Application Logs

Enable debug logging:

import { init } from 'autotel';
import { createBuiltinLogger } from 'autotel/logger';

const logger = createBuiltinLogger('my-app', { level: 'debug' });

init({
  service: 'my-app',
  logger, // Autotel will log export errors
  // ...
});

4. Verify Data is Being Sent

Check for OTLP export errors in application logs:

ERROR: Failed to export traces to https://otlp.datadoghq.com/v1/traces

5. Wait for Data Ingestion

Data may take 1-2 minutes to appear in Datadog UI after export.

Agent Not Receiving Data

1. Check Agent Status

sudo datadog-agent status

Look for OTLP section. If missing, OTLP receiver is not enabled.

2. Verify OTLP is Enabled

cat /etc/datadog-agent/datadog.yaml | grep -A 10 otlp

Should show:

otlp_config:
  receiver:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

3. Check Agent Logs

sudo tail -f /var/log/datadog/agent.log

Look for OTLP receiver startup messages.

4. Test OTLP Endpoint

curl -v http://localhost:4318/v1/traces

Expected: Response from OTLP receiver (not “connection refused”)

5. Check Firewall

Ensure port 4318 is open:

sudo netstat -tulpn | grep 4318

Logs Missing Trace Correlation

Symptom: Logs appear in Datadog but don’t show linked traces

Solution: Ensure you’re using createBuiltinLogger() from Autotel:

// Correct - includes trace correlation
import { createBuiltinLogger } from 'autotel/logger';
const logger = createBuiltinLogger('my-app');

// Wrong - no trace correlation
import pino from 'pino';
const logger = pino();

Verify in Datadog:

1. Go to Logs
2. Click any log entry
3. Check for dd.trace_id and dd.span_id fields
4. If present, correlation should work

High Costs

Solution 1: Use Adaptive Sampling

import { AdaptiveSampler } from 'autotel/sampling';

init({
  service: 'my-app',
  sampler: new AdaptiveSampler({
    baselineSampleRate: 0.01,  // 1% of normal traffic
    alwaysSampleErrors: true,   // But always capture errors
  }),
  // ...
});

Solution 2: Use Datadog Agent

Agent batches/compresses data, reducing egress costs significantly.

Solution 3: Filter Noisy Endpoints

import { AdaptiveSampler } from 'autotel/sampling';

const sampler = new AdaptiveSampler({
  shouldSample: (context) => {
    // Don't sample health checks
    const route = context.attributes['http.route'];
    if (route === '/health' || route === '/metrics') {
      return false;
    }
    return true;
  },
});

Advanced Configuration

Custom Resource Attributes

Add deployment-specific attributes:

import { Resource } from '@opentelemetry/resources';

init({
  service: 'my-app',
  endpoint: '...',
  otlpHeaders: '...',
  resource: new Resource({
    'deployment.environment': 'production',
    'service.namespace': 'payments',
    'service.version': '2.1.0',
    'host.name': process.env.HOSTNAME,
    'cloud.provider': 'aws',
    'cloud.region': 'us-east-1',
  }),
});

Custom Sampling Logic

Implement complex sampling rules:

import { AdaptiveSampler } from 'autotel/sampling';

const sampler = new AdaptiveSampler({
  shouldSample: (context) => {
    // Always sample authenticated requests
    if (context.attributes['user.authenticated'] === true) {
      return true;
    }

    // Sample 100% of payments
    if (context.attributes['http.route']?.includes('/payment')) {
      return true;
    }

    // Sample 5% of everything else
    return Math.random() < 0.05;
  },
});

init({
  service: 'my-app',
  sampler,
  // ...
});

Per-Environment Configuration

const config = {
  development: {
    enableLogs: true,
    sampler: new AdaptiveSampler({ baselineSampleRate: 1.0 }), // 100%
  },
  staging: {
    enableLogs: true,
    sampler: new AdaptiveSampler({ baselineSampleRate: 0.5 }),  // 50%
  },
  production: {
    enableLogs: false, // Use existing log pipeline
    sampler: new AdaptiveSampler({ baselineSampleRate: 0.1 }),  // 10%
    useAgent: true,    // Always use Agent in prod
  },
};

const env = process.env.NODE_ENV as keyof typeof config;

init(createDatadogConfig({
  apiKey: process.env.DATADOG_API_KEY!,
  service: 'my-app',
  environment: env,
  ...config[env],
}));

FAQ

Can I use both pino-datadog-transport and Autotel?

Yes! Use pino-datadog-transport for logs and Autotel for traces/metrics. This is the recommended incremental migration strategy. See Strategy 1: Incremental.

Do I need the Agent or can I just use direct ingestion?

Both work. Use direct ingestion for serverless/edge. Use Agent for production services (lower costs, more features). See Architecture Choices.

What’s the difference between Autotel and @datadog/dd-trace?

• @datadog/dd-trace: Datadog-specific tracing library, proprietary protocol
• autotel: Vendor-neutral OpenTelemetry wrapper, OTLP standard

Autotel is better if you want vendor flexibility. Both work with Datadog.

How do I reduce costs?

1. Use Datadog Agent (batches/compresses data)
2. Use adaptive sampling (lower baseline rate)
3. Filter noisy endpoints (health checks, metrics endpoints)

See High Costs troubleshooting.

Can I send logs to Datadog without OTLP?

Yes! You can:

1. Keep your existing log transport (pino-datadog-transport, winston-datadog)
2. Use Autotel for traces/metrics only
3. Set enableLogs: false in config

Traces will still correlate with logs if you use createBuiltinLogger() (it adds trace IDs to logs).

Does this work with Cloudflare Workers / Vercel Edge?

Use autotel-edge instead of autotel. Same API, optimized for edge runtimes.

import { init, trace } from 'autotel-edge';
// ... rest is the same

How do I see infrastructure metrics?

Install the Datadog Agent. It auto-collects CPU, memory, disk, network, and has 500+ integrations (Redis, PostgreSQL, etc.).

Direct cloud ingestion only sends application metrics (traces, custom metrics), not infrastructure.

Can I switch from Datadog to Honeycomb later?

Yes! That’s the benefit of OTLP. Just change the endpoint and otlpHeaders in your config:

// Switch from Datadog to Honeycomb
init({
  service: 'my-app',
  endpoint: 'https://api.honeycomb.io/v1/traces', // Changed
  otlpHeaders: `x-honeycomb-team=${HONEYCOMB_API_KEY}`, // Changed
  // No code changes needed!
});

Example Application

See the complete working example:

Location: example-datadog

Features demonstrated:

• Direct cloud ingestion setup
• Traces with custom attributes
• Logs with automatic trace correlation
• Custom business metrics
• Error handling and capture
• Nested spans
• Environment-based configuration

Run it:

cd apps/example-datadog
cp .env.example .env
# Add your DATADOG_API_KEY to .env
pnpm install
pnpm start

Additional Resources

• Datadog OTLP Documentation
• OpenTelemetry Specification
• Datadog Unified Service Tagging
• Datadog Agent OTLP Configuration
• Autotel GitHub

Support

Autotel Issues: GitHub Issues Datadog Support: Datadog Help

For integration questions, please open a GitHub issue with:

• Your deployment type (serverless, Kubernetes, etc.)
• Architecture choice (Agent vs direct ingestion)
• Configuration snippet (redact API keys!)
• Error messages or unexpected behavior