Advanced Features

Deterministic Trace IDs

Generate consistent trace IDs from seeds for correlation with external systems:

import { createDeterministicTraceId } from 'autotel/trace-helpers';

const requestId = req.headers['x-request-id'];
const traceId = await createDeterministicTraceId(requestId);
console.log(`View traces: https://your-backend.com/traces/${traceId}`);

Implementation: Uses SHA-256 hashing to generate consistent 128-bit trace IDs. Works in Node.js and edge runtimes (via crypto.subtle).

Use cases:

Correlate external request IDs with OTel traces
Link support tickets to trace data
Associate business entities (orders, sessions) with observability data

Metadata Flattening

Automatically flatten nested objects into dot-notation span attributes:

import { flattenMetadata } from 'autotel/trace-helpers';
import { trace, withTracing } from 'autotel';

export const processOrder = withTracing({})((ctx) => async (order: Order) => {
  const metadata = flattenMetadata({
    user: { id: order.userId, tier: 'premium' },
    payment: { method: 'card', processor: 'stripe' },
    items: order.items.length,
  });

  ctx.setAttributes(metadata);
  // Results in: metadata.user.id, metadata.user.tier, metadata.payment.method, etc.
});

Features:

Auto-serializes non-string values to JSON
Filters out null/undefined values
Gracefully handles circular references (→ <serialization-failed>)
Customizable prefix (default: 'metadata')

Isolated Tracer Provider

For library authors who want to use Autotel without interfering with the application’s global OTel setup:

import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { setAutotelTracerProvider } from 'autotel/tracer-provider';

// Create isolated provider (don't call provider.register())
const exporter = new OTLPTraceExporter({
  url: 'https://your-backend.com/v1/traces',
});
const provider = new NodeTracerProvider();
provider.addSpanProcessor(new BatchSpanProcessor(exporter));

// Set as Autotel's provider (isolated from global OTel)
setAutotelTracerProvider(provider);

// Now all trace(), span(), instrument() calls use this provider

Important limitations:

Isolates span processing and export only
OpenTelemetry context (trace IDs, parent spans) is still shared globally
Spans from isolated provider may inherit context from global spans

Use cases:

Library code with embedded Autotel
SDKs that need observability without forcing users to configure OTel
Separate span processing for different subsystems
Testing with isolated trace collection

Semantic Convention Helpers

Pre-configured trace helpers following OpenTelemetry semantic conventions:

import { traceDB, traceHTTP, traceMessaging } from 'autotel/semantic-helpers';
import { traceGenAI, recordGenAiUsage } from 'autotel-genai/trace';

// LLM operations (Gen AI semantic conventions) — live in autotel-genai
export const generateText = traceGenAI({
  model: 'gpt-4-turbo',
  operation: 'chat',
  provider: 'openai',
})((ctx) => async (prompt: string) => {
  const response = await openai.chat.completions.create({/* ... */});
  // Records gen_ai.usage.input_tokens / gen_ai.usage.output_tokens
  recordGenAiUsage(ctx, 'gpt-4-turbo', {
    inputTokens: response.usage.prompt_tokens,
    outputTokens: response.usage.completion_tokens,
  });
  return response.choices[0].message.content;
});

// Database operations (DB semantic conventions)
export const getUser = traceDB({
  system: 'postgresql',
  operation: 'SELECT',
  database: 'app_db',
  collection: 'users',
  querySummary: 'SELECT users',
})((ctx) => async (userId: string) => {
  return await pool.query('SELECT * FROM users WHERE id = $1', [userId]);
});

// HTTP client operations (HTTP semantic conventions)
export const fetchUser = traceHTTP({
  method: 'GET',
  urlTemplate: '/users/{id}',
})((ctx) => async (userId: string) => {
  const url = `https://api.example.com/users/${userId}`;
  ctx.setAttribute('url.full', url);
  const response = await fetch(url);
  ctx.setAttribute('http.response.status_code', response.status);
  return response.json();
});

// Messaging operations (Messaging semantic conventions)
export const publishEvent = traceMessaging({
  system: 'kafka',
  operation: 'publish',
  destination: 'user-events',
})((ctx) => async (event: Event) => {
  await producer.send({ topic: 'user-events', messages: [event] });
  ctx.setAttribute('messaging.message.id', event.id);
});

Benefits:

Automatic semantic attributes following OTel specs
Stable low-cardinality span names and correct client/producer/consumer kinds
Direct traceDB(config, fn) / traceHTTP(config, fn) forms when context is not needed
Type-safe configuration interfaces
Reduces boilerplate by 60-70%
Links to official OTel semantic convention docs in JSDoc

Available helpers:

traceGenAI() (from autotel-genai/trace): Gen AI operations (chat, completion, embedding)
traceDB(): Database operations (SQL, NoSQL, Redis)
traceHTTP(): HTTP client requests
traceMessaging(): Queue/messaging operations (Kafka, RabbitMQ, SQS)

Event-Driven Observability

First-class support for message-based systems with traceProducer and traceConsumer helpers:

import { traceProducer, traceConsumer } from 'autotel/messaging';

// Producer - auto-sets SpanKind.PRODUCER and semantic attributes
export const publishEvent = traceProducer({
  system: 'kafka', // kafka | sqs | rabbitmq | custom
  destination: 'user-events',
  messageIdFrom: (args) => args[0].id, // Extract message ID
})((ctx) => async (event: Event) => {
  const headers = ctx.getTraceHeaders(); // W3C traceparent/tracestate
  await producer.send({
    topic: 'user-events',
    messages: [{ value: event, headers }],
  });
});

// Consumer - auto-sets SpanKind.CONSUMER, extracts links from headers
export const processEvent = traceConsumer({
  system: 'kafka',
  destination: 'user-events',
  consumerGroup: 'event-processor',
  headersFrom: (msg) => msg.headers, // Extract trace headers
  batchMode: true, // For batch consumers
})((ctx) => async (messages) => {
  // Links to producer spans automatically created
  for (const msg of messages) await process(msg);
});

For KafkaJS eachBatch, use withBatchConsumer from autotel-plugins/kafka for batch-level and optional per-message spans with trace continuation from headers.

Key implementation details:

Uses SpanKind.PRODUCER / SpanKind.CONSUMER for proper trace visualization
ctx.getTraceHeaders() returns { traceparent, tracestate? } for header injection
ctx.recordDLQ(dlqName, reason) for dead-letter queue tracking
Supports lag metrics via lagMetrics.getCurrentOffset / getEndOffset
Automatic semantic attributes: messaging.system, messaging.destination.name, messaging.operation, messaging.consumer.group

Safe Baggage Propagation

Type-safe baggage schemas with built-in guardrails:

import {
  createSafeBaggageSchema,
  BusinessBaggage,
} from 'autotel/business-baggage';

// Pre-built schema for common fields
BusinessBaggage.set(ctx, {
  tenantId: 'acme',
  userId: 'user-123',
  priority: 'high',
});
const { tenantId, priority } = BusinessBaggage.get(ctx);

// Custom schema with validation and guardrails
const OrderBaggage = createSafeBaggageSchema(
  {
    orderId: { type: 'string', maxLength: 36 },
    customerId: { type: 'string', hash: true },
    tier: { type: 'enum', values: ['free', 'pro', 'enterprise'] as const },
  },
  {
    prefix: 'order', // Keys: order.orderId, order.tier
    redactPII: true, // Auto-redact email/phone/SSN patterns
    hashHighCardinality: true, // Hash UUIDs/timestamps
  },
);

Guardrails:

Size limits: maxKeyLength (default 64), maxValueLength (default 256)
PII detection: Regex patterns for email, phone, SSN auto-redacted
High-cardinality hashing: UUIDs and timestamps hashed via FNV-1a
Enum validation: Rejects values not in the defined set
Type coercion: Numbers/booleans properly serialized

Workflow & Saga Tracing

Track distributed workflows with compensation support:

import { traceWorkflow, traceStep } from 'autotel/workflow';

export const orderSaga = traceWorkflow({
  name: 'OrderSaga',
  workflowId: (order) => order.id,
})((ctx) => async (order) => {
  await traceStep({
    name: 'ReserveInventory',
    compensate: async (ctx, error) => {
      await inventoryService.release(order.items); // Rollback
    },
  })((ctx) => async () => {
    await inventoryService.reserve(order.items);
  })();

  await traceStep({
    name: 'ChargePayment',
    linkToPrevious: true, // Link to ReserveInventory span
    compensate: async (ctx, error) => {
      await paymentService.refund(order.id);
    },
  })((ctx) => async () => {
    await paymentService.charge(order);
  })();
});
// If ChargePayment fails, compensations run in reverse order

Key features:

traceWorkflow creates root span with workflow.name, workflow.id attributes
traceStep creates child spans with workflow.step.name, workflow.step.index
linkToPrevious: true creates span links for step sequencing
Compensations run in reverse order on failure
Workflow context: ctx.getWorkflowId(), ctx.getWorkflowName()
Step context: ctx.getStepName(), ctx.getStepIndex(), ctx.getWorkflowContext()
WeakMap-based state isolation tied to span lifecycle

Distributed Workflows Across Services

autotel/workflow uses AsyncLocalStorage, so it stays inside one process. When a workflow spans microservices, autotel/workflow-distributed carries the workflow identity over W3C baggage instead.

import {
  traceDistributedWorkflow,
  traceDistributedStep,
} from 'autotel/workflow-distributed';

// Service A starts the workflow
export const createOrder = traceDistributedWorkflow({
  name: 'OrderFulfillment',
  workflowIdFrom: (order) => order.id,
})((ctx) => async (order) => {
  await publishToInventory(order); // workflow baggage rides the outbound headers
  return { workflowId: ctx.workflowId };
});

// Service B continues it
export const reserveInventory = traceDistributedStep({
  name: 'ReserveInventory',
  extractBaggage: true,
})((ctx) => async (message) => {
  await reserveItems(message.items); // ctx.workflowId matches Service A
});

createWorkflowHeaders() and parseWorkflowFromBaggage() give you manual control when the transport is not HTTP. getWorkflowProgress() and isInDistributedWorkflow() read the current state.

Async Callbacks: the Parking Lot

A webhook or payment callback can land hours after the request that triggered it. autotel/webhook stores the originating trace context under a correlation key and links the later callback span back to it.

import { trace, withTracing } from 'autotel';
import { createParkingLot, InMemoryTraceContextStore } from 'autotel/webhook';

const parkingLot = createParkingLot({
  store: new InMemoryTraceContextStore(),
  defaultTTLMs: 24 * 60 * 60 * 1000,
});

export const initiatePayment = withTracing({})(
  (ctx) => async (orderId: string) => {
    await parkingLot.park(`payment:${orderId}`);
    await stripe.createPaymentIntent({ metadata: { orderId } });
  },
);

export const handleStripeWebhook = parkingLot.traceCallback({
  name: 'stripe.webhook.payment_intent.succeeded',
  correlationKeyFrom: (event) =>
    `payment:${event.data.object.metadata.orderId}`,
})((ctx) => async (event) => {
  // ctx.elapsedMs reports how long the callback took to arrive
  await fulfillOrder(event.data.object);
});

Swap InMemoryTraceContextStore for a Redis-backed store in production so parked context survives a restart.

Correlation IDs

A correlation id is a stable 16-character join key that ties fragmented traces together. autotel/correlation-id generates one at a request boundary and reads it back anywhere in the same async scope.

import {
  generateCorrelationId,
  getCorrelationId,
  runWithCorrelationId,
} from 'autotel/correlation-id';

await runWithCorrelationId(generateCorrelationId(), async () => {
  getCorrelationId(); // available to every call in this async scope
  await handleRequest();
});

See Configuration for wiring correlation ids into init().

Class Method Decorators

autotel/decorators wraps class methods with tracing using TypeScript 5 decorator syntax. The span takes the method name by default.

import { Trace } from 'autotel/decorators';

class OrderService {
  @Trace('order.create', { withMetrics: true })
  async createOrder(data: OrderData) {
    return db.orders.create(data);
  }

  @Trace() // span name: processPayment
  async processPayment(orderId: string) {
    return stripe.charge(orderId);
  }
}

Request Enrichers

autotel/enrichers parses request and response headers into normalized span attributes for user agent, geolocation, and body size.

import { userAgent, geo, requestSize } from 'autotel/enrichers';

const ua = userAgent(req.headers);
if (ua) ctx.setAttribute('user_agent.browser', ua['user_agent.browser']);

const location = geo(req.headers);
if (location?.['geo.country']) {
  ctx.setAttribute('geo.country', location['geo.country']);
}

Error Catalogs

The fourth handler that throws “Card declined” gets its own wording and its own status. Now the same failure reads three ways in your dashboard, and the support runbook covers one of them.

defineErrorCatalog names each failure once and hands back a builder per code:

import { defineErrorCatalog } from 'autotel';

export const billing = defineErrorCatalog('billing', {
  PAYMENT_DECLINED: {
    status: 402,
    message: 'Card declined',
    why: 'The issuer rejected the charge',
    fix: 'Try a different payment method',
    link: 'https://docs.example.com/errors/payment-declined',
  },
  INSUFFICIENT_FUNDS: {
    status: 402,
    message: ({
      available,
      required,
    }: {
      available: number;
      required: number;
    }) => `Insufficient funds: $${available} of $${required}`,
  },
});

throw billing.PAYMENT_DECLINED({ cause: stripeError });
throw billing.INSUFFICIENT_FUNDS({ available: 5, required: 100 });

Each builder produces the same StructuredError that createStructuredError returns, so parseError() on the client keeps working. Codes default to billing.PAYMENT_DECLINED, and message, why, fix, status, and link travel with the entry rather than with the call site.

A message or why written as a function takes typed parameters, and the type flows to every call site. Passing { available: 5 } without required fails to compile.

Catching without magic strings

.match() identifies an error by the catalog entry that built it:

try {
  await charge(order);
} catch (error) {
  if (billing.PAYMENT_DECLINED.match(error)) {
    return retryWithDifferentCard();
  }
  throw error;
}

Matching reads a symbol the builder attached, so renaming an entry breaks the compile rather than skipping the branch in silence. Comparing error.message gives you neither.

autotel map reports repeated inline errors once a project has a catalog, so you find the copies worth folding in.

Drain Pipeline

autotel/drain-pipeline batches async work, retries with backoff, and flushes on an interval. Subscribers and exporters use it internally; reach for it when you build your own sink.

import { createDrainPipeline } from 'autotel/drain-pipeline';

const send = createDrainPipeline<LogEvent>({
  batch: { size: 50, intervalMs: 5000 },
  retry: { maxAttempts: 3, backoff: 'exponential' },
  maxBufferSize: 1000,
})(async (batch) => {
  await logService.send(batch);
});

send(event);
await send.flush();
await send.shutdown();

Retrying only what can succeed

By default every failed attempt gets retried until maxAttempts runs out. A sink that rejects a batch with HTTP 400 rejects it three times, and the last two attempts buy you nothing but latency and log noise.

shouldRetry classifies the failure. Return false and the batch is dropped after the first attempt:

const send = createDrainPipeline<LogEvent>({
  batch: { size: 50, intervalMs: 5000 },
  retry: {
    maxAttempts: 3,
    backoff: 'exponential',
    shouldRetry: (error) => !isClientError(error),
  },
})(async (batch) => {
  await logService.send(batch);
});

The classifier receives the error, the attempt number, and the batch. A classifier that throws stops the retries and surfaces its own error, so a bug in your classification never turns into an infinite loop.

Keep retrying timeouts, connection resets, and 5xx responses. Stop on 4xx, schema rejections, and anything an identical retry cannot fix.

Core Analysis Loop

An alert tells you checkout got slower. It does not tell you which requests. autotel/analysis answers that: give compareCohorts() the events you are investigating plus a comparable baseline, and it ranks the field and value pairs that separate them.

import { compareCohorts } from 'autotel/analysis';

const isSlow = (event) => Number(event['checkout.duration_ms']) >= 800;

const ranked = compareCohorts({
  outlier: events.filter(isSlow),
  baseline: events.filter((event) => !isSlow(event)),
});

console.log(ranked[0]);
// {
//   field: 'payment.provider',
//   value: 'bank-beta',
//   outlierFraction: 1,
//   baselineFraction: 0.33,
//   difference: 0.67,
//   outlierCount: 20,
//   baselineCount: 20,
// }

Any array of flat records works: wide events from getRequestLogger(), TestSpan.attributes from autotel/testing, or rows from a backend query.

Options:

Option	Default	Purpose
`outlier`	required	The events you are investigating
`baseline`	required	A normal population from the same window
`fields`	every field seen	Restrict the scan
`ignoreFields`	none	Skip identifiers you know are unique
`maxValuesPerField`	`50`	Absolute cardinality cap
`maxUniqueRatio`	`0.5`	Skip a field whose values barely repeat
`minDifference`	`0.1`	Drop weak signals
`limit`	`20`	Results returned, strongest first

Why fields get skipped: a field whose values never repeat cannot name a cohort. A request id takes a fresh value on every event, so it would bury the real answer. Two guards remove those fields, an absolute cap for large populations and a ratio for small ones. Numeric fields hit the same guard, so bucket durations and payload sizes when you instrument them:

import { bucket } from 'autotel/analysis';

log.set({ 'payload.size_bucket': bucket(size, [1024, 65_536]) });
// '<1024' | '1024-65536' | '>=65536'

bucket(value, boundaries) sorts the boundaries you pass, so an unordered list still gives you the ranges you meant. A non-finite value and an empty boundary list both return 'unknown'. Filing a NaN duration under the slowest bucket would invent a cohort that never ran.

The ranking is a hypothesis, not a cause. Open two traces from the named cohort and confirm the mechanism before you act on it.

Name the cohorts while you instrument

Deploy timestamps make poor cohort boundaries. You end up reconstructing which requests took the new path from a clock. experiment() records the answer on the span instead:

import { experiment, trace } from 'autotel';

await trace.run('checkout', async () => {
  experiment({
    name: 'checkout-cache',
    variant: useCache ? 'cached' : 'direct',
    expect: 'cached should cut p95 by 200ms',
  });

  return processCheckout();
});

The call stamps experiment.name, experiment.variant and experiment.expectation on the active span, and writes the first two to baggage, so child spans and the services behind this one carry the same answer. Bare baggage keys need init({ baggage: '' }); baggage: true prefixes them.

Call it at any depth inside a traced body. It reads the ambient span, and it does nothing when no trace is running.

Your expectation travels with the result, so a reader a month later sees the claim next to what happened. The devtools Compare view lists the experiments it has seen and fills both cohorts from the arms of the one you pick.

Service Level Objectives

An error rate tells you what is happening now. An error budget tells you whether you can keep spending it until Friday. autotel/slo calculates both from the outcomes you already know.

import { createSloTracker } from 'autotel/slo';

const checkout = createSloTracker({
  name: 'checkout.availability',
  target: 0.99, // 99% of eligible events must succeed
  windowMs: 60 * 60_000, // rolling one-hour window
});

const snapshot = checkout.record(response.ok ? 'good' : 'bad');

Every record() returns the state of the window:

Field	Meaning
`sli`	Observed good ratio. Undefined until the first event.
`budgetConsumed`	Fraction of the permitted failures already spent
`budgetRemaining`	What is left. Negative means overspend.
`burnRate`	Observed failure ratio over permitted failure ratio
`meetsTarget`	Whether the window currently satisfies `target`

A burn rate of 1 spends the budget exactly over the window. A burn rate of 14 spends an hour’s budget in four minutes.

The tracker records an autotel.slo.outcomes counter and an autotel.slo.burn_rate histogram. Pass { recordMetrics: false } when you calculate outside an initialised SDK, and { now } to control the clock in tests.

Alerting on burn rate

One window alerts too late or too often. Google’s SRE workbook pairs a fast window with a slow one, and evaluateBurnRateAlert decides from both:

import { evaluateBurnRateAlert } from 'autotel/slo';

const decision = evaluateBurnRateAlert({
  shortWindow: fastTracker.snapshot(), // 5 minutes
  longWindow: slowTracker.snapshot(), // 1 hour
  shortThreshold: 14,
  longThreshold: 6,
});

if (decision.alerting) page(decision.reason);

Both windows have to breach. The short window catches a real outage in minutes; the long window stops a thirty-second blip from waking anyone. When the decision is quiet, reason names which window held it back.

Forecasting exhaustion

forecast() projects the recent failure rate forward and reports when the budget runs out:

const forecast = checkout.forecast({
  baselineMs: 6 * 60 * 60_000, // estimate from the last six hours
  lookaheadMs: 24 * 60 * 60_000, // project the next day
});

forecast.alerting; // true when the projection exhausts the budget
forecast.timeToExhaustionMs; // when, if it keeps going
forecast.reason; // 'projected-budget-exhaustion' | 'within-budget' | 'no-baseline-traffic'

The lookahead cannot exceed four times the baseline. Projecting a day from five minutes of traffic produces a number with no evidence behind it, so the call throws instead.

Choosing a Sampler

Sampler	Decides on	Use when
`AdaptiveSampler`	Outcome, after execution	Default. Keeps every error and slow request.
`DeterministicSampler`	Hash of a shared key	Several services must agree on one trace.
`KeyTargetRateSampler`	Observed traffic per key	Traffic is skewed and rare operations matter.
`RandomSampler`	Chance, per process	Single service, no cross-process trace to keep whole.
`UserIdSampler`	User identity	Debugging named users or VIP accounts.

Consistent Sampling

RandomSampler rolls the dice in each process, so an API can keep a trace that its worker drops and leave you a waterfall with holes in it. DeterministicSampler hashes a key that travels with the request, so every service reaches the same verdict:

import { DeterministicSampler } from 'autotel/sampling';
import { trace } from '@opentelemetry/api';

const sampler = new DeterministicSampler({
  sampleRate: 0.1,
  key: () => trace.getActiveSpan()?.spanContext().traceId,
});

Per-Key Target Rates

One rate serves a skewed workload badly. A 1% rate floods storage with your busiest endpoint and still loses the rare tenant whose failures you need. KeyTargetRateSampler counts traffic per key over a rolling window, then gives each key its own rate so every key contributes about targetPerKey events. The first window keeps everything, because no traffic history exists yet.

import { KeyTargetRateSampler } from 'autotel/sampling';

const sampler = new KeyTargetRateSampler({
  key: (context) => context.operationName,
  targetPerKey: 10, // ~10 events per key per window
  windowMs: 30_000,
  maxKeys: 1000, // keys beyond this share one overflow bucket
});

Recording the Sample Rate

A query over a 1% sample reports a hundredth of your traffic, and the number looks plausible. Autotel writes autotel.sampling.rate on the span as “1 in N”, so COUNT * rate estimates the population. The attribute appears only when N exceeds 1, so fully captured spans stay clean.

Implement sampleRate() on a custom sampler to take part:

import type { Sampler } from 'autotel/sampling';

class TenantSampler implements Sampler {
  shouldSample(context) {
    /* ... */
  }
  sampleRate() {
    return 20; // each kept event represents 20
  }
}