Security Observability

Record security-relevant behaviour with structured hooks and metrics. Emit security.* events at decision points, derive signals from spans you already have, and triage from the CLI. Alert rules live in your OTLP backend.

Scope

• Autotel exports traces, metrics, and structured events. It does not authenticate, authorize, block, or page anyone.
• You call the hooks at auth checks, tenant boundaries, and secret access. Autotel writes security.* attributes and counters.
• Alert rules and dashboards live in your OTLP backend (Prometheus, Datadog, Loki, etc.).

OWASP A09:2025 (Security Logging & Alerting Failures) targets missing logs, unclear messages, and weak alerting. Autotel exports structured, redaction-safe, sampling-exempt security telemetry plus alertable metrics.

The hooks

Explicit (you call at decision points)	Passive (wire once)
securityEvent() / withSecurity()	createSecuritySignalProcessor()
hashIdentifier()	startSecurityHeartbeat()
defineValidator() + onValidationMismatch()	init({ attributeRedactor })
withAudit() (compliance trail)	SecuritySubscriber (routing)

The pieces

Piece	Package	What it gives you
securityEvent() / withSecurity()	autotel-audit	Typed events, stable security.* schema, force-keep through tail sampling, credential-key guard, auto counter
createSecuritySignalProcessor()	autotel-audit	Zero-code signals from HTTP/LLM spans: probe detection, denied-response metrics, auth-failure bursts, token anomalies
startSecurityHeartbeat()	autotel-audit	autotel.security.heartbeat counter; alert on the absence of telemetry
hashIdentifier()	autotel-audit	Correlate emails/IPs across events without logging raw PII
SecuritySubscriber	autotel-subscribers/security	Forward security.* events to a webhook/SIEM/pager, severity-gated
autotel security summary / events	autotel-cli	Incident triage from the terminal, JSON envelope output
Security lens	autotel-devtools	A live Security tab surfacing security.* spans during local development

Installation

npm install autotel-audit

Setup (one-time)

Register the zero-code signal processor and start the heartbeat when you initialise Autotel:

import { init } from 'autotel';
import {
  createSecuritySignalProcessor,
  startSecurityHeartbeat,
} from 'autotel-audit';

init({
  service: 'api',
  spanProcessors: [createSecuritySignalProcessor()],
});

startSecurityHeartbeat();

Then emit events at your security decision points.

Emitting security events

securityEvent() records a typed event on the active trace and request logger. Events are force-kept through tail sampling by default so investigation traces survive sampling. They carry the stable security.* attribute schema your backend can alert on.

import { securityEvent, hashIdentifier } from 'autotel-audit';

securityEvent({
  name: 'auth.login.failed',
  category: 'authentication',
  outcome: 'failure',
  severity: 'warning',
  actorId: hashIdentifier(email),
  reason: 'invalid_password',
});

Wrapping an operation

withSecurity() wraps a security-sensitive operation and records the outcome automatically. A thrown error is recorded as outcome: 'error', the severity is escalated to at least error, and the error is rethrown:

import { withSecurity } from 'autotel-audit';

await withSecurity(
  {
    name: 'api_key.created',
    category: 'secrets',
    outcome: 'success',
    actorId: userId,
  },
  async () => createApiKey(userId),
);

Best-effort by default

Security telemetry is observability — a missing trace context must never crash the business logic it wraps. When there is no active trace context, securityEvent() and withSecurity() no longer throw into your handler. An onMissingContext option (passed via the options arg) controls the behaviour:

• 'warn' (default): record nothing but run the wrapped handler and log one warning per action.
• 'skip': run un-recorded, silently.
• 'throw': previous fail-fast behaviour — opt in when telemetry is mandatory.

await withSecurity(
  { name: 'api_key.created', category: 'secrets', outcome: 'success', actorId: userId },
  async () => createApiKey(userId),
  { onMissingContext: 'skip' },
);

This makes the hooks safe to drop into a production hot path without a surrounding trace() or a try/catch.

Event fields

Field	Notes
name	Stable, dot-separated event name, e.g. auth.login.failed. Suggested names autocomplete; free-form is allowed.
category	authentication, authorization, data_access, admin_action, configuration, secrets, rate_limit, validation, supply_chain, llm
outcome	success, failure, denied, blocked, error
severity	info (default), warning, error, critical
actorId	A stable id or a hashIdentifier() digest — never raw PII
targetType / targetId / tenantId / reason	Optional context

Cardinality

The event name is a metric attribute. Keep names to a stable catalogue. Never interpolate user input into them.

Zero-code signals from existing spans

createSecuritySignalProcessor() watches ordinary HTTP and LLM spans you already produce and derives security signals with no per-call instrumentation:

• Suspicious request paths: traversal, .env/.git probes, SQLi/XSS probes. Flagged at span start, marked security.suspicious_request, and force-kept through tail sampling.
• Denied responses: 401/403/429 by default, counted into autotel.security.http.denied.
• Auth-failure bursts: per-client sliding window, surfaced via the autotel.security.anomaly metric and an onSignal callback.
• LLM consumption anomalies (OWASP LLM10): per-call token ceiling plus an optional sliding-window token budget per consumer.

init({
  service: 'api',
  spanProcessors: [
    createSecuritySignalProcessor({
      // Burst detection over denied responses (defaults shown)
      burst: { statuses: [401, 403], threshold: 10, windowMs: 60_000 },
      // LLM consumption — per-call ceiling on by default; add a windowed budget
      llm: {
        maxTokensPerCall: 100_000,
        tokenBudget: { budget: 1_000_000, windowMs: 300_000 },
      },
      onSignal: (signal) => {
        // Keep this fast and non-throwing; it runs in the span pipeline
        console.warn('[security signal]', signal);
      },
    }),
  ],
});

Detection thresholds and alerting still belong in your backend; the processor makes the raw signals exist and survive sampling.

Alert on the absence of telemetry

If the telemetry pipeline stops, you lose visibility into security events. startSecurityHeartbeat() emits the autotel.security.heartbeat counter on a fixed interval so you can alert on the absence of security telemetry from a service:

const heartbeat = startSecurityHeartbeat({ intervalMs: 60_000 });
// on graceful shutdown:
heartbeat.stop();

The schema

Metrics

Metric	Attributes	Source
autotel.security.events	event, category, outcome, severity	every securityEvent()
autotel.security.http.denied	status	401/403/429 responses
autotel.security.http.suspicious	pattern	probe-path detection
autotel.security.anomaly	signal, (status)	bursts, LLM consumption
autotel.security.heartbeat	custom	liveness

Span attributes

security.event, security.category, security.outcome, security.severity, security.actor_id, security.tenant_id, security.reason, security.suspicious_request, security.signal, and the autotel.security=true marker (plus force-keep markers).

Detection rules (starter pack)

OTLP metric names map to backend-specific forms; Prometheus renders autotel.security.events as autotel_security_events_total.

Prometheus / Grafana (PromQL)

groups:
  - name: security-observability
    rules:
      # 1. Failed-login spike (tune threshold to your baseline)
      - alert: AuthFailureSpike
        expr: >
          sum(rate(autotel_security_events_total{
            event="auth.login.failed"}[5m])) > 1
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "Failed logins above baseline"

      # 2. Any critical security event
      - alert: CriticalSecurityEvent
        expr: >
          sum(increase(autotel_security_events_total{
            severity="critical"}[5m])) > 0
        labels: { severity: critical }

      # 3. Denied-response surge (credential stuffing / scraping)
      - alert: DeniedResponseSurge
        expr: >
          sum(rate(autotel_security_http_denied_total[5m]))
            > 4 * sum(rate(autotel_security_http_denied_total[1h] offset 1h))
        for: 10m
        labels: { severity: warning }

      # 4. Scanner / probe traffic detected
      - alert: SuspiciousRequestProbes
        expr: >
          sum by (pattern) (
            increase(autotel_security_http_suspicious_total[15m])) > 10
        labels: { severity: info }

      # 5. Auth-failure burst anomaly (pre-aggregated in-process)
      - alert: AuthFailureBurst
        expr: >
          sum(increase(autotel_security_anomaly_total{
            signal="auth_failure_burst"}[5m])) > 0
        labels: { severity: warning }

      # 6. LLM consumption anomaly (OWASP LLM10)
      - alert: LlmConsumptionAnomaly
        expr: >
          sum by (signal) (increase(autotel_security_anomaly_total{
            signal=~"llm_.*"}[15m])) > 0
        labels: { severity: warning }

      # 7. Alert when heartbeat stops (telemetry pipeline went dark)
      - alert: SecurityTelemetryAbsent
        expr: >
          absent(rate(autotel_security_heartbeat_total{
            service_name="api"}[5m]))
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "No security telemetry from api; pipeline dead or service compromised"

Datadog (monitor queries)

# Failed-login spike
sum(last_5m):sum:autotel.security.events{event:auth.login.failed}.as_rate() > 1

# Critical security event
sum(last_5m):sum:autotel.security.events{severity:critical}.as_count() > 0

# Telemetry absent (use a metric-absence monitor)
sum(last_10m):sum:autotel.security.heartbeat{service:api}.as_count() < 1

Loki (LogQL, if you route logs there)

# Security events with severity error+ in the canonical log line
{service_name="api"} | json | security_severity=~"error|critical"

# Tenant-boundary violations
{service_name="api"} | json | security_event="access.tenant.violation"

Triage from the terminal

# What happened in the last hour?
autotel security summary --lookback-minutes 60

# Drill into critical events
autotel security events --severity critical --lookback-minutes 240

# Pivot to a full trace from a sampleTraceId
autotel trace summary <traceId>

security summary returns events by severity/category/outcome, top event names, probe signals by pattern, denied responses by status with top clients, and sample trace IDs for pivoting. Output is one JSON document on stdout. See the CLI reference for the full investigate command set.

Routing alerts without a backend

For teams without SIEM plumbing, SecuritySubscriber forwards events directly to a webhook, pager, or custom handler, gated by severity:

import { Events } from 'autotel/events';
import { SecuritySubscriber } from 'autotel-subscribers/security';

const events = new Events('api', {
  subscribers: [
    new SecuritySubscriber({
      webhookUrl: process.env.SECURITY_WEBHOOK_URL!,
      minSeverity: 'error',
    }),
  ],
});

For a custom destination (PagerDuty, OpsGenie, an internal bus), pass a handler instead of webhookUrl:

new SecuritySubscriber({
  handler: async (alert) => pagerduty.trigger(alert),
  minSeverity: 'critical',
});

See Event Subscribers for the subscriber lifecycle and graceful shutdown.

OWASP Top 10 mapping

You instrument the rows below at the relevant decision point. Autotel standardizes the telemetry; your backend runs detection.

OWASP 2025	What Autotel makes visible
A01 Broken Access Control	access.denied, access.tenant.violation events; denied-response metrics
A03 Software Supply Chain	dependency.scan.failed, config.changed events (emit from CI/deploy hooks)
A05 Injection	validation.failed events; SQLi/XSS probe signals from the processor
A07 Auth Failures	auth.* events; auth-failure burst anomalies
A09 Logging & Alerting Failures	the whole feature set, plus the heartbeat for the meta-failure (logging silently stopped)
LLM01 Prompt Injection	llm.prompt_injection.detected events
LLM10 Unbounded Consumption	llm_excessive_tokens / llm_token_budget_exceeded signals

What NOT to log

Caution

The credential-key guard drops values under token/apiKey/password-shaped keys automatically, but it is a backstop, not a license.

Never put in event metadata: secrets (hashed or not), session IDs, raw card/bank data, raw prompts or retrieved context. Use hashIdentifier() for emails and IPs. The core AttributeRedactingProcessor (presets: default, strict) catches value-shaped PII as a second layer.

Testing your security telemetry

In integration tests, assert that security paths emit the expected spans and attributes:

import { createTraceCollector } from 'autotel/testing';

const collector = createTraceCollector();
await attemptLoginWithBadPassword();
const span = collector.expectSpan({ 'security.event': 'auth.login.failed' });
expect(span.attributes['security.severity']).toBe('warning');

Periodically fire a synthetic probe (e.g. request /.env from a canary) to verify the chain from signal through metric, alert, and on-call response.

Related

• Audit Logging: the autotel-audit package this builds on
• Event Subscribers: routing events to external destinations
• CLI: the autotel security investigate commands
• Devtools: the live Security lens for local development