Observability

Structured logs, metrics, traces, and alerts. Single pane of glass across all providers.

Logs

Every agent emits structured, level-tagged logs (debug, info, warn, error). Logs are streamed in real time via WebSocket and persisted to PostgreSQL for querying. Filter by session, agent, level, or free-text search.

logs.ts

// Stream logs in real time via WebSocket
const stream = theazo.logs.stream({
  sessionId: 'ses_...',
  level: 'error',        // only errors
})

for await (const entry of stream) {
  console.log(entry.timestamp)  // ISO 8601
  console.log(entry.level)      // 'error'
  console.log(entry.agentId)    // 'agt_...'
  console.log(entry.message)    // structured log message
}

SSE streaming

In addition to WebSocket, logs can be consumed via Server-Sent Events (SSE) at GET /v1/agents/:id/logs/stream. SSE is simpler for browser-based dashboards — no WebSocket library needed, just a native EventSource. The server emits two event types: log (with level, message, and timestamp) and heartbeat (keep-alive every 15s).

sse-logs.ts

// Stream logs via SSE (browser-native, no dependencies)
const source = new EventSource(
  'https://api.theazo.com/v1/agents/agt_.../logs/stream',
  // Note: auth header requires polyfill (e.g. eventsource-polyfill)
  // or pass token as query param: ?token=thz_...
)

source.addEventListener('log', (event) => {
  const entry = JSON.parse(event.data)
  console.log(entry.level)      // 'info' | 'warn' | 'error' | 'debug'
  console.log(entry.message)    // structured log message
  console.log(entry.timestamp)  // ISO 8601
  console.log(entry.agentId)    // 'agt_...'
})

source.addEventListener('heartbeat', () => {
  // Connection alive — no action needed
})

// Clean up when done
source.close()

Querying historical logs

query-logs.ts

const logs = await theazo.logs.query({
  sessionId: 'ses_...',
  agentId: 'agt_...',
  level: 'warn',
  search: 'timeout',
  since: '2025-05-01T00:00:00Z',
  until: '2025-05-08T00:00:00Z',
  limit: 100,
})

Metrics

Theazo collects platform-level metrics you can query for dashboards and alerting. Key metrics include agents.active, agents.boot_time, cost.total, and error_rate.

metrics.ts

const metrics = await theazo.metrics.query({
  metric: 'agents.active',
  interval: '1h',           // 1m, 5m, 1h, 1d
  since: '2025-05-01',
  until: '2025-05-08',
})

// Returns a percentile time series:
// [
//   { timestamp: '2025-05-01T00:00:00Z', p50: 12, p95: 18, p99: 22 },
//   ...
// ]

Per-agent cost breakdown

Use theazo.usage.forUser(userId, opts) to get a detailed cost breakdown for a user. The response breaks down spending into models (LLM inference: calls, tokens, cost), compute (sandbox: minutes, cost), storage, and a combined total. All amounts are integer cents. In BYOI compute mode compute.cost returns { amount: 0 } because the AgentCo pays the provider directly.

per-agent-cost.ts

const usage = await theazo.usage.forUser('user_123', { period: 'month' })

console.log(usage.models.cost)   // { amount: 18, currency: 'usd' } — model spend
console.log(usage.compute.cost)  // { amount: 24, currency: 'usd' } — sandbox compute
console.log(usage.total)         // { amount: 42, currency: 'usd' } — total

// BYOI: with your own compute credentials, Theazo doesn't bill compute
const byoiUsage = await theazo.usage.forUser('user_byoi', { period: 'month' })
console.log(byoiUsage.models.cost)   // { amount: 18, currency: 'usd' }
console.log(byoiUsage.compute.cost)  // { amount: 0, currency: 'usd' }
console.log(byoiUsage.total)         // { amount: 18, currency: 'usd' }

Traces

In orchestrator mode (Full Platform and Primitives-only), Theazo creates per-tool-call spans within each agent run. Every model call, tool invocation, and file operation is captured as a span with timing and metadata. In infra-only mode, traces are limited to compute-level events (boot, exec, destroy).

traces.ts

const traces = await theazo.traces.list({ agentId: 'agt_...' })

for (const trace of traces) {
  console.log(trace.traceId)     // 'trc_...'
  console.log(trace.durationMs)  // total duration in ms
  console.log(trace.cost)        // { amount, currency }
  // Fetch per-span detail with theazo.traces.get(trace.traceId)
}

Trace waterfall visualization

Traces render as a waterfall timeline in the dashboard, showing every operation in an agent run as a horizontal bar on a shared time axis. Each span represents one of four operation types: model call, tool execution, file I/O, or sandbox boot. Spans carry a name, duration, start/end timestamps, status (ok or error), and arbitrary metadata. Nested spans show parent-child relationships — for example, a model call that triggers a tool execution which performs file I/O.

Use theazo.traces.get(traceId) to fetch a full trace with nested spans:

trace-waterfall.ts

const trace = await theazo.traces.get('trc_...')

console.log(trace.traceId)     // 'trc_...'
console.log(trace.agentId)     // 'agt_...'
console.log(trace.durationMs)  // total duration in ms
console.log(trace.cost)        // { amount, currency }

for (const span of trace.spans) {
  console.log(span.name)        // 'model.call' | 'tool.exec' | 'file.read' | 'sandbox.boot'
  console.log(span.startTime)   // ISO 8601
  console.log(span.endTime)     // ISO 8601
  console.log(span.durationMs)  // ms
  console.log(span.attributes)  // { model: 'claude-sonnet', tokens: 1847, ... }
}

The dashboard renders trace waterfalls as a visual timeline — click any agent run to see the full span breakdown with timing, status, and metadata for each operation.

Alerts

Theazo fires alerts for conditions that need attention. Alerts appear in the dashboard and can trigger webhooks.

Failed agents — an agent threw an unrecoverable error or exceeded max retries.
Cost limit warnings — a session has used 80% or more of its maxCost budget.
Long-running agents — an agent has been running for more than 30 minutes without completing.

alert-webhook.json

// Subscribe to alerts via webhook
// POST https://yourapp.com/webhooks/theazo
{
  "event": "agent.failed",
  "data": {
    "agentId": "agt_...",
    "sessionId": "ses_...",
    "error": "Max retries exceeded",
    "duration": 847,
    "cost": { "amount": 42, "currency": "usd" }
  }
}

Observability by mode

The depth of observability data depends on which mode you operate in. Full Platform and Primitives-only modes provide the richest data because Theazo controls the agent loop. Infra-only mode provides compute-level metrics only.

Capability

Full Platform

BYOI Primitives

Infra-only

Structured logs

Full

Compute only

Tool-call spans

Yes

Model token tracking

Yes

Compute metrics

Yes

Alerts

Full

Failures only

Observability works across all providers — one dashboard for E2B, Fly, and Docker. Logs, metrics, and traces are normalized into a single schema regardless of the underlying compute backend.

Method reference

theazo.logs.stream(filters)AsyncIterable<LogEntry>Stream logs in real time via WebSocket. Filter by session, agent, level.

theazo.logs.query(filters)Promise<LogEntry[]>Query historical logs with filters and search.

theazo.metrics.query(opts)Promise<MetricDataPoint[]>Query a metric time series (p50/p95/p99) by name, interval, and time range.

theazo.traces.list(filters)Promise<Trace[]>List traces (traceId, agentId, durationMs, cost) for an agent or session.

theazo.traces.get(traceId)Promise<TraceDetail>Fetch a single trace with all its spans (name, timing, attributes).

GET /v1/agents/:id/logs/streamSSE streamStream logs via Server-Sent Events. Event types: log, heartbeat.

theazo.usage.forUser(userId)Promise<UserUsage>Per-user cost breakdown: model, compute, total, and billing mode.