# Metrics

cronpad's metrics use OpenTelemetry names (dot-separated). The Prometheus exporter converts them to Prometheus conventions: dots become underscores and the unit becomes a suffix, so `cronpad.run.start_lag` (milliseconds) is exposed as `cronpad_run_start_lag_milliseconds_bucket`, `_sum` and `_count`. Labels are deliberately low-cardinality: there is never a per-job or per-workspace label. Per-job numbers are in the dashboard and the [stats API](../api/workspace.md#stats).

## Runs

| Metric | Type | Labels | Description |
|---|---|---|---|
| `cronpad.run.start_lag` | histogram (ms) | `sink`, `trigger` | Scheduled time → first attempt started. The SLA number. |
| `cronpad.run.queue_wait` | histogram (ms) | | Claimed → started by a worker. |
| `cronpad.run.sink_duration` | histogram (ms) | `sink`, `outcome` | Duration of each attempt. |
| `cronpad.run.e2e` | histogram (ms) | `sink`, `status` | Scheduled time → final attempt finished, including retries. |
| `cronpad.runs` | counter | `sink`, `status` | Runs reaching a final or deferred state: `succeeded`, `failed`, `deferred`, `skipped`, `cancelled`. |
| `cronpad.runs.sla_breached` | counter | `sink` | Runs whose start lag exceeded their SLA threshold. |
| `cronpad.attempts` | counter | `sink`, `outcome` | Attempts by `success`, `retryable`, `fatal`, `timeout`. |
| `cronpad.duplicates_suppressed` | counter | | Redeliveries of already-finished runs that were skipped. |

Histogram buckets (ms): 1, 5, 10, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000. `cronpad.results.batch_size` instead uses command-count buckets 1, 10, 50, 100, 250, 500. Observable queue gauges are sampled every five seconds.

## Scheduler

| Metric | Type | Labels | Description |
|---|---|---|---|
| `cronpad.dispatcher.concurrency_deferred` | counter | | Claims deferred 250ms by workspace concurrency caps; no workspace label. |
| `cronpad.dispatcher.claimed` | counter | | Occurrences claimed. |
| `cronpad.dispatcher.tick_duration` | histogram (ms) | | Time per claim-loop iteration. |
| `cronpad.dispatcher.backpressure` | counter | | Iterations where all workers were busy. Rising = add workers. |
| `cronpad.dispatcher.lock_held` | gauge (0/1) | | Whether this process is the active dispatcher. |
| `cronpad.queue.due_size` | gauge | | Entries in the `due` queue (future and due). |
| `cronpad.queue.inflight_size` | gauge | | Entries currently leased. |
| `cronpad.queue.oldest_due_age` | gauge (ms) | | How long the oldest due-but-unclaimed entry has waited. The best single "falling behind" signal. |
| `cronpad.leases.expired` | counter | | Leases that expired and were redelivered. Should be 0. |
| `cronpad.leases.lost` | counter | | Workers that found their lease taken while still running. |
| `cronpad.reconciler.repaired` | counter | `kind` | Queue entries restored from the database (`lost_member`, `lost_retry`, `lost_inflight`, `full_sweep`). |
| `cronpad.reconciler.backlogged` | counter | `kind` | Overdue jobs that were simply waiting in the queue (load, not loss). |

## Storage and background work

| Metric | Type | Labels | Description |
|---|---|---|---|
| `cronpad.results.batch_size` | histogram | `db` | Writes per database transaction. |
| `cronpad.results.flush_duration` | histogram (ms) | `db` | Time per batched transaction. |
| `cronpad.janitor.deleted` | counter | `table` | Rows removed by retention. |
| `cronpad.janitor.duration` | histogram (ms) | | Time per pruning pass. |
| `cronpad.queue.errors` | counter | `op` | Queue file failures (claim, lease renewal, lock). Called `cronpad.redis.errors` before Redis was removed. |
| `cronpad.db.errors` | counter | `op` | Database errors. |

## Jobs and alerts

| Metric | Type | Labels | Description |
|---|---|---|---|
| `cronpad.jobs` | gauge | `state` | Jobs by state. |
| `cronpad.jobs.auto_disabled` | counter | | Jobs disabled for consecutive failures. |
| `cronpad.alerts.sent` | counter | `channel`, `status` | Alert deliveries by channel kind and result. |

## Suggested alerts

```yaml
groups:
  - name: cronpad
    rules:
      - alert: CronpadFallingBehind
        expr: cronpad_queue_oldest_due_age_milliseconds > 30000
        for: 2m
      - alert: CronpadWorkersDying
        expr: increase(cronpad_leases_expired_total[10m]) > 0
      - alert: CronpadSLABreaches
        expr: sum(rate(cronpad_runs_sla_breached_total[15m])) / sum(rate(cronpad_runs_total[15m])) > 0.01
        for: 15m
      - alert: CronpadNotDispatching
        expr: max(cronpad_dispatcher_lock_held) == 0
        for: 1m
```

The shipped alert rules also include `CronpadNotReady`, using a blackbox-exporter probe of `/readyz` with job label `cronpad-readyz`.