Metrics

cronpad’s metrics use OpenTelemetry names (dot-separated). The Prometheus exporter converts them to Prometheus conventions: dots become underscores and the unit becomes a suffix, so cronpad.run.start_lag (milliseconds) is exposed as cronpad_run_start_lag_milliseconds_bucket, _sum and _count. Labels are deliberately low-cardinality: there is never a per-job or per-workspace label. Per-job numbers are in the dashboard and the stats API.

Runs

Metric	Type	Labels	Description
`cronpad.run.start_lag`	histogram (ms)	`sink`, `trigger`	Scheduled time → first attempt started. The SLA number.
`cronpad.run.queue_wait`	histogram (ms)		Claimed → started by a worker.
`cronpad.run.sink_duration`	histogram (ms)	`sink`, `outcome`	Duration of each attempt.
`cronpad.run.e2e`	histogram (ms)	`sink`, `status`	Scheduled time → final attempt finished, including retries.
`cronpad.runs`	counter	`sink`, `status`	Runs reaching a final or deferred state: `succeeded`, `failed`, `deferred`, `skipped`, `cancelled`.
`cronpad.runs.sla_breached`	counter	`sink`	Runs whose start lag exceeded their SLA threshold.
`cronpad.attempts`	counter	`sink`, `outcome`	Attempts by `success`, `retryable`, `fatal`, `timeout`.
`cronpad.duplicates_suppressed`	counter		Redeliveries of already-finished runs that were skipped.

Histogram buckets (ms): 1, 5, 10, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000. cronpad.results.batch_size instead uses command-count buckets 1, 10, 50, 100, 250, 500. Observable queue gauges are sampled every five seconds.

Scheduler

Metric	Type	Labels	Description
`cronpad.dispatcher.concurrency_deferred`	counter		Claims deferred 250ms by workspace concurrency caps; no workspace label.
`cronpad.dispatcher.claimed`	counter		Occurrences claimed.
`cronpad.dispatcher.tick_duration`	histogram (ms)		Time per claim-loop iteration.
`cronpad.dispatcher.backpressure`	counter		Iterations where all workers were busy. Rising = add workers.
`cronpad.dispatcher.lock_held`	gauge (0/1)		Whether this process is the active dispatcher.
`cronpad.queue.due_size`	gauge		Entries in the `due` queue (future and due).
`cronpad.queue.inflight_size`	gauge		Entries currently leased.
`cronpad.queue.oldest_due_age`	gauge (ms)		How long the oldest due-but-unclaimed entry has waited. The best single “falling behind” signal.
`cronpad.leases.expired`	counter		Leases that expired and were redelivered. Should be 0.
`cronpad.leases.lost`	counter		Workers that found their lease taken while still running.
`cronpad.reconciler.repaired`	counter	`kind`	Queue entries restored from the database (`lost_member`, `lost_retry`, `lost_inflight`, `full_sweep`).
`cronpad.reconciler.backlogged`	counter	`kind`	Overdue jobs that were simply waiting in the queue (load, not loss).

Storage and background work

Metric	Type	Labels	Description
`cronpad.results.batch_size`	histogram	`db`	Writes per database transaction.
`cronpad.results.flush_duration`	histogram (ms)	`db`	Time per batched transaction.
`cronpad.janitor.deleted`	counter	`table`	Rows removed by retention.
`cronpad.janitor.duration`	histogram (ms)		Time per pruning pass.
`cronpad.queue.errors`	counter	`op`	Queue file failures (claim, lease renewal, lock). Called `cronpad.redis.errors` before Redis was removed.
`cronpad.db.errors`	counter	`op`	Database errors.

Jobs and alerts

Metric	Type	Labels	Description
`cronpad.jobs`	gauge	`state`	Jobs by state.
`cronpad.jobs.auto_disabled`	counter		Jobs disabled for consecutive failures.
`cronpad.alerts.sent`	counter	`channel`, `status`	Alert deliveries by channel kind and result.

Suggested alerts

groups:
  - name: cronpad
    rules:
      - alert: CronpadFallingBehind
        expr: cronpad_queue_oldest_due_age_milliseconds > 30000
        for: 2m
      - alert: CronpadWorkersDying
        expr: increase(cronpad_leases_expired_total[10m]) > 0
      - alert: CronpadSLABreaches
        expr: sum(rate(cronpad_runs_sla_breached_total[15m])) / sum(rate(cronpad_runs_total[15m])) > 0.01
        for: 15m
      - alert: CronpadNotDispatching
        expr: max(cronpad_dispatcher_lock_held) == 0
        for: 1m

The shipped alert rules also include CronpadNotReady, using a blackbox-exporter probe of /readyz with job label cronpad-readyz.