# Scheduling engine

cronpad keeps every upcoming fire in a queue table (`due`) ordered by the time it should run. One goroutine, the claim loop, sleeps until the earliest entry is due, then runs a single transaction that moves due entries to a second table (`inflight`) with a lease expiry. Workers deliver the work, renew the lease while they're busy, and remove the entry (scheduling a retry in the same step if needed). A reaper returns expired leases, and a reconciler repairs anything the queue lost, using the databases as the source of truth.

This page is the detailed reference. It's useful if you operate cronpad, want to reason about its failure modes, or want to contribute to it.

## The queue file

The queue is its own SQLite file, separate from `cronpad.db` and `history.db`. By default it is named after the control database and sits beside it: `cronpad.db` gets `cronpad-queue.db`. Set `queue.path` to put it elsewhere.

| Table | Row | Ordered by |
|---|---|---|
| `due` | occurrence ID, when it should run (Unix ms) | `(at, member)` |
| `inflight` | occurrence ID, when its lease expires (Unix ms) | `(at, member)` |
| `meta` | `lock:dispatcher` (token of the process allowed to dispatch, with an expiry), `epoch` (random value written at startup), `heartbeat` (last claim-loop tick) | — |

The file is an index, not a record: everything in it can be rebuilt from `cronpad.db` and `history.db`. That's why it is never backed up, and why a damaged file is moved aside (`…-queue.db.corrupt-<unix time>`) and recreated at startup instead of stopping the server.

### Why a separate file

SQLite allows one writer per file. The queue takes two or three small writes per run, and `cronpad.db` already has a single writer shared by the API and the run recorder. Keeping the queue in its own file gives it its own write lock, and keeps the one file you must not lose small and quiet.

### Occurrence IDs

Each entry is a self-describing string:

```
{job_id}|{scheduled_at_ms}|{attempt}|{job_version}|{trigger}
job_01J9ZK3Q…|1794281400000|0|3|s
```

| Part | Meaning |
|---|---|
| `job_id` | The job. |
| `scheduled_at_ms` | The logical fire time. It stays the same across retries, so lag is always measured from the original schedule. |
| `attempt` | `0` for the first try, `1+` for retries. |
| `job_version` | Incremented on every edit. Entries with an old version are dropped when they come up. |
| `trigger` | `s` scheduled, `m` manual "run now", `r` manual retry. Only `s` entries with attempt `0` advance the schedule. |

Because the ID is deterministic and is the table's primary key, adding the same occurrence twice is harmless: the second insert just updates its time. Every repair path relies on this: when in doubt, re-add.

**Invariant:** every active job has exactly one attempt-0 scheduled entry for its next fire time, either in `due` or in `inflight`, plus any pending retries. The reconciler enforces this.

## The queue operations

Each operation that changes the queue is one SQLite transaction that takes the write lock up front (`BEGIN IMMEDIATE`). Nothing else can write between its steps. That's what makes "read what's due, then move it" safe: two claimers can't take the same entry, and a crash halfway through rolls the whole step back.

### claim

Moves up to `limit` due entries into `inflight` with a lease and returns them with their due times.

```sql
-- in one immediate transaction
DELETE FROM due
 WHERE member IN (SELECT member FROM due WHERE at <= :now ORDER BY at, member LIMIT :limit)
RETURNING member, at;
-- then, for each returned member:
INSERT INTO inflight(member, at) VALUES(:member, :now + :lease)
  ON CONFLICT(member) DO UPDATE SET at = excluded.at;
```

The `(at, member)` index means the cost depends on how many entries are claimed, not on how many jobs exist.

### ack

Finishes an entry and, in the same transaction, schedules its retry if there is one.

```sql
DELETE FROM inflight WHERE member = :member;
DELETE FROM due      WHERE member = :member;  -- in case the reaper re-queued it meanwhile
-- only when the attempt failed and attempts remain:
INSERT INTO due(member, at) VALUES(:retry_member, :retry_at)
  ON CONFLICT(member) DO UPDATE SET at = excluded.at;
```

There is never a moment when a failed run is neither in flight nor queued for retry.

### extend

Workers call this every `lease_duration / 3` while a webhook or command is still running, so long calls don't look abandoned.

```sql
UPDATE inflight SET at = :new_expiry WHERE member = :member;
-- 1 row: still ours.
-- 0 rows: if the reaper had put it back in due and nobody took it yet, move it
-- back to inflight (reclaimed); otherwise it's lost: another worker has it, or
-- it was already finished.
```

A worker that loses its lease can't take back a request it already sent. It finishes, records the attempt, logs a warning and counts `cronpad.leases.lost`. The run record deduplicates the result.

### reap

Returns entries whose lease expired (the worker crashed or hung) to `due`, ready immediately. It's the same move as `claim`, in the other direction: delete expired rows from `inflight` and insert them into `due` with the current time.

### the dispatcher lock

The lock is a row in `meta`. Acquiring it inserts the row, or takes it over only if the previous owner's expiry has passed. Renew and release only touch the row if it still holds this process's token and hasn't expired, so a process can never extend or release a lock it has already lost. The lock lives in the queue file, so a second cronpad started on the same data directory waits in standby instead of dispatching.

Startup recovery also moves every orphaned in-flight entry back to `due` in bounded batches, without waiting for their lease to expire.

## Startup

1. Open the databases and the queue file, and start the HTTP server (the API works while scheduling starts).
2. Acquire the dispatcher lock. If another process holds it, wait in **standby** and retry every second. This prevents two processes from scheduling from the same data directory, for example during an overlapping deploy.
3. **Requeue everything in `inflight`.** Only one process dispatches, and it no longer holds the lock, so anything it left in flight is abandoned. Requeueing now avoids waiting up to a full lease after a crash.
4. Write a fresh `epoch` and run a full reconcile, including [misfire handling](../schedules/misfires-and-overlap.md). A new or replaced queue file is rebuilt here.
5. Start the reaper, reconciler, workers and claim loop.
6. Renew the lock every 5 s. If renewal fails, stop claiming immediately, let workers finish, and go back to step 2.

On `SIGTERM`, cronpad stops claiming, waits up to `shutdown_grace` (30 s) for workers, flushes pending writes, releases the lock and exits.

## The claim loop

```
loop:
  sleep until: the earliest due time | a job was created/edited with an earlier time | shutdown
  if the lock isn't held: continue
  free := free worker slots
  if free == 0: count backpressure; sleep tick_interval; continue
  batch := claim(now, min(claim_batch, free), lease_duration)
  for each entry:
      parse it (malformed entries are acked, logged and counted — never fatal)
      if it's a scheduled first attempt: advance the job to its next occurrence
      hand it to a worker (never blocks: the batch was sized to free slots)
  if the batch was full: loop again immediately
  else: sleep until the next earliest due time (at most tick_interval)
```

Design choices:

- **Claim only what workers can start now.** A claimed entry starts within milliseconds, so lease time measures real work, not queueing, and a backlog stays visible in `due` where metrics can see it.
- **Sleep until the earliest due time.** No busy polling: claiming takes a timer wake-up plus one indexed query. Recording a start then takes a few database commits; the result writer commits as soon as it is free and groups whatever arrives meanwhile into the next transaction (`results_flush_interval = "0s"`, the default; `results_max_batch`, default 500). A single job firing once a second starts 2–4 ms after its scheduled time. At 200 jobs/s (12,000 runs/min) start lag measured 37 ms median / 54 ms p99 on 15 CPUs and 43 ms / 60 ms on two (median of three runs); at 500 jobs/s, 67 ms / 139 ms. The older fixed 5 ms flush window measured about 12 ms for the single job and 47 ms / 80 ms at 200 jobs/s; the 50 ms window before it, about 165 ms median.
- **Advance independently of delivery.** The next occurrence is computed from the *scheduled* time (not "now", so schedules never drift) and queued before delivery in durable mode; opt-in fast mode overlaps these commits and delivery, so a slow endpoint never delays the next run. See [Durable and fast delivery](delivery-guarantees.md#durable-and-fast-delivery).

## Workers

For each entry a worker:

1. Loads the job from cache. If the job is paused, deleted, or the entry's version is old, it drops the entry.
2. Creates the run record (unique per job, scheduled time and trigger). If that run already finished (a redelivery), it acks and stops.
3. Executes the sink with the job's timeout, renewing the lease in the background.
4. Classifies the result and calls `ack`, passing a retry entry when the failure is retryable and attempts remain.
5. Records the attempt and run status, updates the job's failure counter, emits events for alerting, and records metrics.

Every log line written during a run carries `job_id`, `run_id`, `occ`, `attempt`, `workspace_id` and the trace ID. See [Logging](../observability/logging.md).

## Recovery

| What happened | What cronpad does | Effect on your jobs |
|---|---|---|
| Worker crashed or hung | Lease stops being renewed → reaper returns the entry to `due` | Redelivered within `lease_duration` |
| cronpad process crashed | In-flight entries survive in the queue file; the next process requeues them at startup | Redelivered within seconds of restart |
| One queue entry lost | Reconciler's overdue check (every 10 s) finds an active job past its `next_run_at` with no entry, and re-adds it | Late by at most ~40 s, then counted in `cronpad.reconciler.repaired` |
| Queue file deleted, emptied or corrupt | The `epoch` is missing → immediate full rebuild from the databases (a corrupt file is moved aside first) | Future runs fire on time |
| Queue file can't be written (disk full, read-only volume) | Claim loop pauses; `/readyz` reports `queue: false` | Runs start late once it recovers; nothing lost |
| Job edited while queued | Old-version entry is dropped when it comes up | New settings apply from the next fire |

### Why the overdue check is enough

If a job's queue entry disappears, nothing advances the job's `next_run_at` in `cronpad.db`. So that job *must* eventually appear in the indexed query "active jobs whose `next_run_at` is more than `overdue_grace` in the past". The check costs nothing when everything is healthy, and its cost grows with the number of broken jobs, not the total number of jobs. A full sweep of every job still runs at startup, when queue data loss is detected, and every 6 hours.

## Alternatives we didn't use

| Option | Why not |
|---|---|
| Polling `jobs.next_run_at` in `cronpad.db` | Works, but puts a hot query and every lease update on the control database's single writer, competing with the API and run recording. A separate queue file keeps that traffic off it. |
| The queue tables inside `cronpad.db` | Would let an ack and the job update commit together, but the same writer contention applies, and the critical file would churn constantly. |
| An external queue server (Redis, which earlier releases used) | Another service to install, size and secure, for no measurable gain on one node: the SQLite queue matched it within a few milliseconds at every load we tested. Removed. |

Readiness reports database writability, queue availability, dispatcher state (`starting`, `standby`, `active`, `stopped`) and heartbeat freshness. A full worker pool or a blocked result writer pauses claiming; a backwards clock jump is logged for operators.

## Tuning

| Setting | Default | Raise it when | Lower it when |
|---|---|---|---|
| `workers.count` | 64 | Many slow endpoints; `cronpad.dispatcher.backpressure` is rising | Memory-constrained |
| `scheduler.claim_batch` | 500 | Very large bursts at the same second | — |
| `scheduler.lease_duration` | 60s | Endpoints routinely take close to a minute | You want faster redelivery after crashes |
| `scheduler.reconcile_interval` | 10s | — | You want faster repair of lost entries |
| `scheduler.tick_interval` | 100ms | — | Rarely needed; the loop already wakes at the exact due time |

All settings: [Configuration reference](../self-hosting/configuration.md#scheduler).

## Workspace admission

Before handing off a claim, the dispatcher checks workspace suspension, the monthly run quota and the workspace concurrency cap. Suspension and a blocking quota record a skipped occurrence with a `skip_reason` while the schedule keeps advancing. A concurrency cap holds the occurrence in the due queue for 250ms and tries again; it is not skipped and the schedule does not advance twice.

Both delivery modes use these checks. Automatic retries do not use another quota slot. Lateness caused by a cap is visible in history but excluded from SLA breach counts. Misfires and overlap skips also record a `skip_reason` (`misfire`, `previous_run_in_progress`) rather than a delivery error. See [Limits](../platform/limits.md).