Scheduling engine

cronpad keeps every upcoming fire in a queue table (due) ordered by the time it should run. One goroutine, the claim loop, sleeps until the earliest entry is due, then runs a single transaction that moves due entries to a second table (inflight) with a lease expiry. Workers deliver the work, renew the lease while they’re busy, and remove the entry (scheduling a retry in the same step if needed). A reaper returns expired leases, and a reconciler repairs anything the queue lost, using the databases as the source of truth.

This page is the detailed reference. It’s useful if you operate cronpad, want to reason about its failure modes, or want to contribute to it.

The queue file

The queue is its own SQLite file, separate from cronpad.db and history.db. By default it is named after the control database and sits beside it: cronpad.db gets cronpad-queue.db. Set queue.path to put it elsewhere.

The file is an index, not a record: everything in it can be rebuilt from cronpad.db and history.db. That’s why it is never backed up, and why a damaged file is moved aside (…-queue.db.corrupt-<unix time>) and recreated at startup instead of stopping the server.

Why a separate file

SQLite allows one writer per file. The queue takes two or three small writes per run, and cronpad.db already has a single writer shared by the API and the run recorder. Keeping the queue in its own file gives it its own write lock, and keeps the one file you must not lose small and quiet.

Occurrence IDs

Each entry is a self-describing string:

{job_id}|{scheduled_at_ms}|{attempt}|{job_version}|{trigger}
job_01J9ZK3Q…|1794281400000|0|3|s

Because the ID is deterministic and is the table’s primary key, adding the same occurrence twice is harmless: the second insert just updates its time. Every repair path relies on this: when in doubt, re-add.

Invariant: every active job has exactly one attempt-0 scheduled entry for its next fire time, either in due or in inflight, plus any pending retries. The reconciler enforces this.

The queue operations

Each operation that changes the queue is one SQLite transaction that takes the write lock up front (BEGIN IMMEDIATE). Nothing else can write between its steps. That’s what makes “read what’s due, then move it” safe: two claimers can’t take the same entry, and a crash halfway through rolls the whole step back.

claim

Moves up to limit due entries into inflight with a lease and returns them with their due times.

-- in one immediate transaction
DELETE FROM due
 WHERE member IN (SELECT member FROM due WHERE at <= :now ORDER BY at, member LIMIT :limit)
RETURNING member, at;
-- then, for each returned member:
INSERT INTO inflight(member, at) VALUES(:member, :now + :lease)
  ON CONFLICT(member) DO UPDATE SET at = excluded.at;

The (at, member) index means the cost depends on how many entries are claimed, not on how many jobs exist.

ack

Finishes an entry and, in the same transaction, schedules its retry if there is one.

DELETE FROM inflight WHERE member = :member;
DELETE FROM due      WHERE member = :member;  -- in case the reaper re-queued it meanwhile
-- only when the attempt failed and attempts remain:
INSERT INTO due(member, at) VALUES(:retry_member, :retry_at)
  ON CONFLICT(member) DO UPDATE SET at = excluded.at;

There is never a moment when a failed run is neither in flight nor queued for retry.

extend

Workers call this every lease_duration / 3 while a webhook or command is still running, so long calls don’t look abandoned.

UPDATE inflight SET at = :new_expiry WHERE member = :member;
-- 1 row: still ours.
-- 0 rows: if the reaper had put it back in due and nobody took it yet, move it
-- back to inflight (reclaimed); otherwise it's lost: another worker has it, or
-- it was already finished.

A worker that loses its lease can’t take back a request it already sent. It finishes, records the attempt, logs a warning and counts cronpad.leases.lost. The run record deduplicates the result.

reap

Returns entries whose lease expired (the worker crashed or hung) to due, ready immediately. It’s the same move as claim, in the other direction: delete expired rows from inflight and insert them into due with the current time.

the dispatcher lock

The lock is a row in meta. Acquiring it inserts the row, or takes it over only if the previous owner’s expiry has passed. Renew and release only touch the row if it still holds this process’s token and hasn’t expired, so a process can never extend or release a lock it has already lost. The lock lives in the queue file, so a second cronpad started on the same data directory waits in standby instead of dispatching.

Startup recovery also moves every orphaned in-flight entry back to due in bounded batches, without waiting for their lease to expire.

Startup

1. Open the databases and the queue file, and start the HTTP server (the API works while scheduling starts).
2. Acquire the dispatcher lock. If another process holds it, wait in standby and retry every second. This prevents two processes from scheduling from the same data directory, for example during an overlapping deploy.
3. Requeue everything in inflight. Only one process dispatches, and it no longer holds the lock, so anything it left in flight is abandoned. Requeueing now avoids waiting up to a full lease after a crash.
4. Write a fresh epoch and run a full reconcile, including misfire handling. A new or replaced queue file is rebuilt here.
5. Start the reaper, reconciler, workers and claim loop.
6. Renew the lock every 5 s. If renewal fails, stop claiming immediately, let workers finish, and go back to step 2.

On SIGTERM, cronpad stops claiming, waits up to shutdown_grace (30 s) for workers, flushes pending writes, releases the lock and exits.

The claim loop

loop:
  sleep until: the earliest due time | a job was created/edited with an earlier time | shutdown
  if the lock isn't held: continue
  free := free worker slots
  if free == 0: count backpressure; sleep tick_interval; continue
  batch := claim(now, min(claim_batch, free), lease_duration)
  for each entry:
      parse it (malformed entries are acked, logged and counted — never fatal)
      if it's a scheduled first attempt: advance the job to its next occurrence
      hand it to a worker (never blocks: the batch was sized to free slots)
  if the batch was full: loop again immediately
  else: sleep until the next earliest due time (at most tick_interval)

Design choices:

• Claim only what workers can start now. A claimed entry starts within milliseconds, so lease time measures real work, not queueing, and a backlog stays visible in due where metrics can see it.
• Sleep until the earliest due time. No busy polling: claiming takes a timer wake-up plus one indexed query. Recording a start then takes a few database commits; the result writer commits as soon as it is free and groups whatever arrives meanwhile into the next transaction (results_flush_interval = "0s", the default; results_max_batch, default 500). A single job firing once a second starts 2–4 ms after its scheduled time. At 200 jobs/s (12,000 runs/min) start lag measured 37 ms median / 54 ms p99 on 15 CPUs and 43 ms / 60 ms on two (median of three runs); at 500 jobs/s, 67 ms / 139 ms. The older fixed 5 ms flush window measured about 12 ms for the single job and 47 ms / 80 ms at 200 jobs/s; the 50 ms window before it, about 165 ms median.
• Advance independently of delivery. The next occurrence is computed from the scheduled time (not “now”, so schedules never drift) and queued before delivery in durable mode; opt-in fast mode overlaps these commits and delivery, so a slow endpoint never delays the next run. See Durable and fast delivery.

Workers

For each entry a worker:

1. Loads the job from cache. If the job is paused, deleted, or the entry’s version is old, it drops the entry.
2. Creates the run record (unique per job, scheduled time and trigger). If that run already finished (a redelivery), it acks and stops.
3. Executes the sink with the job’s timeout, renewing the lease in the background.
4. Classifies the result and calls ack, passing a retry entry when the failure is retryable and attempts remain.
5. Records the attempt and run status, updates the job’s failure counter, emits events for alerting, and records metrics.

Every log line written during a run carries job_id, run_id, occ, attempt, workspace_id and the trace ID. See Logging.

Recovery

Why the overdue check is enough

If a job’s queue entry disappears, nothing advances the job’s next_run_at in cronpad.db. So that job must eventually appear in the indexed query “active jobs whose next_run_at is more than overdue_grace in the past”. The check costs nothing when everything is healthy, and its cost grows with the number of broken jobs, not the total number of jobs. A full sweep of every job still runs at startup, when queue data loss is detected, and every 6 hours.

Alternatives we didn’t use

Readiness reports database writability, queue availability, dispatcher state (starting, standby, active, stopped) and heartbeat freshness. A full worker pool or a blocked result writer pauses claiming; a backwards clock jump is logged for operators.

Tuning

All settings: Configuration reference.

Workspace admission

Before handing off a claim, the dispatcher checks workspace suspension, the monthly run quota and the workspace concurrency cap. Suspension and a blocking quota record a skipped occurrence with a skip_reason while the schedule keeps advancing. A concurrency cap holds the occurrence in the due queue for 250ms and tries again; it is not skipped and the schedule does not advance twice.

Both delivery modes use these checks. Automatic retries do not use another quota slot. Lateness caused by a cap is visible in history but excluded from SLA breach counts. Misfires and overlap skips also record a skip_reason (misfire, previous_run_in_progress) rather than a delivery error. See Limits.