Skip to main content
One idea at a time

A lease expires when a worker disappears

Worker A claims Maya's email job and then stops. A permanent processing status would leave the job stuck forever. A lease gives the claim an expiry time.

The claim stores a random ownership token and a deadline. Until that deadline, other workers leave the job alone. After the deadline, a worker may claim it with a new token.

Ownership can change​

Consider this timeline:

  1. Worker A receives token claim-a.
  2. A stalls long enough for its lease to expire.
  3. Worker B reclaims the job with token claim-b.
  4. A resumes and tries to mark the job complete.

If the completion update checks only the job ID, A can overwrite B's work. Taskboard checks the ownership token as well.

Illustrative SQL: completion requires current ownership
UPDATE jobs
SET completed_at = now()
WHERE id = $1
  AND lease_token = $2;

If $2 is claim-a, this update changes zero rows after B's claim. Zero rows is a meaningful result. A no longer owns the record and must not report that its claim completed the job.

This check is sometimes called fencing. The database rejects a stale actor's writes because the actor no longer presents current ownership.

The boundary of the fence​

Postgres can enforce that condition on a database update. SMTP cannot inspect Taskboard's lease token. A resumed worker might still send an email even though its eventual completion update fails. A lease therefore improves recovery without providing exactly-once external execution.

Lease duration also creates a tradeoff. A very short lease can expire during normal work. A very long lease delays recovery after a real crash. Taskboard defaults to a sixty-second lease and renews it about every third of that interval while the handler runs. Renewal checks the current token and an unexpired lease. If renewal fails, the worker treats ownership as lost and does not complete the record. SMTP still cannot retract a send already in progress.

Look for the ownership token in Read api/src/schema.ts and the matching completion condition in Read api/src/jobs.ts.

Take away

Expiry makes abandoned work recoverable. A claim token prevents an old worker from changing the current job record, but cannot retract an external send.

Does a failed completion update prove that no email was sent?

No. The external send may already have happened. The update proves only that this worker no longer had permission to complete the database record.

The worker limits each batch to the pool size minus two connections. Those spare connections let lease renewal and control queries run while billing jobs wait on Stripe. Claiming every connection would let leases expire even while the process is alive.