Our synthetic controller permits one active job and receives three webhook requests together. Checking capacity separately in each handler can let every handler observe the same free slot before any handler reserves it. The limit is a state transition and needs a serialized check and reservation.

We build an independent Node.js fixture that admits one job and queues two others. It persists small invented job manifests in a disposable temporary directory and removes the directory after the test. There are no provider credentials or infrastructure identifiers in those records.

Serialize the check and the reservation

The critical section decides whether capacity is available and records the admission result before another handler makes the same decision. It should not hold the lock across a long virtual machine provisioning request. The reserved slot represents work in progress while the external request happens outside that short section.

In Rust a Tokio mutex can protect the asynchronous admission boundary. The JavaScript fixture uses a promise chain to serialize the same decision. Both designs need every admission path to use the same authority because an unguarded administrative path can bypass the limit even when webhook handlers behave correctly.

function admit(id) {
  return serialized(() => {
    const status = active < 1 ? "reserved" : "queued";
    const job = { id, status, sequence: sequence++ };
    persist(job);
    if (status === "reserved") active++;
    return job;
  });
}

The fixture persists each admission result before returning success to its caller. Its queue order uses an explicit sequence number so directory enumeration order does not become scheduling policy after a restart.

Recovery needs a state model

A queued job has not yet consumed an active execution slot. A provisioning job may already have created an external resource. Automatically treating both as safe to retry can create duplicate infrastructure after a controller restart.

We recover only queued manifests in this example. Other statuses require reconciliation against the external provider before choosing whether to resume or clean up. The fixture reports malformed records separately instead of interpreting them as an empty queue.

We preserve the distinction between queued work and provisioning work because a controller restart can erase process memory without erasing a resource request already accepted by an external provider and blindly replaying every saved job turns a recovery procedure into a second source of duplicate execution that bypasses the original admission limit.

A full state machine needs transitions for cancellation and failure as well as successful completion. Each transition should define who owns any created resource and which retry operation is idempotent. A status label without those rules cannot guide safe recovery.

Persistence has several failure levels

Our fixture writes a temporary file and renames it into place. That prevents a reader from seeing a partially replaced file during an ordinary process restart on the same filesystem. It does not claim power loss durability because it does not synchronize file and directory metadata to storage.

A database can supply transactional admission and recovery state across multiple controller processes. A process local mutex cannot enforce a global limit when two controllers run independently. The coordination boundary must match the scope of the capacity promise.

1 reserved; 2 queued; both queued jobs recovered in order
7 assertions passed

The example starts three admissions concurrently and checks that exactly one is active. It then reads the queued manifests as a fresh recovery pass and verifies their order. An intentionally malformed manifest produces an explicit recovery error while the two valid jobs remain recoverable.

Acknowledgement is another boundary

A webhook sender can retry when it does not receive a successful acknowledgement. Deduplication therefore needs an identity for the intended operation and should distinguish a repeated delivery from a legitimate later update. Returning success before recording the job creates a window where work can vanish after acknowledgement.

The fixture gives jobs unique symbolic identifiers but does not implement webhook signature verification or deduplication. Those checks belong before accepting untrusted input as an executable job. The recovered manifest must also be validated before its fields become shell arguments or infrastructure parameters.

The uncertain provisioning state deserves its own failure drill. Stop the controller after a provider accepts a request but before the local status becomes active and reconcile the job using a recorded operation identifier before permitting any retry.

Sources and examples