Designing reliable AI workflows: queues, retries, and idempotency

An AI feature is only one step in a product workflow. A customer uploads a document, a model produces a draft, a person approves it, and a notification leaves the system. The model can work perfectly while the product still loses a job, sends the same email twice, or tells a customer that work failed when it actually completed.

Consider a fictional document-review application. Its job is to prepare a summary, wait for approval, then notify the requester. The architecture below is a design exercise, not a client case study or a performance claim. Its purpose is to make failure states visible before they become customer support incidents.

Define what accepted means

Start with a contract the interface can explain. “Accepted” means that the system durably recorded a valid request. It does not mean that a summary is ready or that a notification reached an inbox. Return a stable job identifier, show a status page, and separate queued, running, awaiting approval, completed, and failed states.

For this example, the invariant is precise: each approved report version should produce at most one logical notification. A second intentional notification needs a new operation identifier. Editing a report while reusing an old identifier would blur two different actions. This small product decision determines the database keys, worker behavior, and recovery controls.

Commit the intention alongside the record

A database transaction groups related writes into one atomic change. PostgreSQL’s transaction tutorial explains why other transactions do not see the intermediate, uncommitted state. Use that boundary to save the report change and its pending outbox entry together.

A worker later reads that pending entry and performs the remote action. This is the transactional outbox pattern: recording an intention in the same transaction avoids a database commit followed by a forgotten dispatch when the application crashes. It still permits duplicate dispatch attempts, so the consumer must handle repeats.

Keep claim, attempt, and outcome timestamps. With several workers, claim jobs atomically and use an expiring lease so an interrupted worker does not own a job forever. A durable queue can replace an outbox polling worker, but it must preserve the same recovery contract. An in-memory array cannot provide that durability.

Treat a timeout as an unknown outcome

Suppose the provider accepts a notification and the network drops its response. The worker observes a timeout; the customer may already have a delivery in progress. Immediately repeating the action with a new identifier could create a duplicate. Marking it permanently failed could also be misleading.

An idempotent operation has the same intended effect when repeated. RFC 9110, section 9.2.2 distinguishes that property from simply trying a request again. A POST endpoint does not acquire it automatically. A stable application key only helps if the receiving implementation enforces a corresponding contract.

Use a unique database constraint for each logical local operation and a stable provider key for its remote effect. A local “sent” flag alone cannot close the gap between a remote success and a crash before the flag is saved. If the provider cannot safely replay or reconcile an uncertain action, send it to human review instead of promising exactly-once delivery.

Spend a finite retry budget

Retries should target transient failures and remain bounded. The Amazon Builders’ Library describes how backoff reduces repeated pressure and jitter spreads synchronized attempts. Configure a timeout, an attempt limit, and a maximum job age; do not let a failed dependency create unlimited work.

One possible delay policy is full jitter. After failed attempt n, with n starting at zero, draw the next delay uniformly between zero and a capped exponential ceiling:

Here d₀ is the initial ceiling and dₘₐₓ is the largest permitted delay. Jitter spreads attempts; it does not guarantee recovery. Validation errors need correction rather than repetition. Honor applicable provider retry instructions and rate limits, and exhaust the budget into an explicit review or failure state.

Run the lost-response experiment

Run the JavaScript cell below. The mock provider accepts its first request and then loses the reply. The retry keeps the same key and retrieves the existing delivery. A duplicate queue entry is skipped, while an invalid request stops immediately. Delays are printed rather than awaited, so the demonstration finishes quickly.

// A deterministic simulation: no network requests or real waiting.
const jobs = ["report-042", "report-042", "invalid-report"];
const accepted = new Map(); // Mock provider's idempotency ledger.
const completed = new Set();
const calls = new Map();
const maxAttempts = 3;
let seed = 12345;
function random() {
  seed = (seed * 16807) % 2147483647;
  return seed / 2147483647;
}
function mockDeliver(jobId, key) {
  if (jobId === "invalid-report") {
    throw Object.assign(new Error("Invalid recipient"), {retryable: false});
  }
  const attempt = (calls.get(key) || 0) + 1;
  calls.set(key, attempt);
  if (!accepted.has(key)) {
    accepted.set(key, "delivery-" + String(accepted.size + 1).padStart(3, "0"));
  }
  if (attempt === 1) {
    // The provider accepted the action, but its reply disappeared.
    throw Object.assign(new Error("Reply timed out"), {retryable: true});
  }
  return accepted.get(key); // Same key returns the existing delivery.
}
for (const jobId of jobs) {
  if (completed.has(jobId)) {
    console.log(jobId + ": duplicate job skipped");
    continue;
  }
  const key = jobId + ":notify:v1";
  for (let n = 0; n < maxAttempts; n++) {
    try {
      const deliveryId = mockDeliver(jobId, key);
      completed.add(jobId);
      console.log(jobId + ": completed as " + deliveryId);
      break;
    } catch (error) {
      if (!error.retryable) {
        console.log(jobId + ": failed validation; do not retry");
        break;
      }
      if (n === maxAttempts - 1) {
        console.log(jobId + ": needs review; outcome remains uncertain");
        break;
      }
      const ceilingMs = Math.min(8000, 500 * 2 ** n);
      const delayMs = Math.floor(random() * ceilingMs);
      console.log(jobId + ": " + error.message +
        "; next attempt after " + delayMs + " ms (simulated)");
    }
  }
}
console.log("Actual mock deliveries:", accepted.size);

The final count should be one actual mock delivery. Try moving key creation into the attempt loop and adding the attempt number; inspect how that breaks the provider’s deduplication boundary. This is an educational simulation: its maps disappear on reload, it has no concurrency control, and its mock idempotency ledger has no expiry. Production implementations need durable storage and tested provider semantics.

Keep model output inside a controlled workflow

Save a model result before starting downstream actions. A retry that generates a fresh answer can change the document after someone approved an earlier version. Record the input version, model configuration, generated artifact, and approval decision as separate facts. Route consequential or ambiguous decisions to a person with enough context to review them.

Keep private model and delivery credentials on the server. OWASP’s secrets-management guidance recommends controlled access and a managed secret lifecycle. For this workflow, give each worker only the permissions it needs and avoid storing tokens or sensitive document contents in routine logs.

Measure the handoffs, then test recovery

Monitor the age of the oldest pending job, not just the number of HTTP errors. Track attempts per logical operation, lease expiry, time spent awaiting approval, and unresolved outcomes. Attach a correlation identifier to the report, outbox row, and provider response so an operator can reconstruct one journey without exposing its full contents.

Test deliberately: interrupt a worker after provider acceptance, replay a webhook, deliver the same job twice, let a lease expire, and exhaust the retry budget. For each case, check both the stored state and the visible customer experience. A useful recovery screen explains what is known, what remains uncertain, and which actions are safe to take next.

The practical goal is a workflow that remembers its commitments. Durable intentions, explicit states, stable operation identifiers, and limited retries turn a collection of API calls into a system that an operator can inspect and recover. That foundation matters whether the expensive step is an AI model, document conversion, or an ordinary notification.