Module 5 · Agents and Reliability · 45 min
State, Retries, Idempotency, and Human Approval
Is lesson ke baad tum
Reliability comes from durable state and enforceable invariants: retry safely, execute each intended effect once, and bind human approval to the exact action that will occur.
In this lesson
Test One Approved Action
Test whether a local purchase-order mock refuses an unapproved action and reuses one recorded result on repeat. The supplied Node program needs no account, email provider, or API key. Its sender is a counter, so no order or message leaves your machine.
State records progress; bounded retries repeat only safe operations; idempotency ties attempts to one business action; approval binds an authorized person to the exact payload. The in-memory storage below demonstrates those checks within one process. It does not survive a restart.
Model the workflow as states
Do not represent a job with only done: true/false. Use named states and allowed transitions. A purchase-order assistant might use:
RECEIVED -> VALIDATED -> RESEARCHED -> DRAFTED -> AWAITING_APPROVAL
AWAITING_APPROVAL -> APPROVED -> SENDING -> SENT
AWAITING_APPROVAL -> REJECTED
before an external effect -> FAILED_RETRYABLE or FAILED_FINAL
SENDING -> UNKNOWN_OUTCOME -> reconcile before any further actionStore the job ID, current state, input reference, authenticated requester, workflow version, source versions, tool results, approval record, retry count, timestamps, and idempotency keys. Store only the data needed for the purpose, and define retention. State must survive a process crash; an in-memory variable is useful for a lesson but insufficient for production.
Validate every transition. RECEIVED -> SENT should be impossible. An approval made against draft version 2 should not authorize silently changed version 3. If a reviewer changes a material field, produce a new draft hash and request approval again.
Retries are for temporary failures
Retry network timeouts, rate limits, and transient service errors when the operation is documented as retry-safe. Do not retry invalid input, forbidden access, an exhausted budget, or a policy rejection. Use exponential backoff with jitter so many workers do not retry simultaneously:
delay = min(max_delay, base_delay * 2^attempt) + random_jitterCap attempts and elapsed time. After the cap, move the job to a visible dead-letter or manual-review queue with enough context to diagnose it. An endless retry loop is neither resilient nor cheap.
A timeout is ambiguous: the remote service may have completed the action while your client missed the response. That is why retries and idempotency must be designed together.
Idempotency prevents duplicate effects
An idempotency key identifies one business action, not one HTTP attempt. For example, send-purchase-order:PO-2026-014:v2 represents sending version 2 of one approved purchase order. Store the key and result. If the same webhook arrives twice or a worker restarts, return the recorded result instead of sending again.
Generate the key from stable business identifiers or create it when the job begins. Do not use the current timestamp for every retry; that turns each retry into a new action. For database work, use a unique constraint or transactional outbox pattern rather than hoping two workers do not race.
Worked build: a supplier purchase-order assistant
A Lahore electronics retailer uploads a restock sheet. The workflow validates rows, retrieves approved supplier terms, drafts a purchase order, and asks a purchasing manager to approve. It may never email the supplier before approval.
Save the following as approval-lab.mjs. It demonstrates control logic with in-memory storage. Then append the driver below; never substitute a real email sender for this exercise.
import crypto from "node:crypto";
import assert from "node:assert/strict";
const jobs = new Map();
const effects = new Map();
function hash(value) {
return crypto.createHash("sha256").update(JSON.stringify(value)).digest("hex");
}
function cloneAndFreeze(value) {
const clone = structuredClone(value);
const freeze = (item) => {
if (item && typeof item === "object" && !Object.isFrozen(item)) {
Object.values(item).forEach(freeze);
Object.freeze(item);
}
return item;
};
return freeze(clone);
}
function createJob(id, draft) {
if (jobs.has(id)) throw new Error("Job ID already exists");
const immutableDraft = cloneAndFreeze(draft);
const job = {
id,
state: "AWAITING_APPROVAL",
draft: immutableDraft,
draftHash: hash(immutableDraft),
attempts: 0,
};
jobs.set(id, job);
return structuredClone(job);
}
// authContext must come from trusted server-side authentication middleware,
// never from reviewer fields submitted in request JSON.
function approve(jobId, authContext, reviewedHash) {
const job = jobs.get(jobId);
if (!job || job.state !== "AWAITING_APPROVAL") throw new Error("Invalid state");
if (typeof authContext?.userId !== "string" || !authContext.userId.trim() ||
!Array.isArray(authContext.roles) || !authContext.roles.includes("purchase-approver")) {
throw new Error("Authenticated purchase approver required");
}
const currentHash = hash(job.draft);
if (currentHash !== job.draftHash || reviewedHash !== currentHash) {
throw new Error("Draft changed; review again");
}
job.state = "APPROVED";
job.approval = { reviewerId: authContext.userId, reviewedHash: currentHash, at: new Date().toISOString() };
}
async function sendOnce(jobId, sendEmail) {
const job = jobs.get(jobId);
if (!job || !job.approval) throw new Error("Approval required");
const currentHash = hash(job.draft);
if (currentHash !== job.draftHash || currentHash !== job.approval.reviewedHash) {
throw new Error("Approved payload changed; execution blocked");
}
const key = `send-po:${job.id}:${currentHash}`;
if (effects.has(key)) return effects.get(key);
if (job.state !== "APPROVED") throw new Error("Job is not ready to send");
job.state = "SENDING";
try {
const result = await sendEmail(structuredClone(job.draft), { idempotencyKey: key });
effects.set(key, result);
job.state = "SENT";
return result;
} catch {
job.state = "UNKNOWN_OUTCOME";
throw new Error("Reconcile the outcome before any new send attempt");
}
}Append this driver to the same file:
let mockCalls = 0;
const draft = { recipient: "supplier@example.invalid", item: "DEMO-CABLE", quantity: 2, totalPaisa: 50000 };
const job = createJob("demo-1", draft);
const approver = { userId: "reviewer-demo", roles: ["purchase-approver"] };
const mockSend = async (payload, options) => {
mockCalls += 1;
assert.equal(payload.recipient, "supplier@example.invalid");
assert.ok(options.idempotencyKey.startsWith("send-po:demo-1:"));
return { providerId: "MOCK-1", status: "simulated" };
};
await assert.rejects(sendOnce(job.id, mockSend), /Approval required/);
assert.throws(() => approve(job.id, { userId: "guest", roles: [] }, job.draftHash));
assert.throws(() => approve(job.id, approver, "wrong-hash"), /Draft changed/);
approve(job.id, approver, job.draftHash);
const first = await sendOnce(job.id, mockSend);
const second = await sendOnce(job.id, mockSend);
assert.deepEqual(second, first);
assert.equal(mockCalls, 1);
console.log({ mockCalls, state: jobs.get(job.id).state, result: first });This illustrates the invariants, but it still has a production gap: the process could crash after sendEmail succeeds and before effects.set. A provider that enforces the same idempotency key across retries can prevent a repeated effect within its documented key scope and retention window. Store and reuse the key durably; merely passing a parameter does not establish that protection. Otherwise, use a durable outbox and reconcile ambiguous sends rather than blindly trying again.
Design an approval people can actually review
An approval screen must show the proposed action, recipient, amount or scope, evidence, assumptions, policy flags, and what will happen after approval. “Approve agent output?” is not informed review. The reviewer should see a readable draft and a machine-stable hash or version behind it.
Record:
- authenticated reviewer identity and role;
- exact draft version reviewed;
- approval or rejection time;
- any comment or required change;
- policy basis for requiring approval;
- the final execution result.
Approvals should expire when the underlying facts can change. A stock order approved on Monday should not remain executable after quantities or prices change on Friday. Define expiry and revalidation rules.
Use approval before irreversible or high-impact actions: sending messages to external people, publishing content, changing permissions, deleting data, committing code, placing orders, issuing refunds, or submitting official forms. For low-risk drafts, post-action review may be enough. Match the control to consequence.
OpenAI's agent guidance supports resumable approval flows and guardrails, but an SDK does not decide your business policy. You still define which tool calls pause, who may approve, and how the approved payload is bound to execution.
Failure cases to force
A webhook arrives three times and creates three jobs.
A rate-limit response triggers immediate retries from hundreds of workers.
A send call times out after success and creates a duplicate on retry.
A reviewer approves a summary, but the hidden recipient or amount differs.
The draft changes after approval but retains the approved state.
A rejected job is accidentally picked up by a generic retry worker.
Two workers execute the same approved action concurrently.
An agent writes “approved” in its own output and application code mistakes that for human authorization.
State lives only in conversation history and disappears after compaction or restart.
Logs contain complete supplier banking or personal contact information without need.
Hands-on deliverable
Run approval-lab.mjs with the supplied driver. Expect all assertions to pass and mockCalls to equal 1 after two successful sendOnce requests. The second request must return the first result.
Extend the driver with a new job demo-uncertain using the same synthetic draft. Approve its returned hash. Use a different mock sender that increments a second counter and then throws new Error("simulated lost response"). Catch that failure and inspect the stored state: it must be UNKNOWN_OUTCOME.
Call sendOnce again for that job and assert that it rejects before invoking the mock. The second counter must still equal 1. This represents an ambiguous remote effect: reconcile the provider's record before deciding whether another attempt is safe. Do not change the state to APPROVED just to make the test pass.
Your result proves local control behavior only. On restart, both Maps disappear. A real implementation needs transactional durable state, a stable business key, provider idempotency where supported, concurrency control, approval expiry/revalidation, and reconciliation. A mock cannot prove those production properties.
Completion rubric
- Unapproved, wrong-role, and wrong-hash attempts fail before the mock sender runs.
- Two approved send requests return one stored result with one mock invocation.
- A thrown response records
UNKNOWN_OUTCOME; a second call cannot invoke the mock again. - You explain the restart/crash gap and the missing production controls without claiming exactly-once delivery.
Sources
4 reference sources. Open the source to check the details.
Practice quiz
Check yourself
4 questions · practise what you just learned
1 / 4 · key_idea
The approval-lab driver approves the correct hash and calls sendOnce twice successfully in the same process. What should the two calls produce?
Apna kaam check karo.
Practice complete ki? Result ko example aur lesson ke checks se compare karo.
This is your own completion record, not a graded assessment.