Five things we keep re-learning.

A client times out waiting for your POST /orders. It retries. Now you have two orders and one very confused customer. The standard fix is a client-generated Idempotency-Key header, a unique row keyed on it, and a short-circuit when the key is seen twice.

That is the version in every blog post, and it is incomplete in three ways that only show up under load.

1. Store the response, not just the key

If you insert the key and then execute the handler, a second request arriving mid-flight finds the key and has nothing to return. It either 500s or, worse, returns an empty 200. The row needs somewhere to keep the finished status code and body, and the replay path returns exactly that.

2. Concurrent retries need a lock, not a check

Two retries landing in the same millisecond both read "key missing" before either writes. The unique constraint saves you from a duplicate row, but one request now gets a constraint error. Take a row lock with SELECT … FOR UPDATE, or write a locked_until timestamp and return 409 for the in-flight request. The client retries; the second attempt gets the stored response.

3. Scope the key

A key is only meaningful per account and per endpoint. Two customers using "1" as a test key should not collide, and the same key on /orders and /refunds should not replay the wrong body. Hash the request body as well: replaying a stored order for a request that changed its amount is a silent data bug, not a safe retry.

# replay path, in full
row = repo.get(key, account_id)
if row is None:            # first time — claim it atomically
    row = repo.insert_locked(key, account_id, body_hash)
if row.request_hash != body_hash:
    raise Conflict("key reused with a different body")
if row.status_code is None:
    raise InFlight("retry in a moment")   # 409
return Response(row.response, status=row.status_code)

Expire keys after 24 hours, not forever — you are protecting against retries, not building an audit log. Those are two different tables with two different retention rules.

Every system eventually needs to write to a database and publish an event. Doing both directly gives you the dual-write problem: the commit succeeds and the publish fails, or the publish succeeds and the commit rolls back. Either way, state and events drift apart with no way to tell which one is wrong.

The transactional outbox is the least glamorous fix that works: write the event into a table in the same transaction as the business change, then have a separate relay worker read that table and publish.

BEGIN;
INSERT INTO orders (id, amount, status) VALUES ($1, $2, 'created');
INSERT INTO outbox (id, topic, payload, created_at)
VALUES ($3, 'order.created', $4, now());
COMMIT;

Both rows commit or neither does. The relay is then free to be dumb: poll for unpublished rows, publish, mark as published. If it crashes halfway, it publishes again on restart.

At-least-once is the whole deal

The relay will duplicate. Not "might" — will, the first time a worker dies between publish and the UPDATE. That is fine, provided consumers deduplicate on an event id. If they do not, you have traded a rare consistency bug for a common duplicate-processing bug, which is usually worse.

When not to use it

Single service, single database, no async consumers? You do not need an outbox. Reaching for one on day one buys you a table and a worker and a monitoring dashboard for a problem you do not have yet. Add it at the moment you add the first cross-service event — not before.

Logs are written for the person debugging at 3am, who is not you, and who has never seen this codebase. That single sentence rules out most of what passes for logging.

One request, one id, every line

Generate a request id at the edge, put it in the context, and emit it on every line — including the ones from the database driver and the HTTP client. Without it, a 40-line log for a single request is indistinguishable from 40 requests each logging one line, and you will spend the incident trying to find out which is which.

Log the decision, not the loop

"processing item 37" in a loop is noise that costs money and hides signal. What you actually need at 3am is the branch that was taken: which retry attempt this is, why the request was rejected, what the fallback returned. Log boundaries and decisions; let metrics handle volume.

Never log the payload

Bodies contain tokens, emails, card fragments and whatever a customer pasted into a form. Log the shape — field names, a hash, a length — not the content. It is trivial to add and nearly impossible to remove once it is sitting in a log store with a 90-day retention policy.

{
  "ts": "2026-08-15T08:22:19.041Z",
  "level": "warn",
  "request_id": "req_9f2c1a",
  "event": "payment.retry",
  "attempt": 3,
  "provider": "stripe",
  "reason": "timeout",
  "will_retry": true
}

Nine fields, one line, everything needed to reconstruct the decision — and nothing that would make a compliance officer wince.

Almost every outage I have caused personally came from a schema change that was correct in isolation: one ALTER TABLE, applied at 11pm, on a table that turned out to be the hot one.

Never change a column in one step

Renaming or retyping a column requires every reader and writer to change at the same moment, which is not a thing that happens. Use expand/contract instead: add the new column, write to both, backfill, switch reads, stop writing the old one, drop it — five deployments, each reversible on its own.

-- deploy 1: expand
ALTER TABLE accounts ADD COLUMN email_norm text;
-- deploy 2: dual write, then backfill in batches
UPDATE accounts SET email_norm = lower(email)
WHERE id IN (SELECT id FROM accounts
             WHERE email_norm IS NULL LIMIT 5000);
-- deploy 3: switch reads · deploy 4: stop writing email
-- deploy 5: contract
ALTER TABLE accounts DROP COLUMN email;

Locks are the real risk, not correctness

In Postgres a seemingly instant ALTER TABLE still needs an exclusive lock, and while it waits, every query behind it waits too. One long-running read and your "instant" migration stops the application for as long as it takes. Set a lock_timeout, retry, and build indexes with CREATE INDEX CONCURRENTLY.

Add NOT NULL last, never first — after the backfill has run and the column is fully populated. Adding it early makes every existing row a violation and forces a full rewrite of the table.

?page=3&per_page=50 looks reasonable until you realise that "page 3" is a promise about the contents of rows 101–150, and the table is being written to the whole time. Insert one row at the top and every page after it shifts by one — which is how a client silently skips a record or sees it twice.

Keyset, not offset

Paginate on a stable, unique sort key instead. Fetch one more row than you return to know whether there is a next page, and hand back an opaque cursor built from the last row's sort values.

SELECT id, created_at
FROM orders
WHERE (created_at, id) < ($1, $2)   -- the cursor
ORDER BY created_at DESC, id DESC
LIMIT 51;                              -- page size + 1

The tuple comparison is what makes ties safe. Sorting by created_at alone breaks the moment two rows share a timestamp, which on a busy table is immediately.

Make the cursor opaque

Base64 the pair, or sign it. Clients that build cursors by hand will eventually build one that is subtly wrong, and you will be debugging their arithmetic instead of your server. If the cursor contains a timestamp, be explicit about the timezone and precision.

The one thing cursors cannot do is jump to page 47. If a UI genuinely needs that, it needs a count and an offset view — accept the inconsistency and document it, rather than pretending offset pagination is stable.