Before You Add a Queue: The Failure Modes You Are Agreeing to Own

Before You Add a Queue: The Failure Modes You Are Agreeing to Own

A queue often enters an architecture diagram as a small rectangle between two services.

The rectangle looks peaceful.

Inside it live duplicates, retries, poison messages, ordering questions, stuck work, partial failure, backpressure, retention, schema evolution, and the awkward moment when nobody knows whether a message was processed.

Queues are useful. I use them. But I do not treat "make it asynchronous" as a free simplification.

My rule is:

Add a queue when asynchronous ownership is part of the product requirement, not merely because a synchronous call feels inelegant.

Before choosing a broker, I want to name the failure modes we are agreeing to operate.

First ask what problem the queue is solving

"Decoupling" is too vague to justify infrastructure.

I ask which of these is actually needed:

• The user should not wait for slow work.
• Work must survive an application restart.
• A temporary dependency outage should not fail the original request.
• Several independent consumers need the same event.
• Producers and consumers need independent scaling.
• Work must be buffered during load spikes.
• Events need retention or replay.
• Delivery crosses a trust or network boundary.
• A database change and an event must be committed consistently.

Different needs lead to different solutions.

A slow email send may need a database-backed job table.

A public event stream consumed by five teams may need a real broker.

A request that must tell the user whether payment succeeded may need to remain synchronous, even if a queue would make the diagram look modern.

Synchronous is a reliability feature when the outcome is immediate

Synchronous work gives the caller a direct answer.

text
request
  -> perform operation
  -> return success or failure

That is valuable when the user must know what happened now.

I keep work synchronous when:

• it is fast enough for the latency budget
• the dependency is part of the immediate product outcome
• failure should be visible to the caller
• retries can be controlled by the caller or idempotency layer
• there is no useful state between accepted and completed

The failure mode is simple: the request fails.

That simplicity should not be discarded casually.

A timeout still creates ambiguity. The server may have completed work after the client stopped waiting. Idempotency keys and status resources may still be needed. But adding a queue does not remove that ambiguity; it moves it.

A database-backed job table is often enough

For one service with one PostgreSQL database, a job table is a strong default.

The application already knows how to secure, back up, observe, and operate the database. The job can participate in the same transaction as the state change that created it.

A minimal schema might look like this:

sql
CREATE TABLE jobs (
    id bigserial PRIMARY KEY,
    kind text NOT NULL,
    payload jsonb NOT NULL,
    status text NOT NULL DEFAULT 'pending',
    available_at timestamptz NOT NULL DEFAULT now(),
    attempts integer NOT NULL DEFAULT 0,
    locked_until timestamptz,
    last_error_code text,
    idempotency_key text,
    created_at timestamptz NOT NULL DEFAULT now(),
    completed_at timestamptz
);

CREATE UNIQUE INDEX jobs_idempotency_key_uidx
ON jobs (idempotency_key)
WHERE idempotency_key IS NOT NULL;

CREATE INDEX jobs_claim_idx
ON jobs (available_at, id)
WHERE status = 'pending';

The payload should contain the minimum durable input needed to perform the job. It should not become a convenient landfill for access tokens, entire user records, or data with no retention decision.

A worker can claim available work using row locks:

sql
WITH candidate AS (
    SELECT id
    FROM jobs
    WHERE status = 'pending'
      AND available_at <= now()
    ORDER BY available_at, id
    FOR UPDATE SKIP LOCKED
    LIMIT 1
)
UPDATE jobs AS job
SET status = 'running',
    attempts = attempts + 1,
    locked_until = now() + interval '5 minutes'
FROM candidate
WHERE job.id = candidate.id
RETURNING job.*;

PostgreSQL explicitly documents SKIP LOCKED as unsuitable for a general-purpose consistent view but useful for multiple consumers accessing a queue-like table. See the current `SELECT` documentation.

The claim transaction should be short. I commit the lease, perform external work outside the transaction, then mark the job complete. Holding a database transaction open while calling an email provider is an inventive way to turn their latency into my lock problem.

A lease means you must recover abandoned work

A worker can die after marking a job running.

The lease makes that state recoverable:

sql
UPDATE jobs
SET status = 'pending',
    locked_until = NULL,
    available_at = now()
WHERE status = 'running'
  AND locked_until < now();

That immediately introduces questions:

• How long should the lease be?
• Can long-running work extend it?
• Can two workers process the same job after a lease expires?
• Is the operation idempotent?
• What evidence distinguishes a retry from duplicate concurrent work?

This is the queue revealing its real shape.

I design the handler as if duplicate processing will happen.

At-least-once is the honest default

Many real queue systems provide at-least-once delivery.

Amazon SQS, for example, documents that a message copy can occasionally be delivered again and tells consumers to be idempotent. See Amazon SQS at-least-once delivery.

A database job worker can duplicate work too:

text
worker performs external action
  -> process crashes before marking job complete
  -> lease expires
  -> another worker retries

The database cannot atomically commit a transaction inside an unrelated email, payment, or webhook provider.

So I ask what duplicate-safe means for the actual side effect.

For a webhook:

• include a stable delivery ID
• sign the request
• expect the receiver to deduplicate
• treat repeated 2xx as success

For a payment:

• use the provider's idempotency key
• persist the provider operation ID
• reconcile ambiguous outcomes before retrying

For an email:

• decide whether an occasional duplicate is acceptable
• use a stable message identity where the provider supports it
• do not pretend an SMTP response is a global exactly-once proof

"Exactly once" can be achieved at a carefully defined boundary. It is not a property I assume across an arbitrary distributed workflow.

Retries are a scheduling policy, not a loop

This is not a retry policy:

go
for {
	if err := send(); err == nil {
		return nil
	}
}

A useful retry policy defines:

• which error classifications are retryable
• maximum attempts or maximum age
• exponential backoff
• jitter
• provider Retry-After handling
• per-destination rate limits
• a dead-letter or terminal-failure state
• operator visibility
• manual replay rules

A simple next-attempt calculation might be:

go
func nextDelay(attempt int, max time.Duration) time.Duration {
	base := time.Second * time.Duration(1<<min(attempt, 10))
	if base > max {
		base = max
	}

	jitter := time.Duration(rand.Int63n(int64(base / 4)))
	return base + jitter
}

The exact formula is less important than bounded, observable behaviour.

I store a stable last_error_code, not only raw prose. That lets me answer:

text
How many webhook jobs are waiting because of rate limiting?
Which destinations repeatedly return permanent 4xx errors?
How old is the oldest retryable job?

Poison messages need a terminal state

Some work will never succeed.

A malformed payload, deleted account, invalid destination, or invariant violation should not circulate forever.

I use a terminal state such as failed with:

• final error classification
• attempt count
• first and last failure time
• safe diagnostic context
• an explicit replay action
• an audit record of manual intervention

A dead-letter queue is not a solution by itself. It is a place where failed work waits for someone who may not know it exists.

The operating model must answer:

• Who reviews terminal failures?
• How are they alerted without paging for every item?
• Can a payload be corrected?
• Is replay safe?
• When is failed data deleted?

Ordering is expensive; ask where it actually matters

People often say a queue must preserve order.

Usually they mean one of several different things:

• all events globally
• events for one tenant
• events for one resource
• state transitions for one aggregate
• user-visible notifications
• retries relative to newer work

Global ordering destroys concurrency and is rarely the real requirement.

Per-resource ordering can often be handled with:

• a partition key
• optimistic version numbers
• sequence numbers
• idempotent state application
• rejecting stale transitions
• serializing work only for the affected resource

I prefer to encode the invariant in the message and consumer rather than rely on arrival order as a hidden law.

json
{
  "event_id": "evt_01J...",
  "aggregate_id": "order_123",
  "aggregate_version": 17,
  "event_type": "order_shipped"
}

A consumer that has already applied version 17 can reject version 16 even if the network delivers it late.

Use an outbox when database state and publishing must agree

A common failure looks like this:

text
commit order
  -> process crashes
  -> publish event never happens

Reversing the order is not better:

text
publish event
  -> database transaction fails
  -> consumers observe an order that does not exist

The transactional outbox pattern writes both the domain change and an outbox row in one database transaction:

sql
BEGIN;

UPDATE orders
SET status = 'paid',
    version = version + 1
WHERE id = $1
  AND status = 'pending';

INSERT INTO outbox (
    event_id,
    aggregate_id,
    event_type,
    payload,
    created_at
)
VALUES (
    $2,
    $1,
    'order_paid',
    $3,
    now()
);

COMMIT;

A publisher later reads the outbox and sends events to the broker.

This closes the "state committed but event forgotten" gap. It does not create exactly-once delivery to consumers. The publisher can still crash after sending and before marking the outbox row published, so event IDs and consumer idempotency remain necessary.

The outbox is valuable because it makes one boundary atomic: the domain database transaction.

A real broker earns its place at larger boundaries

I reach for a broker when one or more of these are true:

• several independently operated consumers need the event
• producers and consumers must scale or deploy independently
• sustained throughput exceeds what the service database should carry
• retention and replay are product requirements
• partitioning and consumer groups are useful
• cross-region or cross-domain delivery is required
• the organization already operates the broker well
• queue semantics are a platform capability rather than a one-service invention

Then the broker is not "extra complexity." It is the correct owner of a real requirement.

But the service still owns:

• message schemas
• compatibility
• idempotency
• retry and dead-letter behaviour
• observability
• backpressure
• privacy and retention
• producer and consumer runbooks

Managed infrastructure reduces maintenance of the servers. It does not remove semantics.

The decision tree I actually use

I ask these questions in order.

Must the caller know the final outcome now?

Keep it synchronous unless the latency or availability model makes that impossible.

Does one service need durable background work?

Use a database-backed job table first.

Must a database change and future publication agree?

Use an outbox, with or without a broker downstream.

Do several independent consumers need a durable event stream?

A broker is likely justified.

Is the only argument "we may need scale later"?

Measure the current workload and define the threshold that would trigger a change. Future scale is not a requirement until it has a shape.

The minimum evidence I want before launch

For any asynchronous workflow, I want to see:

• a stable work or event ID
• an idempotency strategy
• retryable versus permanent error classification
• maximum retry age
• queue depth and oldest-item age
• success, retry, and terminal-failure counts
• lease or visibility-timeout behaviour
• payload schema version
• retention and deletion policy
• a replay procedure
• a test that kills the worker at an awkward point

The awkward kill test is revealing:

text
perform side effect
  -> terminate process
  -> restart worker

What happens next is the system's actual delivery guarantee.

The quiet conclusion

A queue is not merely a way to move work later.

It is a promise that someone will own delayed success, duplicates, retries, stale work, and evidence about what happened.

Sometimes that promise belongs in PostgreSQL. Sometimes it belongs in a managed broker. Sometimes the most reliable queue is no queue at all.

The right choice is the smallest mechanism that makes the required failure behaviour explicit.