AI video API dead-letter queue video job recovery submission uncertainty idempotency concurrent claims

AI video API dead-letter queue: recover without duplicate spend

Nov 6, 2026·11 min read

Your video creation request times out, the worker retries, and two generations appear later. An AI video API dead-letter queue should prevent this escalation by parking ambiguous work for reconciliation, not automatically creating another paid generation. A dead-letter queue, or DLQ, is an application-controlled collection of jobs removed from normal processing because their next safe action requires investigation or a bounded recovery policy.

The practical fix is to classify failures by stage, preserve evidence, and require an explicit decision before any new paid submission. A failed status check is not a failed generation. A failed download is not a reason to generate again. This guide provides a local state machine you can test before connecting your application to a video service.

What belongs in an application-owned dead-letter queue?

Store recovery records for your application's jobs, rather than assuming the video provider offers a DLQ with your business rules. Application-owned means your database controls classification, review, recovery permissions, and audit history; a broker may transport notifications, but it does not decide whether spending is authorized.

For a GenRelay integration, start with the video API overview and verify current integration documentation separately. The design here describes application behavior, not a GenRelay endpoint, parameter, or service guarantee.

A useful record contains:

Your stable business operation ID and a distinct submission attempt ID.
The failure stage, last observation time, and sanitized diagnostic evidence.
A provider job identifier when one was actually received.
A request fingerprint, meaning a digest of the normalized intended request.
Polling and delivery attempt counts, next eligible time, and retry deadline.
A spending reservation reference, review decision, reviewer identity, and recovery action.

Retain source asset references with access controls and expiry information. Avoid copying credentials, signed download URLs, or sensitive prompts into unrestricted queue messages. Choose a retention period that supports investigation without retaining customer content indefinitely.

How should submission uncertainty differ from polling and delivery failures?

Submission uncertainty requires reconciliation; polling and delivery failures usually require recovering an existing job or asset. Reconciliation means comparing recorded evidence with an authoritative job or billing record to determine what actually happened.

Failure class	What you know	Next permitted action
Submission uncertain	A request may have been accepted, but its response is missing	Investigate; do not automatically submit again
Polling exhausted	A provider job ID exists, but status retrieval has exceeded your policy	Resume bounded status checks against that ID after review
Delivery exhausted	Generation completion was observed, but transfer or notification failed	Retry the existing asset transfer or notification after review
Generation failed	An authoritative observation reports failure	Review billing, inputs, and whether a new attempt is justified

A timeout proves that your client did not receive a timely answer. It does not prove that generation failed or that no charge occurred. Likewise, a temporary missing-job response may need investigation against the documented API behavior rather than immediate replacement.

Define separate retry policies for status checks, downloads, and customer notifications. Use backoff, an increasing delay between attempts, and jitter, a randomized delay component, to avoid synchronized recovery traffic. Honor documented rate limits and server retry instructions. For broader incident handling, see AI video API job recovery.

How do reviewed recovery and concurrent claims prevent duplicates?

Require a recorded review decision and an atomic claim before a worker starts a recovery action. An atomic claim is a state change that succeeds for one competing worker while rejecting other workers observing the same available task.

Idempotency means repeating an operation has the same intended effect as performing it once. Locally, enforce a unique operation ID to prevent duplicate queue records and a unique recovery action ID to prevent repeated decisions from creating multiple tasks. Provider-side idempotency is a separate capability whose documented scope and retention period must be checked; a locally generated key cannot establish remote deduplication by itself.

Use this review sequence:

Inspect sanitized request evidence, status observations, and accounting records.
If an accepted job is discovered, attach its identifier and recover that job.
If acceptance remains unknown, keep the record parked and preserve its spending hold.
If a fresh generation is justified, record explicit approval, authorize a fresh budget, and create a distinct submission attempt linked to the original.

Do not turn the fourth step into a generic retry button. A reviewer should see the potential duplicate cost and the evidence supporting another submission. An accepted request, a terminal failure, and a billing reversal are different observations.

For multiple machines, put claims and review transitions in database transactions with conditional updates. A lease, which grants temporary ownership, helps recover crashed workers. A fencing token, which increases with each claim, lets downstream state updates reject stale workers. Neither mechanism makes an external paid request atomic with a database commit; ambiguous sends must still enter reconciliation.

Can you test the recovery state machine without calling an API?

Yes: the following executable Python example tests local decision gates, duplicate registration, and competing claims without network calls or paid submissions. A state machine is a model that permits specific transitions between named states and rejects others.

Save the block as dlq_demo.py and run python3 dlq_demo.py. It uses the Python standard library. The lock models atomic transitions inside one process; it is not a replacement for durable database coordination.

from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
from threading import Lock
import unittest


@dataclass
class Job:
    stage: str
    provider_id: str = ""
    state: str = "parked"
    action: str = ""
    owner: str = ""


class Queue:
    stages = {"uncertain", "poll", "delivery", "failed"}

    def __init__(self):
        self.jobs = {}
        self.audit = []
        self.lock = Lock()

    def park(self, operation, stage, provider_id=""):
        if stage not in self.stages:
            raise ValueError("unknown stage")
        if stage in {"poll", "delivery"} and not provider_id:
            raise ValueError("existing job identifier required")
        with self.lock:
            if operation in self.jobs:
                old = self.jobs[operation]
                if (old.stage, old.provider_id) != (stage, provider_id):
                    raise ValueError("conflicting registration")
                return False
            self.jobs[operation] = Job(stage, provider_id)
            return True

    def reconcile(self, operation, provider_id, reviewer, evidence):
        if not all((provider_id, reviewer, evidence)):
            raise ValueError("identifier and review evidence required")
        with self.lock:
            job = self.jobs[operation]
            if job.state != "parked" or job.stage != "uncertain":
                raise ValueError("not awaiting reconciliation")
            job.provider_id = provider_id
            job.stage = "poll"
            self.audit.append((operation, reviewer, evidence, "reconciled"))

    def approve(self, operation, action, reviewer, evidence):
        if not reviewer or not evidence:
            raise ValueError("review required")
        with self.lock:
            job = self.jobs[operation]
            allowed = {"poll": "resume_poll", "delivery": "retry_delivery"}
            if job.state != "parked" or allowed.get(job.stage) != action:
                raise ValueError("unsafe or duplicate decision")
            job.action = action
            job.state = "ready"
            self.audit.append((operation, reviewer, evidence, action))

    def claim(self, operation, worker):
        if not worker:
            raise ValueError("worker identity required")
        with self.lock:
            job = self.jobs[operation]
            if job.state != "ready":
                return False
            job.state, job.owner = "claimed", worker
            return True

    def finish(self, operation, worker, success):
        with self.lock:
            job = self.jobs[operation]
            if job.state != "claimed" or job.owner != worker:
                raise ValueError("not the current owner")
            # 'done' completes this recovery task, not the video generation.
            job.state = "done" if success else "parked"
            job.owner = ""


class RecoveryTests(unittest.TestCase):
    def test_uncertainty_requires_reconciliation(self):
        q = Queue()
        q.park("order-a", "uncertain")
        for action in ("submit_again", "resume_poll"):
            with self.assertRaises(ValueError):
                q.approve("order-a", action, "reviewer", "timeout")
        self.assertFalse(q.claim("order-a", "worker"))
        q.reconcile("order-a", "known-job", "reviewer", "matched record")
        q.approve("order-a", "resume_poll", "reviewer", "job confirmed")
        self.assertTrue(q.claim("order-a", "worker"))

    def test_duplicate_and_conflicting_registration(self):
        q = Queue()
        self.assertTrue(q.park("order-b", "poll", "job-b"))
        self.assertFalse(q.park("order-b", "poll", "job-b"))
        with self.assertRaises(ValueError):
            q.park("order-b", "poll", "different-job")

    def test_review_and_concurrent_claims(self):
        q = Queue()
        q.park("order-c", "delivery", "job-c")
        with self.assertRaises(ValueError):
            q.approve("order-c", "retry_delivery", "", "asset found")
        q.approve("order-c", "retry_delivery", "reviewer", "asset found")
        with self.assertRaises(ValueError):
            q.approve("order-c", "retry_delivery", "reviewer", "duplicate")
        with ThreadPoolExecutor(max_workers=8) as pool:
            results = list(pool.map(
                lambda n: q.claim("order-c", str(n)), range(8)))
        self.assertEqual(sum(results), 1)
        owner = q.jobs["order-c"].owner
        with self.assertRaises(ValueError):
            q.finish("order-c", "outsider", True)
        q.finish("order-c", owner, False)
        self.assertFalse(q.claim("order-c", "next-worker"))
        q.approve("order-c", "retry_delivery", "reviewer", "storage repaired")
        self.assertTrue(q.claim("order-c", "next-worker"))
        q.finish("order-c", "next-worker", True)
        self.assertEqual(q.jobs["order-c"].state, "done")

    def test_new_generation_is_not_a_recovery_action(self):
        q = Queue()
        q.park("order-d", "failed")
        with self.assertRaises(ValueError):
            q.approve("order-d", "submit_again", "reviewer", "failure seen")


if __name__ == "__main__":
    unittest.main()

The deliberate omission is a submission executor. Approved polling or delivery work cannot accidentally become a paid generation through this interface. Production code should authenticate reviewers rather than trust caller-supplied names, persist audit events transactionally, and handle worker crashes. Successful polling can still report a running video; dispatch its next status check instead of marking the business operation complete.

How should budget reservations survive a dead-letter event?

Keep uncertain generation commitments reserved until accounting evidence supports settlement or release. A budget reservation is an application-side hold against available spending; moving a record into a DLQ does not cancel a remote job or erase its possible charge.

Using the supplied prices dated November 6, 2026, Omni Flash costs $0.10/generation at 720p and $0.15/generation at 1080p. Accidentally repeating 20 billable generations would add $2.00 or $3.00 respectively. These are arithmetic examples, not claims about refund behavior or whether a failed request is billed.

Grok 1.0 is $0.010/sec, and Grok 1.5 is $0.022/sec. If a selected configuration supports a hypothetical ten-second billable duration, one generation calculates to $0.10 or $0.22. Twenty additional generations at that supported billable duration calculate to $2.00 or $4.40. Confirm accepted durations, billing increments, and applicable charges before reserving funds; requested playback length may differ from billable duration.

Track the original commitment separately from any reviewed replacement. A recovery action that checks status should not automatically authorize a generation budget. The budget reservations guide explains the accounting boundary in more detail.

What should an operator inspect before releasing parked work?

Inspect failure stage, evidence freshness, current ownership, and spending exposure before approving recovery. Sort the queue by age and business impact rather than repeatedly restarting every stalled task.

Display queue age by stage, unresolved submission count, review latency, claim conflicts, and repeated delivery failures. Alert on growing uncertainty even when the total queue remains small: a handful of ambiguous submissions can hide duplicate spending or customer-visible delays.

Keep a timeline from initial intent through each observation and review. Record why an operator resumed polling, retried delivery, accepted a terminal failure, or approved a separate generation attempt. Close records after verifying the intended recovery outcome, not merely after a worker receives a message.

What are common implementation questions?

The recurring questions concern ownership, safe retry scope, authorization, and spending holds.

Does a broker dead-letter queue replace the application table?

No. A broker can retain failed messages, but an application table should preserve job identity, billing context, and review decisions across message redelivery. A redelivered message must not independently authorize a new paid request.

Can a polling timeout trigger another generation?

No. Preserve the known job identifier and recover status retrieval according to your bounded policy. If the job outcome remains unknown, escalate the investigation rather than treating a communication failure as evidence that generation never started.

What if a reviewer approves the same recovery twice?

Reject the duplicate decision or return its existing action identifier. Enforce that rule transactionally, then use an atomic worker claim. Recovery execution also needs stage-specific deduplication, particularly when delivering notifications to external systems.

When can an uncertain submission's budget hold be released?

Release it when authoritative evidence and your accounting policy justify doing so. A local timeout, queue expiry, or operator frustration is insufficient evidence. If uncertainty persists, escalate it with an explicit accounting decision and retained audit history.