AI video API job recovery: Build a Durable Worker
Your worker submits a video generation request, receives a job identifier, and crashes before updating your database. AI video API job recovery starts with designing for that gap—not with adding another retry loop. A replacement worker must distinguish an existing generation from a request whose acceptance remains unknown.
This architecture tutorial builds a local recovery ledger and a restartable status worker using Python's standard library. The examples run without credentials or network access. They demonstrate application behavior, not a live GenRelay integration: endpoints, response schemas, and provider status names have not been verified here.
Updated October 30, 2026.
What architecture makes video jobs recoverable?
A recoverable architecture separates submission, status observation, and artifact delivery around one durable job record. Durable means committed to storage that survives a process restart; surviving disk loss additionally requires backups or replicated storage.
Use four components:
- Submission service: records intent before contacting the provider.
- Job ledger: stores application identity, provider identity, and recovery decisions.
- Status worker: observes an existing provider job without creating another generation.
- Artifact worker: securely copies completed output into controlled storage.
A provider job ID is the identifier returned for an accepted generation. Keep it alongside the provider account and model context so workers query the correct service. Your application job ID identifies the user's requested operation; it is not interchangeable with the provider identifier.
For a GenRelay deployment, start with the video API overview, then verify the current integration documentation before implementing the network adapter. An adapter translates an external interface into your application's internal states. All states below belong to this tutorial, not to a documented GenRelay contract.
How should the durable ledger represent submission uncertainty?
The ledger should preserve uncertainty explicitly instead of treating a missing provider ID as permission to submit again. Commit a submission-intent record first, perform the network operation second, and commit the returned provider ID immediately afterward.
A crash after acceptance but before persistence leaves an uncertain submission: the generation may exist, but your database cannot prove its identity. A database transaction cannot make an unrelated remote request atomic with a local write.
The following standalone snippet creates that ledger and attaches a clearly fictional provider ID. Save it as ledger_demo.py and run python3 ledger_demo.py. Repeated execution preserves existing records. SQLite's WAL, or write-ahead log, records database changes before incorporating them into the main file.
import sqlite3
import time
DB = "video_jobs.sqlite3"
def connect():
db = sqlite3.connect(DB, timeout=5)
db.execute("PRAGMA journal_mode=WAL")
db.execute("PRAGMA synchronous=FULL")
return db
with connect() as db:
db.execute("""
CREATE TABLE IF NOT EXISTS jobs (
app_id TEXT PRIMARY KEY,
provider_id TEXT UNIQUE,
state TEXT NOT NULL,
created REAL NOT NULL,
owner TEXT,
lease_until REAL NOT NULL DEFAULT 0,
polls INTEGER NOT NULL DEFAULT 0,
next_poll REAL NOT NULL DEFAULT 0,
artifact_ref TEXT
)
""")
db.execute("""
INSERT OR IGNORE INTO jobs(app_id, state, created)
VALUES (?, 'submission_uncertain', ?)
""", ("demo-001", time.time()))
# Intent is committed before any hypothetical remote submission.
# MOCK ONLY: no API call, endpoint, or response schema is used.
mock_provider_id = "mock-provider-job-001"
with connect() as db:
db.execute("""
UPDATE jobs SET provider_id=?, state='observing'
WHERE app_id=? AND state='submission_uncertain'
AND provider_id IS NULL
""", (mock_provider_id, "demo-001"))
print(db.execute(
"SELECT app_id, provider_id, state FROM jobs"
).fetchall())
In production, store a sanitized request digest, account reference, submission timestamp, and any documented correlation identifier. A digest is a hash of the normalized request, useful for comparing intent but not proof that the provider accepted it.
Reconcile uncertain rows using documented lookup facilities or an operator investigation. Do not blindly resubmit. If the provider offers no reliable lookup, keep the operation unresolved until an explicit decision accounts for possible duplicate generation and charges. This design assumes neither native webhook delivery nor provider idempotency support. Idempotency means repeated equivalent requests produce one logical operation; a local unique key alone cannot impose it on a remote service.
How can a replacement worker resume bounded polling safely?
A replacement worker should claim a temporary lease, query the persisted provider ID, and save progress only while it still owns that lease. A lease is a time-limited ownership claim; expiration lets another worker recover abandoned work.
Save this second snippet as recover_demo.py beside the database and run python3 recover_demo.py. Its mock status adapter returns application-defined values. It never calls GenRelay or any other provider. Run it again after completion to confirm that completed jobs are not reclaimed.
import sqlite3
import time
import uuid
DB = "video_jobs.sqlite3"
MAX_POLLS = 6
LEASE_SECONDS = 10
def mock_status(provider_id, attempt):
"""Local simulation; NOT a verified provider response contract."""
if not provider_id.startswith("mock-provider-job-"):
raise ValueError("This demo accepts mock IDs only")
if attempt >= 3:
return "artifact_pending", "mock://video/output.mp4"
return "observing", None
def tick():
owner = uuid.uuid4().hex
db = sqlite3.connect(DB, timeout=5)
db.execute("PRAGMA synchronous=FULL")
try:
now = time.time()
db.execute("BEGIN IMMEDIATE")
row = db.execute("""
SELECT app_id, provider_id, polls FROM jobs
WHERE state='observing' AND provider_id IS NOT NULL
AND lease_until<=? AND next_poll<=?
ORDER BY created LIMIT 1
""", (now, now)).fetchone()
if row is None:
db.commit()
return False
app_id, provider_id, polls = row
if polls >= MAX_POLLS:
db.execute("""
UPDATE jobs SET state='review_required'
WHERE app_id=?
""", (app_id,))
db.commit()
return True
attempt = polls + 1
db.execute("""
UPDATE jobs SET owner=?, lease_until=?, polls=?
WHERE app_id=?
""", (owner, now + LEASE_SECONDS, attempt, app_id))
db.commit()
# No transaction is held during the simulated external read.
state, artifact = mock_status(provider_id, attempt)
finished = time.time()
delay = min(2 ** (attempt - 1), 4)
with db:
changed = db.execute("""
UPDATE jobs SET state=?, artifact_ref=?, next_poll=?,
owner=NULL, lease_until=0
WHERE app_id=? AND owner=? AND lease_until>?
""", (state, artifact, finished + delay,
app_id, owner, finished)).rowcount
print(app_id, state, "committed" if changed else "lease lost")
return True
finally:
db.close()
# Bounded process lifetime, even if no job becomes claimable.
for _ in range(12):
tick()
time.sleep(0.5)
with sqlite3.connect(DB) as db:
print(db.execute(
"SELECT app_id, state, polls FROM jobs"
).fetchall())
The ownership condition prevents an expired worker from overwriting a successor's result. Persisting the poll counter before the observation conservatively counts crashes against the budget. If a process dies after claiming work, restart after lease expiration; the saved provider ID remains unchanged.
The demo uses one local database and wall-clock time. For distributed deployments, use a shared transactional database and its clock to reduce clock-skew errors. Set network deadlines shorter than the lease, or renew ownership safely. Lease protection fences database writes, not external side effects; artifact publication needs its own duplicate-safe design.
Where should polling deadlines and retry policy live?
Polling deadlines and retry budgets should live in durable application state, separate from the provider's execution state. The sample persists a poll budget and next-observation time; production records should also include an overall observation deadline and last classified error.
Use capped exponential backoff—progressively longer waits with an upper limit—and add random jitter to spread worker traffic. Honor documented rate-limit guidance and Retry-After when present. Transport errors may justify another observation, while authentication failures should trigger configuration review. An unrecognized status or missing job must enter investigation, not become an assumed success or failure.
A client timeout means your client stopped waiting. It does not mean the provider cancelled generation, and it does not prove that submission failed. Likewise, exhausting six local observations moves this example toward review; it says nothing about remote completion. The numeric limits here are demonstration settings, not measured provider recommendations.
Keep submission retries distinct from status-read retries. For additional scheduling context, see async video API polling. Track uncertain submissions, overdue observations, lost leases, and pending artifact transfers as separate operational signals.
How should completed artifacts enter application storage?
Completed artifacts should enter application storage through a separate validated transfer stage, not through an unchecked URL fetch inside the status transaction. The sample stops at artifact_pending; its mock address is deliberately not downloadable.
An artifact is the generated media file or a reference to it. Treat returned locations as untrusted input: permit HTTPS and approved hosts, reject private and link-local destinations, and revalidate redirects and resolved addresses. These controls reduce server-side request forgery, where a supplied URL causes your server to contact unintended resources.
Bound download size, transfer time, and decompression behavior. Verify the media format rather than trusting the filename or declared content type. Write into temporary storage, calculate a checksum, then publish the completed object under an application-controlled key. Commit a delivered state only after durable storage succeeds.
Restrict access by tenant, avoid logging signed URLs or credentials, and define retention and deletion policies. A signed URL grants time-limited access through embedded authorization. If it expires, consult documented artifact retrieval behavior instead of starting a new generation. Review current pricing when estimating duplicate-submission exposure; recovery reads and artifact transfers must not be assumed free without verification.
What questions should developers ask before deployment?
Developers should verify acceptance lookup, lease behavior, observation limits, and artifact retention before enabling unattended recovery.
Can I recover a job without its provider job ID?
Sometimes, if documented lookup or account records establish its identity. Without reliable evidence, preserve submission uncertainty and require review rather than guessing or submitting again.
Does restarting a worker restart video generation?
Not in this architecture. Restarting resumes observations of the saved provider ID; neither executable example contains a live generation request.
Does a local unique application ID prevent duplicate charges?
No. It prevents duplicate ledger keys, not duplicate remote acceptance. Charge prevention depends on verified provider behavior and disciplined handling of uncertain submissions.
Should a timed-out job be marked cancelled?
No. Record the local timeout separately. Mark provider cancellation only after a documented cancellation flow and confirmation establish that outcome.