05 / BACKEND · ASYNC PIPELINE
Production-ready document backend
Asynchronous document pipeline built so that a job cannot be silently lost, with failure paths tested rather than described. Invoice extraction is the worked example.
Overview
Asynchronous document pipeline built so that a job cannot be silently lost, with failure paths tested rather than described. Invoice extraction is the worked example.
The challenge
In job queues, work vanishes when Redis goes down, or errors that can never succeed get retried while burning paid calls.
Architecture approach
PostgreSQL as source of truth, Redis and ARQ as transport, a worker that claims each job conditionally in the database.
The solution
Document and job are committed to PostgreSQL first, the enqueue comes after; stale jobs are recovered without consuming an attempt; retryability is a classification, not a counter.
Key capabilities
- Jobs survive a Redis outage
- Retries only for retryable errors
- Migrations tested in CI on real PostgreSQL
- 90% test coverage gate
Technology direction
Project scope
The case study describes the project scope and the technical choices. Open-source projects have public code on GitHub; details of confidential projects are available on request.
Next project