The Brokerless Worker: Architecting Simplified, Transactional Python Background Jobs with Oban

In the modern landscape of Python development, the default answer to "background jobs" is almost reflexively "Celery and Redis." While this stack is battle-tested, it introduces significant operational complexity: a separate broker to manage, synchronization issues between the application database and the job queue, and the dreaded "dual-write" problem where a transaction commits but the job fails to enqueue (or vice versa).

But what if you could eliminate the broker entirely? What if your database was your queue?

Enter the Brokerless Worker architecture. Popularized by Oban in the Elixir ecosystem, this pattern leverages the power of PostgreSQL to handle job scheduling with transactional integrity. For Python developers—especially those working in data-heavy or AI environments where Python is king—adopting this architecture offers a path to simplified infrastructure and atomic reliability. In this post, we explore how to architect simplified, transactional background jobs by applying the Oban philosophy to the Python ecosystem.

The Hidden Cost of the External Broker

To understand the value of a brokerless architecture, we must first dissect the friction points of the traditional setup. In a standard microservices or monolithic architecture using Celery, you have your primary datastore (PostgreSQL) and your message broker (Redis or RabbitMQ).

This separation creates a distributed system problem inside even the simplest applications. Consider a user registration flow where you need to send a welcome email:

• Step 1: Insert user into Postgres.
• Step 2: Enqueue email task to Redis.

If Step 1 succeeds but Step 2 fails (network blip, Redis is full), you have a user with no welcome email. If you reverse the order, you might send an email to a user who failed to commit to the database. This is the lack of Transactional Integrity.

"The network is not reliable. By decoupling the queue from the database, you are opting into distributed system problems that you may not need to solve."

Furthermore, maintaining a highly available Redis cluster adds DevOps overhead. You need monitoring, persistence configuration, and scaling strategies distinct from your primary database. By removing the broker, you collapse the stack, reducing the surface area for failure.

The Oban Philosophy: The Database is the Queue

Oban revolutionized background jobs in the Elixir/Phoenix world by proving that modern PostgreSQL is fast enough to handle job queueing for all but the most massive scales (think millions of jobs per minute). The core philosophy relies on the Transactional Outbox Pattern.

In this architecture, jobs are simply rows in a table (e.g., oban_jobs) within your application database. This unlocks ACID compliance for your background tasks. When you wrap your business logic and your job enqueueing in a single database transaction, they succeed or fail together atomically.

with transaction():
    user = create_user(email)
    enqueue_job("send_welcome_email", user.id)

If create_user fails, the job is never inserted. If enqueue_job fails, the user creation rolls back. This guarantees consistency without complex two-phase commit protocols.

While Oban is an Elixir library, its architectural pattern is universal. For Python developers, this means we can leverage tools that utilize PostgreSQL's SKIP LOCKED functionality to create performant, brokerless workers that rival the throughput of Redis-based systems while maintaining strict data integrity.

Implementing the Pattern in Python

So, how do we bring this reliability to Python? You have two primary architectural paths: Polyglot Integration or Native Adoption.

1. Polyglot Integration (The Hybrid Approach)
If your infrastructure already utilizes Elixir for orchestration, you can use Oban to manage the queue table, while Python workers (running as sidecars or separate services) poll the database for specific job types. By connecting to the same PostgreSQL instance, Python scripts can lock rows in the oban_jobs table, execute heavy AI/ML workloads, and update the job status. This allows you to use Elixir's superior concurrency for IO-bound management and Python's rich ecosystem for CPU-bound processing.

2. Native Adoption (Procrastinate & Custom SQL)
For pure Python shops, libraries like Procrastinate act as the spiritual successor to Oban. Procrastinate uses PostgreSQL specific features (LISTEN/NOTIFY and SELECT FOR UPDATE SKIP LOCKED) to achieve the same brokerless benefits.

Here is what the architecture looks like in practice:

• Job Table: A table storing arguments, scheduled execution time, and status (scheduled, processing, completed, discarded).
• Producers: Your web app inserts rows into this table within standard transactions.
• Consumers: Python workers poll the table using SKIP LOCKED. This SQL clause allows multiple workers to concurrently select the next available job without blocking each other—a feature that makes Postgres a surprisingly high-performance queue.

By adopting this method, you gain scheduled jobs, job cancellation, and history retention essentially for free, using SQL queries you already understand.

The era of defaulting to Redis for every background job is ending. While external brokers have their place in high-throughput fire-and-forget systems, the Brokerless Worker architecture offers a compelling alternative for applications prioritizing data integrity and operational simplicity.

Whether you integrate Python workers into an existing Oban schema or adopt Python-native PostgreSQL queues like Procrastinate, the result is the same: fewer moving parts, atomic transactions, and a system that is easier to reason about. At Nohatek, we specialize in architecting resilient cloud systems that reduce complexity without sacrificing performance. If you are looking to streamline your backend infrastructure or integrate robust Python automation, our team is ready to help you build the future.