django on Saurabh Kumar

django-ox

Wed, 30 Sep 2026 12:00:00 +0000

django-ox makes the database the Django task queue: transactional enqueue inside atomic(), no Redis/RabbitMQ, retries, worker-death recovery, recurring schedules, admin and metrics. Adoption read: pilot it for ordinary Django jobs; keep Celery for broker-scale workflows.

django-crawl

Fri, 24 Jul 2026 12:00:01 +0000

django-crawl by Adam Johnson: BFS-crawls your Django site using the test client, no real HTTP overhead. Origin story: diffing CSP headers during a Django 6.0 migration accidentally caught 7 non-CSP bugs in a 100%-coverage project. HTML parsing via Rust (html5ever), follows links/sitemaps/feeds. Works as a management command or a TestCase smoke test that raises ExceptionGroup on errors.

Django ORM: From sync_to_async Threads to Native psycopg3

Sat, 13 Jun 2026 00:00:00 +0000

Django shipped async ORM methods back in 3.1. The a-prefixed methods, including aget(), afilter(), and acreate(), let calling code use coroutines, but the database work still blocked an OS thread. I want to explain that distinction and how the execution path changed.

Django 6.0 fixed it for real in December 2025. I want to walk through the path from “async via threads” to “native async via psycopg3,” because it is not a story about a feature landing. It is a story about three constraints that boxed Django in for five years: psycopg2 blocks, connection state lived in thread-locals, and the entire ORM assumed DB-API2. Once you see those three, every design decision Django made falls out of them almost mechanically.

The Thread-Based Async Path

When async views arrived in 3.1, the ORM underneath them was entirely synchronous. psycopg2 — the default PostgreSQL driver for most of Django’s life — wraps libpq, a C library with no async I/O surface at all. There is no flag you set to make psycopg2 async. You either replace the driver or you find a workaround.

Django found the workaround: asgiref.sync.sync_to_async.

# Simplified — what aget() actually did under the hood in Django 5.x
from asgiref.sync import sync_to_async

async def aget(self, *args, **kwargs):
 return await sync_to_async(self.get, thread_sensitive=True)(*args, **kwargs)

sync_to_async pushes the callable into a ThreadPoolExecutor. The event loop is not blocked, because the coroutine awaits the thread’s future. But a real OS thread is blocked, sitting on a TCP socket waiting for Postgres to answer. You moved the blocking, you did not remove it.

The thread_sensitive=True flag is the part that gives the game away. Django’s database connections lived in thread-locals:

# django/db/backends/base/base.py (pre-6.0)
import threading
_thread_local = threading.local()

Without thread_sensitive=True, two consecutive sync_to_async calls from the same request might land on different threads, see different thread-locals, and tear your transaction state in half. The flag pins every call within the same coroutine context to one dedicated thread, so the connection still looks like a single connection per request. This preserves the connection’s thread affinity.

Put it together and the cost is obvious. Five hundred concurrent async requests, each touching the database, need five hundred threads. That is the thread-per-connection model, even though the calling code uses coroutines.

Why psycopg2 Was a Dead End

You cannot retrofit async I/O onto psycopg2. It calls into libpq synchronously and blocks the calling thread until the query returns. The only way to run it in an async context is exactly what Django did: park it in a thread and await that thread. There was never a clever escape here.

The obvious alternative was asyncpg, a fully async PostgreSQL driver that beats psycopg3 on raw throughput. Django ruled it out, and the reasons are worth their own section further down. The short version is that asyncpg does not implement DB-API2 (PEP 249)¹, and Django’s ORM is built on DB-API2 from the ground up — cursor objects, execute(), fetchone(), fetchall(). You do not bolt a different contract onto twenty years of ORM and call it a driver swap.

psycopg3 is the answer that fits, because it does not force that choice.

psycopg3 Does Async And DB-API2 At The Same Time

psycopg3 — the package is just psycopg, not psycopg3 — was redesigned to support both synchronous and asynchronous execution through one shared core². The same query compilation logic runs both paths. What changes is the connection and cursor type you hand it.

# psycopg3 — synchronous path
with psycopg.connect(dsn) as conn:
 with conn.cursor() as cur:
 cur.execute("SELECT * FROM posts WHERE id = %s", [pk])
 row = cur.fetchone()

# psycopg3 — async path (same SQL, same parameters, different objects)
async with await psycopg.AsyncConnection.connect(dsn) as aconn:
 async with aconn.cursor() as cur:
 await cur.execute("SELECT * FROM posts WHERE id = %s", [pk])
 row = await cur.fetchone()

AsyncConnection and AsyncCursor use asyncio directly. The await cur.execute(...) suspends the coroutine and releases the event loop while the network I/O happens. No thread sits idle on a socket. That is the whole difference, and it is the difference that matters.

Django 6.0 made psycopg3 the recommended PostgreSQL driver and wired AsyncConnection and AsyncCursor into the backend.

What Django 6.0 Actually Changed

Three refactors enabled the switch. None of them are glamorous, and that is the point.

1. ContextVar Replaces Thread-Locals

Thread-locals are indexed by thread ID. Coroutines do not have thread IDs of their own — they share threads and switch between each other at await points. So thread-local connection storage is not merely awkward under async, it is structurally wrong. Think of a thread-local as a coat hook bolted to a single chair: it works only as long as one person stays in that chair the whole time. Coroutines stand up and swap chairs constantly.

The fix is contextvars.ContextVar, which arrived in Python 3.7. A ContextVar is scoped to the current execution context, and each asyncio Task gets its own context, copied from its parent when it is created. Django 6.0 moved connection storage off threading.local() and onto a ContextVar:

# django/db/backends/base/base.py (Django 6.0)
from contextvars import ContextVar

_connection: ContextVar = ContextVar('django_db_connection', default=None)

Two concurrent requests, both awaiting a query, each live in their own asyncio Task, each with their own context copy, each seeing their own connection. No locking, no thread affinity, no thread_sensitive gymnastics. The coat hook now travels with the person.

2. DatabaseWrapper Gets an Async Connection Path

DatabaseWrapper is the Django object that represents a database connection. Before 6.0 it exposed one path, self.connection, a synchronous psycopg2 connection. After 6.0 it also exposes async_connection():

# Conceptual — the async context manager on DatabaseWrapper
@asynccontextmanager
async def async_connection(self):
 conn = await psycopg.AsyncConnection.connect(**self.get_connection_params())
 _connection.set(conn)
 try:
 yield conn
 finally:
 await conn.close()

Django’s connection-handling code detects whether it is running inside an asyncio event loop and routes to the right path. You do not pick the path. The runtime does.

3. SQLCompiler Gets async_execute

The query compiler, SQLCompiler and its subclasses, is where Django turns a QuerySet into SQL and sends it to the database. Before 6.0 there was one path, execute_sql(). Django 6.0 adds async_execute_sql():

async def async_execute_sql(self, result_type=MULTI, chunked_fetch=False, chunk_size=GET_ITERATOR_CHUNK_SIZE):
 async with self.connection.async_connection() as conn:
 async with conn.cursor() as cursor:
 await cursor.execute(self.as_sql())
 if result_type == MULTI:
 return [row async for row in cursor]
 return await cursor.fetchone()

aget(), afilter(), aiterator() and the rest now route through async_execute_sql() instead of wrapping execute_sql() in sync_to_async. The thread pool drops out of the picture entirely.

The New Execution Stack

Here is how await Post.objects.aget(id=1) flows through Django 6.0:

await Post.objects.aget(id=1)
 └── QuerySet.aget()
 └── SQLCompiler.async_execute_sql()
 └── DatabaseWrapper.async_connection() # ContextVar-scoped
 └── psycopg.AsyncConnection
 └── AsyncCursor.execute()
 └── asyncio TCP socket write
 └── await socket read ← event loop released here
 └── row data returned

And here is the same call under Django 5.x:

await Post.objects.aget(id=1)
 └── QuerySet.aget()
 └── sync_to_async(thread_sensitive=True)(QuerySet.get)
 └── ThreadPoolExecutor — OS thread blocked
 └── SQLCompiler.execute_sql()
 └── psycopg2 connection (thread-local)
 └── libpq blocking TCP call
 └── thread wakes, result returned

The event loop is no longer handing work off to a thread. It suspends the coroutine, does other work, and resumes when the socket is readable. That is the entire reform, and it is enough.

What This Means For Concurrency

The practical payoff is in connection count. With the old thread approach, five hundred concurrent async requests meant up to five hundred threads and up to five hundred Postgres connections. Threads are cheap but not free — each costs roughly 8MB of stack by default, plus scheduler overhead.

With native async, those same five hundred requests can be multiplexed over a connection pool sized to what your database can actually handle, typically 10 to 50 connections, with the event loop running the wait queue. You use far fewer threads, one or a handful for the event loop workers, and far fewer database connections.

Django 6.0 also ships native connection pooling for the async path³, built on psycopg_pool.AsyncConnectionPool:

# settings.py — async pooling in Django 6.0
DATABASES = {
 "default": {
 "ENGINE": "django.db.backends.postgresql",
 "OPTIONS": {
 "pool": {
 "min_size": 2,
 "max_size": 10,
 },
 },
 "CONN_MAX_AGE": 0,
 }
}

This is the same pool option I covered in Django Native PostgreSQL Connection Pooling, with one difference that matters. In that post I warned against native pooling under ASGI — Django’s own docs recommended PgBouncer instead — because the sync pool and the event loop did not cooperate. Django 6.0’s AsyncConnectionPool is what closes that gap. The pool itself is now async-aware, so the ASGI caveat no longer applies on the native psycopg3 path. I was right to warn you then. You can stop worrying about it now.

Is sync_to_async Gone?

No, and it should not be. It is the fallback for drivers that cannot go native async:

Driver	Async path
psycopg3 (PostgreSQL)	Native `AsyncCursor` — no threads
psycopg2 (PostgreSQL)	`sync_to_async` thread wrap — still blocked
mysqlclient / MySQL	`sync_to_async` thread wrap
sqlite3	`sync_to_async` thread wrap

Run Django 6.0 against psycopg2 or MySQL in an async context and Django still falls back to the thread pool. The native path only exists when psycopg3 is installed and configured. The thread trick did not become wrong. It became the path of last resort.

The way forward for other databases is psycopg3-equivalent async drivers. MySQL has aiomysql, SQLite has aiosqlite. Django contributions for those backends are in progress but not stable in 6.0, so for now PostgreSQL is the database that gets the real thing.

asyncpg Is Still Faster, And Still Not Supported

Benchmark asyncpg against psycopg3 on raw async throughput and asyncpg wins, often by 20 to 40 percent on bulk read workloads. Django will not adopt it. The reasons are structural, not political, and I think Django is right.

No DB-API2. asyncpg uses positional $1, $2 parameters, not %s. The entire ORM query compiler assumes %s.
No synchronous fallback. Management commands, migrations, and sync views would need a whole separate code path.
A different transaction model. asyncpg handles transactions differently from the DB-API2 autocommit and explicit model Django manages internally.

Using asyncpg would require changes to the ORM, not just a driver swap. A driver benchmark alone does not measure that work. If you genuinely need asyncpg throughput with a Django-like ORM, look at SQLAlchemy 2.0 with its async session support and asyncpg backend, running under Starlette or FastAPI. For Django, psycopg3 is the correct answer.

Upgrading

To get the native async path today:

pip install "psycopg[binary,pool]"

# settings.py
DATABASES = {
 "default": {
 "ENGINE": "django.db.backends.postgresql",
 "NAME": "mydb",
 "USER": "myuser",
 "PASSWORD": "mypassword",
 "HOST": "localhost",
 "CONN_MAX_AGE": 0,
 "OPTIONS": {
 "pool": True, # or dict with min_size/max_size
 },
 }
}

If you are on Django 5.2 LTS with psycopg3, you already get the synchronous psycopg3 path and connection pooling. The native async query path needs Django 6.0. You can upgrade the driver without upgrading Django and still get the pooling benefits in sync contexts, which is a fine thing to do while you wait.

Why It Took Five Years

Django’s async ORM story ran from “async with threads” in 3.1 in 2020 to “native async with psycopg3” in 6.0 in 2025. Five years sounds slow until you look at what had to be true first:

psycopg3 had to be stable enough to depend on, which it reached around 2022 to 2023.
Thread-local connection state was woven through the codebase, and the ContextVar migration touched hundreds of files.
The Django project is conservative about breaking changes, especially in database handling, where bugs are not cosmetic.

The result is correct and maintainable rather than the absolute fastest thing money can buy, and I do not think that was the wrong trade. For most Django applications — web services handling thousands of concurrent HTTP requests — native async with psycopg3 is already more than fast enough, and it integrates cleanly with everything else in the ecosystem.

Django Native PostgreSQL Connection Pooling — the sync side of the same story: configuring the psycopg3 pool, sizing min_size/max_size, and avoiding the CONN_MAX_AGE foot-gun.

PEP 249 — Python Database API Specification v2.0 — The DB-API2 spec that Django’s ORM is built on top of; asyncpg deliberately does not implement it. ↩︎
psycopg3 — Async operations — Official psycopg documentation on the async connection and cursor API. ↩︎
Django 6.0 Release Notes — Full changelog including async ORM changes, background tasks, and template partials. ↩︎

Django Modern REST

Thu, 02 Apr 2026 12:00:00 +0000

Django Modern REST seems to be an emerging REST API library with focus on performance, scalability and pluggable serializers. Supporting msgpack, CBV and less of sync_to_async. Might evaluate it against the my current preferred django-ninja. gemini

zoocache

Fri, 13 Feb 2026 12:00:00 +0000

zoocache is a sematic dependency based cache manager, that support in-memory, LMDB or redis backends. Integration with Django looks interesting.

Django Postgres Migration Tools

Wed, 07 Jan 2026 12:00:00 +0000

“Django Postgres Migration Tools - add-on for safer and more scalable migrations in django.

Django rapid architecture

Fri, 19 Dec 2025 12:00:01 +0000

Notes from “Django rapid architecture”

Recommends organizing the codebase with interfaces at the top layer followed by readers and “actions”, Finally, ‘data’ as the bottom layer to organize Django codebase to allow for a simple code structure that allows for scaling

Django Native PostgreSQL Connection Pooling

Wed, 18 Jun 2025 00:00:00 +0000

Opening PostgreSQL connections adds overhead. Django 5.1 (released August 7, 2024) added native connection pooling¹, which lets an application reuse connections. Measure connection overhead and response times before and after enabling it; the benefit depends on your workload.

If you’re on Django 6+ and running async/ASGI, also see the async pool notes in How Django’s ORM Went From sync_to_async Threads to Native psycopg3.

Native Pooling vs External Pooling

PgBouncer runs as a separate pooling server. Django’s native pooling runs in the application process, so it does not require a separate server. It does require psycopg’s pool package. Third-party Django pooling packages and manual connection management are other options, with their own maintenance and configuration needs.

Configure Native Pooling

Requirements: Django 5.1+ and PostgreSQL (this feature does not work with psycopg2)²

Install the correct package:

pip install "psycopg[binary,pool]"

Critical: The package is called psycopg (not psycopg3). The pool functionality requires a separate package that this command installs automatically³.

Add these lines to your settings.py:

DATABASES = {
 "default": {
 "ENGINE": "django.db.backends.postgresql",
 "NAME": "your_db_name",
 "USER": "your_db_user",
 "PASSWORD": "your_db_password",
 "HOST": "your_db_host",
 "PORT": "5432",
 "CONN_MAX_AGE": 0, # Required - pooling fails without this
 "OPTIONS": {
 "pool": True,
 },
 }
}

Deploy this change. Your app now reuses connections instead of creating new ones for every request.

Prevent the Configuration Error That Breaks Production

Must set CONN_MAX_AGE = 0. Without this, Django throws ImproperlyConfigured: Pooling doesn't support persistent connections and your app crashes⁴. The pool manages connection lifetimes, making Django’s persistence redundant.

Scale for High-Traffic Production

Replace "pool": True with specific parameters for production loads:

DATABASES = {
 "default": {
 "ENGINE": "django.db.backends.postgresql",
 "NAME": "your_db_name",
 "USER": "your_db_user",
 "PASSWORD": "your_db_password",
 "HOST": "your_db_host",
 "PORT": "5432",
 "CONN_MAX_AGE": 0,
 "OPTIONS": {
 "pool": {
 "min_size": 4, # Keeps connections warm
 "max_size": 16, # Handles traffic spikes
 "timeout": 10, # Fails fast under extreme load
 "max_lifetime": 1800, # 30 minutes maximum connection age
 "max_idle": 300, # Close idle connections after 5 minutes
 },
 },
 }
}

Start with max_size equal to expected concurrent users divided by 10. Scale up if you see timeout errors in logs.

Monitor Your Performance Gains

Enable pool monitoring to track your improvements:

# In settings.py
import logging
logging.getLogger('psycopg.pool').setLevel(logging.INFO)

Watch pool utilization, connection wait times, timeout errors, and response times. Record median and 95th percentile latency before and after the change under comparable load. Use those measurements to size the pool and judge whether pooling helps your application.

Critical Fix for Threading and Celery Applications

Essential for background processing: This isn’t a pooling bug — it’s expected Django behavior that pooling simply makes painful. Any connection opened outside Django’s request-response cycle (for example, in a thread you spawn yourself) stays open until you explicitly close it. Without a pool you might not notice; with one, those un-returned connections exhaust max_size and you start seeing PoolTimeout errors⁵. Add explicit cleanup:

from django.db import connection
from threading import Thread

class DataProcessingThread(Thread):
 def run(self):
 # Your database operations here
 User.objects.filter(active=True).update(last_seen=timezone.now())

 # Critical: returns the connection to the pool
 connection.close()

For code that may touch multiple database aliases, Django’s documented helper django.db.close_old_connections() is the more robust choice — it closes all old or unusable connections across every configured database:

from django.db import close_old_connections

class DataProcessingThread(Thread):
 def run(self):
 try:
 User.objects.filter(active=True).update(last_seen=timezone.now())
 finally:
 close_old_connections()

Without this cleanup, each background thread permanently holds a connection from your pool.

Deployment Scenarios That Require Different Approaches

ASGI applications: On Django 5.1–5.2, Django’s documentation recommends against native pooling with ASGI — use PgBouncer or similar external poolers instead. This changed in Django 6.0, which ships an async-aware AsyncConnectionPool that works correctly under ASGI. I cover how that fits together with the truly-async ORM in How Django’s ORM Went From sync_to_async Threads to Native psycopg3.

Serverless deployments: Connection pools don’t persist across serverless invocations. Skip this optimization for Lambda/Cloud Functions.

Multi-tenant applications: Each tenant database needs its own pool configuration in DATABASES.

When to Evaluate Pooling

Evaluate pooling when connection establishment is a measurable part of request latency, concurrency is growing, or the application is approaching PostgreSQL connection limits. Compare measurements before and after rather than assuming a fixed latency or cost saving.

Django 5.1 Release Notes - Official Django documentation announcing native connection pooling support ↩︎
Django Database Configuration - Complete guide to Django database settings including pooling requirements ↩︎
psycopg Installation Guide - Official psycopg documentation for package installation and pool dependencies ↩︎
Stack Overflow: Django Pooling Configuration Error - Community discussion of the CONN_MAX_AGE requirement ↩︎
Django Ticket #35672 - Closed as invalid: connections opened in threads must be returned manually. See also the Databases → Caveats docs, which recommend django.db.close_old_connections() for connections created outside the request-response cycle. ↩︎

Handling Django Authentication Redirects in HTMX Applications

Fri, 02 May 2025 00:00:00 +0000

When a user’s session expires, an HTMX request can follow Django’s login redirect and swap the login page into the target element. I ran into this recently and used middleware to redirect the whole browser instead.

What’s Going Wrong?

HTMX updates part of a page, but an expired session can cause Django to redirect a request to the login page. The returned login HTML can then appear inside a div or table cell rather than replacing the page.

What we really want is to redirect the whole browser to the login page, and then bring users right back to where they were after they log in again. Simple idea, slightly tricky execution.

Redirect with middleware

This middleware converts HTMX 302 responses into a full-page redirect using HX-Redirect. It sets a next parameter from the referring page or request path.

from urllib.parse import urlparse


class HtmxAuthRedirectMiddleware:
 """
 Middleware to handle HTMX authentication redirects properly.

 When an HTMX request results in a 302 redirect (typically for authentication),
 this middleware:
 1. Changes the response status code to 204 (No Content)
 2. Adds an HX-Redirect header with the redirect URL
 3. Preserves the original request path in the 'next' query parameter

 This ensures that after authentication, the user is returned to the page
 they were attempting to access, maintaining a seamless UX with HTMX.

 Credits: https://www.caktusgroup.com/blog/2022/11/11/how-handle-django-login-redirects-htmx/
 """

 def __init__(self, get_response):
 self.get_response = get_response

 def __call__(self, request):
 response = self.get_response(request)
 # HTMX request returning 302 likely is login required.
 # Take the redirect location and send it as the HX-Redirect header value,
 # with 'next' query param set to where the request originated. Also change
 # response status code to 204 (no content) so that htmx will obey the
 # HX-Redirect header value.
 if request.headers.get("HX-Request") == "true" and response.status_code == 302:
 # Determine the next path from referer or current request path
 ref_header = request.headers.get("Referer", "")
 if ref_header:
 referer = urlparse(ref_header)
 next_path = referer.path
 else:
 next_path = request.path

 # Parse the redirect URL
 redirect_url = urlparse(response["location"])

 # Set response status code to 204 for HTMX to process the redirect
 response.status_code = 204

 # Update the "?next" query parameter
 query_params = parse_qs(redirect_url.query)
 query_params["next"] = [next_path]
 new_query = urlencode(query_params, doseq=True)

 # Set the new HX-Redirect header
 response.headers["HX-Redirect"] = f"{redirect_url.path}?{new_query}"

 return response

This middleware is inspired by this post by the caktus group.

How It Works

The middleware changes a 302 response to 204 so HTMX can process the HX-Redirect header. That header redirects the whole browser. The next parameter records the path to return to after login.

This applies to all HTMX 302 responses, not just authentication redirects. Add checks if other redirects in your application need different handling.

Adding This to Your Project

Add the middleware to your Django project and its MIDDLEWARE list in settings.py:

MIDDLEWARE = [
 # Your other middleware stuff...
 'yourapp.middleware.HtmxAuthRedirectMiddleware',
]

Just make sure it comes after Django’s authentication middleware in the list - order matters here!

Edits:

2025-05-04: Use urlparse and parse_qs to securely handle urls. Thanks Adam Johnson