databases on Saurabh Kumar

django-ox

Wed, 30 Sep 2026 12:00:00 +0000

django-ox makes the database the Django task queue: transactional enqueue inside atomic(), no Redis/RabbitMQ, retries, worker-death recovery, recurring schedules, admin and metrics. Adoption read: pilot it for ordinary Django jobs; keep Celery for broker-scale workflows.

jetify-com/typeid

Sat, 26 Sep 2026 12:00:00 +0000

jetify-com/typeid is a spec for Stripe-style IDs: user_01h2xcejqtf2nbrexx3vqjhp41 is a snake_case type prefix, _, then a UUIDv7 in 26 chars of Crockford base32 (hex UUIDs take 36). Time-ordered, so B-tree inserts stay local, unlike UUIDv4. Double-click selects the whole ID, and a serializer expecting user_ rejects a pasted post_ ID. Strip the prefix and you get a valid UUIDv7 back, so the database never has to know. Cheap way to adopt: keep the column a native 16-byte uuid (DEFAULT uuidv7() on Postgres 18, uuid.uuid7() on Python 3.14) and encode/decode only at the API edge, e.g. a DRF serializer field on top of typeid-python. If PKs are already uuid, no schema change, and IDs explain themselves in logs and support tickets.

Lobste.rs moved to SQLite

Sat, 18 Jul 2026 12:00:00 +0000

Lobste.rs moved to SQLite, and the HN thread had sharper lessons than the post itself:

WAL checkpoints only run during “reader gaps.” Keep reads/writes continuously overlapping and the WAL file grows unbounded, a DoS vector kgeist reproduced with short-lived reads/writes, not just long-lived readers as the docs imply.
Writes are fully serialized, one fsync each, yet app-layer batching still gets 100k+ writes/sec.
Plain rsync on a live .db file can silently corrupt it since it doesn’t understand SQLite transactions. Use sqlite3_rsync instead.
The real weak point is schema migrations: treat SQLite like a KV store, or reach for CQRS, to keep the churn low.

PostgreSQL UNLOGGED tables as a cache

Sun, 28 Jun 2026 12:00:00 +0000

PostgreSQL UNLOGGED tables skip the write-ahead log, making writes considerably faster than ordinary tables. Trade-off: they are truncated after a crash or unclean shutdown and are not replicated to standbys. hynek/psycache exploits this as a PostgreSQL-native cache replacing Redis — crash truncation is acceptable since it’s a cache anyway, and you reuse existing Postgres connections. Interesting idea: port this to Django as a custom BaseCache backend — swap DatabaseCache’s regular table for CREATE UNLOGGED TABLE, reuse Django’s existing DB connections, and skip Redis entirely. ~200 lines.

Django ORM: From sync_to_async Threads to Native psycopg3

Sat, 13 Jun 2026 00:00:00 +0000

Django shipped async ORM methods back in 3.1. The a-prefixed methods, including aget(), afilter(), and acreate(), let calling code use coroutines, but the database work still blocked an OS thread. I want to explain that distinction and how the execution path changed.

Django 6.0 fixed it for real in December 2025. I want to walk through the path from “async via threads” to “native async via psycopg3,” because it is not a story about a feature landing. It is a story about three constraints that boxed Django in for five years: psycopg2 blocks, connection state lived in thread-locals, and the entire ORM assumed DB-API2. Once you see those three, every design decision Django made falls out of them almost mechanically.

The Thread-Based Async Path

When async views arrived in 3.1, the ORM underneath them was entirely synchronous. psycopg2 — the default PostgreSQL driver for most of Django’s life — wraps libpq, a C library with no async I/O surface at all. There is no flag you set to make psycopg2 async. You either replace the driver or you find a workaround.

Django found the workaround: asgiref.sync.sync_to_async.

# Simplified — what aget() actually did under the hood in Django 5.x
from asgiref.sync import sync_to_async

async def aget(self, *args, **kwargs):
 return await sync_to_async(self.get, thread_sensitive=True)(*args, **kwargs)

sync_to_async pushes the callable into a ThreadPoolExecutor. The event loop is not blocked, because the coroutine awaits the thread’s future. But a real OS thread is blocked, sitting on a TCP socket waiting for Postgres to answer. You moved the blocking, you did not remove it.

The thread_sensitive=True flag is the part that gives the game away. Django’s database connections lived in thread-locals:

# django/db/backends/base/base.py (pre-6.0)
import threading
_thread_local = threading.local()

Without thread_sensitive=True, two consecutive sync_to_async calls from the same request might land on different threads, see different thread-locals, and tear your transaction state in half. The flag pins every call within the same coroutine context to one dedicated thread, so the connection still looks like a single connection per request. This preserves the connection’s thread affinity.

Put it together and the cost is obvious. Five hundred concurrent async requests, each touching the database, need five hundred threads. That is the thread-per-connection model, even though the calling code uses coroutines.

Why psycopg2 Was a Dead End

You cannot retrofit async I/O onto psycopg2. It calls into libpq synchronously and blocks the calling thread until the query returns. The only way to run it in an async context is exactly what Django did: park it in a thread and await that thread. There was never a clever escape here.

The obvious alternative was asyncpg, a fully async PostgreSQL driver that beats psycopg3 on raw throughput. Django ruled it out, and the reasons are worth their own section further down. The short version is that asyncpg does not implement DB-API2 (PEP 249)¹, and Django’s ORM is built on DB-API2 from the ground up — cursor objects, execute(), fetchone(), fetchall(). You do not bolt a different contract onto twenty years of ORM and call it a driver swap.

psycopg3 is the answer that fits, because it does not force that choice.

psycopg3 Does Async And DB-API2 At The Same Time

psycopg3 — the package is just psycopg, not psycopg3 — was redesigned to support both synchronous and asynchronous execution through one shared core². The same query compilation logic runs both paths. What changes is the connection and cursor type you hand it.

# psycopg3 — synchronous path
with psycopg.connect(dsn) as conn:
 with conn.cursor() as cur:
 cur.execute("SELECT * FROM posts WHERE id = %s", [pk])
 row = cur.fetchone()

# psycopg3 — async path (same SQL, same parameters, different objects)
async with await psycopg.AsyncConnection.connect(dsn) as aconn:
 async with aconn.cursor() as cur:
 await cur.execute("SELECT * FROM posts WHERE id = %s", [pk])
 row = await cur.fetchone()

AsyncConnection and AsyncCursor use asyncio directly. The await cur.execute(...) suspends the coroutine and releases the event loop while the network I/O happens. No thread sits idle on a socket. That is the whole difference, and it is the difference that matters.

Django 6.0 made psycopg3 the recommended PostgreSQL driver and wired AsyncConnection and AsyncCursor into the backend.

What Django 6.0 Actually Changed

Three refactors enabled the switch. None of them are glamorous, and that is the point.

1. ContextVar Replaces Thread-Locals

Thread-locals are indexed by thread ID. Coroutines do not have thread IDs of their own — they share threads and switch between each other at await points. So thread-local connection storage is not merely awkward under async, it is structurally wrong. Think of a thread-local as a coat hook bolted to a single chair: it works only as long as one person stays in that chair the whole time. Coroutines stand up and swap chairs constantly.

The fix is contextvars.ContextVar, which arrived in Python 3.7. A ContextVar is scoped to the current execution context, and each asyncio Task gets its own context, copied from its parent when it is created. Django 6.0 moved connection storage off threading.local() and onto a ContextVar:

# django/db/backends/base/base.py (Django 6.0)
from contextvars import ContextVar

_connection: ContextVar = ContextVar('django_db_connection', default=None)

Two concurrent requests, both awaiting a query, each live in their own asyncio Task, each with their own context copy, each seeing their own connection. No locking, no thread affinity, no thread_sensitive gymnastics. The coat hook now travels with the person.

2. DatabaseWrapper Gets an Async Connection Path

DatabaseWrapper is the Django object that represents a database connection. Before 6.0 it exposed one path, self.connection, a synchronous psycopg2 connection. After 6.0 it also exposes async_connection():

# Conceptual — the async context manager on DatabaseWrapper
@asynccontextmanager
async def async_connection(self):
 conn = await psycopg.AsyncConnection.connect(**self.get_connection_params())
 _connection.set(conn)
 try:
 yield conn
 finally:
 await conn.close()

Django’s connection-handling code detects whether it is running inside an asyncio event loop and routes to the right path. You do not pick the path. The runtime does.

3. SQLCompiler Gets async_execute

The query compiler, SQLCompiler and its subclasses, is where Django turns a QuerySet into SQL and sends it to the database. Before 6.0 there was one path, execute_sql(). Django 6.0 adds async_execute_sql():

async def async_execute_sql(self, result_type=MULTI, chunked_fetch=False, chunk_size=GET_ITERATOR_CHUNK_SIZE):
 async with self.connection.async_connection() as conn:
 async with conn.cursor() as cursor:
 await cursor.execute(self.as_sql())
 if result_type == MULTI:
 return [row async for row in cursor]
 return await cursor.fetchone()

aget(), afilter(), aiterator() and the rest now route through async_execute_sql() instead of wrapping execute_sql() in sync_to_async. The thread pool drops out of the picture entirely.

The New Execution Stack

Here is how await Post.objects.aget(id=1) flows through Django 6.0:

await Post.objects.aget(id=1)
 └── QuerySet.aget()
 └── SQLCompiler.async_execute_sql()
 └── DatabaseWrapper.async_connection() # ContextVar-scoped
 └── psycopg.AsyncConnection
 └── AsyncCursor.execute()
 └── asyncio TCP socket write
 └── await socket read ← event loop released here
 └── row data returned

And here is the same call under Django 5.x:

await Post.objects.aget(id=1)
 └── QuerySet.aget()
 └── sync_to_async(thread_sensitive=True)(QuerySet.get)
 └── ThreadPoolExecutor — OS thread blocked
 └── SQLCompiler.execute_sql()
 └── psycopg2 connection (thread-local)
 └── libpq blocking TCP call
 └── thread wakes, result returned

The event loop is no longer handing work off to a thread. It suspends the coroutine, does other work, and resumes when the socket is readable. That is the entire reform, and it is enough.

What This Means For Concurrency

The practical payoff is in connection count. With the old thread approach, five hundred concurrent async requests meant up to five hundred threads and up to five hundred Postgres connections. Threads are cheap but not free — each costs roughly 8MB of stack by default, plus scheduler overhead.

With native async, those same five hundred requests can be multiplexed over a connection pool sized to what your database can actually handle, typically 10 to 50 connections, with the event loop running the wait queue. You use far fewer threads, one or a handful for the event loop workers, and far fewer database connections.

Django 6.0 also ships native connection pooling for the async path³, built on psycopg_pool.AsyncConnectionPool:

# settings.py — async pooling in Django 6.0
DATABASES = {
 "default": {
 "ENGINE": "django.db.backends.postgresql",
 "OPTIONS": {
 "pool": {
 "min_size": 2,
 "max_size": 10,
 },
 },
 "CONN_MAX_AGE": 0,
 }
}

This is the same pool option I covered in Django Native PostgreSQL Connection Pooling, with one difference that matters. In that post I warned against native pooling under ASGI — Django’s own docs recommended PgBouncer instead — because the sync pool and the event loop did not cooperate. Django 6.0’s AsyncConnectionPool is what closes that gap. The pool itself is now async-aware, so the ASGI caveat no longer applies on the native psycopg3 path. I was right to warn you then. You can stop worrying about it now.

Is sync_to_async Gone?

No, and it should not be. It is the fallback for drivers that cannot go native async:

Driver	Async path
psycopg3 (PostgreSQL)	Native `AsyncCursor` — no threads
psycopg2 (PostgreSQL)	`sync_to_async` thread wrap — still blocked
mysqlclient / MySQL	`sync_to_async` thread wrap
sqlite3	`sync_to_async` thread wrap

Run Django 6.0 against psycopg2 or MySQL in an async context and Django still falls back to the thread pool. The native path only exists when psycopg3 is installed and configured. The thread trick did not become wrong. It became the path of last resort.

The way forward for other databases is psycopg3-equivalent async drivers. MySQL has aiomysql, SQLite has aiosqlite. Django contributions for those backends are in progress but not stable in 6.0, so for now PostgreSQL is the database that gets the real thing.

asyncpg Is Still Faster, And Still Not Supported

Benchmark asyncpg against psycopg3 on raw async throughput and asyncpg wins, often by 20 to 40 percent on bulk read workloads. Django will not adopt it. The reasons are structural, not political, and I think Django is right.

No DB-API2. asyncpg uses positional $1, $2 parameters, not %s. The entire ORM query compiler assumes %s.
No synchronous fallback. Management commands, migrations, and sync views would need a whole separate code path.
A different transaction model. asyncpg handles transactions differently from the DB-API2 autocommit and explicit model Django manages internally.

Using asyncpg would require changes to the ORM, not just a driver swap. A driver benchmark alone does not measure that work. If you genuinely need asyncpg throughput with a Django-like ORM, look at SQLAlchemy 2.0 with its async session support and asyncpg backend, running under Starlette or FastAPI. For Django, psycopg3 is the correct answer.

Upgrading

To get the native async path today:

pip install "psycopg[binary,pool]"

# settings.py
DATABASES = {
 "default": {
 "ENGINE": "django.db.backends.postgresql",
 "NAME": "mydb",
 "USER": "myuser",
 "PASSWORD": "mypassword",
 "HOST": "localhost",
 "CONN_MAX_AGE": 0,
 "OPTIONS": {
 "pool": True, # or dict with min_size/max_size
 },
 }
}

If you are on Django 5.2 LTS with psycopg3, you already get the synchronous psycopg3 path and connection pooling. The native async query path needs Django 6.0. You can upgrade the driver without upgrading Django and still get the pooling benefits in sync contexts, which is a fine thing to do while you wait.

Why It Took Five Years

Django’s async ORM story ran from “async with threads” in 3.1 in 2020 to “native async with psycopg3” in 6.0 in 2025. Five years sounds slow until you look at what had to be true first:

psycopg3 had to be stable enough to depend on, which it reached around 2022 to 2023.
Thread-local connection state was woven through the codebase, and the ContextVar migration touched hundreds of files.
The Django project is conservative about breaking changes, especially in database handling, where bugs are not cosmetic.

The result is correct and maintainable rather than the absolute fastest thing money can buy, and I do not think that was the wrong trade. For most Django applications — web services handling thousands of concurrent HTTP requests — native async with psycopg3 is already more than fast enough, and it integrates cleanly with everything else in the ecosystem.

Django Native PostgreSQL Connection Pooling — the sync side of the same story: configuring the psycopg3 pool, sizing min_size/max_size, and avoiding the CONN_MAX_AGE foot-gun.

PEP 249 — Python Database API Specification v2.0 — The DB-API2 spec that Django’s ORM is built on top of; asyncpg deliberately does not implement it. ↩︎
psycopg3 — Async operations — Official psycopg documentation on the async connection and cursor API. ↩︎
Django 6.0 Release Notes — Full changelog including async ORM changes, background tasks, and template partials. ↩︎

pageindex

Tue, 24 Mar 2026 12:00:00 +0000

pageindex generates a semantic tree-like json index of a lengthy document to allow for reasoning based RAG without the need for vectordb.

slingdata-io/sling-cli

Thu, 19 Feb 2026 12:00:00 +0000

slingdata-io/sling-cli is a promising tool move/sync data between databases and files, esp. helpful for local testing, ci/cd while able to do stage/sql based transformations.

alibaba/zvec

Sun, 15 Feb 2026 12:00:02 +0000

alibaba/zvec by alibaba is an embedded vector database supports both spare and dense vectors, along with structured data. It can be considered the sqlite of vector databases.

zoocache

Fri, 13 Feb 2026 12:00:00 +0000

zoocache is a sematic dependency based cache manager, that support in-memory, LMDB or redis backends. Integration with Django looks interesting.

Vortex

Thu, 05 Feb 2026 12:00:01 +0000

Vortex a newer format is supported by duckdb promises to be faster for random access and has zero-copy metadata, with better compression. While the support across the board is limited but worth considering for LLM/RAG based uses over parquet.

Designing Data-Intensive Applications

Thu, 05 Feb 2026 12:00:00 +0000

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann manages to be both rigorous technical manual and something approaching a philosophical treatise on the nature of truth, consistency, and trust in distributed systems. [Claude][https://claude.ai/chat/0f58f2f6-bd56-41a1-a785-8267afa5a3d1]

Foundation. He starts by asking the questions “What do we actually want in our data systems”, answers them as - reliability, scalability and maintainability.
- Reliability. The question isn’t whether failure will happen, it will, but the whether the systems can survive them.
- Scalability. It’s not a binary question of whether a system is “scalable” or “not scalable”. Ask - “What happens when specific load parameter increase?”
- Maintainability - most underappreciated of the three, he argues the majority of cost isn’t in initial development but in ongoing maintainence.
The data model wars - the skill isn’t in choosing the “best” database but in understanding which tradeoffs matter for your specific problem.
Storage engines. Two major approaches to read & write data from disk, neither is universally better.
- Log-structured storage (like LSM-trees) optimizes for writes, every write is appended and the merged/compacted later.
- Page-oriented storage (like B-trees) optimizes for reads, data is stored in fixed-sized block, which then get updated in-place, more like filing cabinet with each document has a designated slot.
Instead of asking “how do I build this?” ask “what does it mean for this to work correctly?”

etcd, a distributed key-value store

Mon, 12 Jan 2026 12:00:02 +0000

Distributed Key-value store – https://etcd.io/

Redis’s License Change and the Valkey Fork

Mon, 12 Jan 2026 00:00:00 +0000

The License Change

On March 20, 2024, Redis Inc. announced a switch from the BSD 3-clause license to dual source-available licensing: RSALv2 and SSPLv1. The new licensing takes effect with Redis 7.4 and restricts how the code can be offered as a service.

I see this as a loss for the open-source community, even if Redis Inc. considers it a business necessity.

The Thing About Open Source

Redis was created by Salvatore Sanfilippo in 2009, known as “antirez” in the community. He built Redis to solve a simple problem: web applications were slow because databases couldn’t keep up. What if, instead of storing data on disk, you kept everything in memory? What if you made it blazingly fast?

Redis became legendary. By 2024, it was everywhere—caching layers for social media platforms, session stores for e-commerce sites, real-time analytics for financial systems. Developers loved it. Not just because it was fast, but because it was theirs. They could use it, modify it, deploy it however they wanted.

Then Redis Inc. changed the rules.

The Rebellion

Kyle Davis, a longtime Redis contributor, said something interesting after the split: “From this point forward, Redis and Valkey are two different pieces of software.” Not just different licenses—different software.

Just eight days later, on March 28, 2024, something remarkable happened. The Linux Foundation announced Valkey, a fork starting from Redis 7.2.4, the last truly open version. But this wasn’t some ragtag group of angry developers. This was AWS, Google Cloud, Oracle, Ericsson, and Snap Inc.—companies with deep pockets and deeper technical expertise.

They weren’t just copying Redis and giving it a new name. They were going to improve it.

The Performance Revolution

Here’s where the story gets interesting, because it stops being about philosophy and starts being about engineering.

Redis, for all its speed, had a design constraint: it was single-threaded. Salvatore made this choice deliberately. Single-threading avoids all the complexity of concurrent programming—no locks, no race conditions, no debugging nightmares. For years, this was fine. Processors were getting faster, and Redis was fast enough.

But by 2024, we’re not making processors faster anymore—we’re making them wider. Your laptop has eight cores, maybe sixteen. Your cloud server has dozens. And Redis, with its elegant single-threaded design, was using… one.

The Valkey team saw their opportunity.

Valkey 8.0, released on September 15, 2024, introduced enhanced multi-threaded I/O. Not for the core data operations—those stayed single-threaded to preserve Redis’s simplicity and safety—but for network operations, for handling connections, for moving data in and out.

The results were dramatic. In benchmarks on AWS hardware, Valkey 8.0 hit 1.19 million requests per second. That’s 230% more than Valkey 7.2. That’s the kind of improvement that makes CTOs take notice.

But the engineers didn’t stop there.

The Memory Game

Valkey 8.1, released on March 31, 2025, arrived with something even more impressive: a complete redesign of how data is stored in memory.

The team rebuilt the hash table—the fundamental data structure at the heart of any key-value store—using modern algorithms inspired by Google’s “Swiss Tables” design. Every key-value pair now uses about 20-30 bytes less memory. That might not sound like much, but scale it up.

When researchers tested Valkey 8.1 against Redis 8.2 with 50 million entries, Valkey consumed 3.77 GB while Redis used 4.83 GB. That’s a 28% reduction. For a company running terabytes of cached data across hundreds of servers, that’s not just impressive—it’s millions of dollars in annual savings.

The Feature Race

Of course, efficiency isn’t everything. Redis Inc. didn’t just sit back and watch. In May 2025, Redis released version 8.0, bundling their enterprise features—JSON support, vector search for AI applications, time series databases—directly into the core product. These are sophisticated capabilities that took years to develop.

Valkey’s roadmap is playing catch-up here. Version 8.1 added JSON and Bloom filters. Vector search support became available through the valkey-search module. Time series support is planned. But there’s still a gap.

This is the classic innovator’s dilemma: Redis has more features today. Valkey has better fundamentals and a trajectory that suggests they’ll have those features tomorrow—and they’ll run faster when they do.

The Cloud Complication

Here’s where it gets messy. When you go to AWS and spin up what was formerly called “Redis,” you might now get Valkey. Same with Google Cloud’s Memorystore.

These managed services have their own quirks and limitations, shaped by what the cloud providers choose to support. The result? More fragmentation.

Managed Redis-compatible services now include both projects, with different feature sets and provider support.

The Choice

So where does this leave developers in 2026?

If you’re starting a new project, Valkey offers true open-source freedom, better performance on multi-core systems, and superior memory efficiency. You sacrifice some advanced features, but its BSD license permits use, modification, and redistribution under its terms.

If you need those advanced features right now—vector search for AI applications, sophisticated time series analysis, enterprise support—Redis has them. But you’re accepting the terms Redis Inc. sets, and those terms have already changed once.

For existing Redis deployments, the calculation is harder. Migration costs are real. But so is the risk of depending on a vendor that rewrote the rules mid-game.

My Take

Valkey started with an established codebase and backing from major companies, including AWS, Google Cloud, Oracle, Ericsson, and Snap Inc. Its performance and memory work give developers reasons to evaluate it beyond the license.

I prefer Valkey’s permissive licensing and foundation governance. To me, the fork shows why the ability to keep developing an existing open-source codebase matters when a vendor changes terms. Both projects continue to evolve, and the choice still depends on features, migration costs, and workload measurements.

github recommendor

Sun, 11 Jan 2026 12:00:00 +0000

Puzer published a github recommendor that uses semantic embedding from user’s github stars all client side, I found some great recommendations which I plan to use:

pamburus/hl: A fast and powerful log viewer and processor that converts JSON logs or logfmt logs into a clear human-readable format. (⭐2657)
samwho/spacer: CLI tool to insert spacers when command output stops (⭐1663)
darrenburns/posting: The modern API client that lives in your terminal. (⭐11134)
plutov/oq: Terminal OpenAPI Spec viewer (⭐943)
wey-gu/py-pglite: PGlite wrapper in Python for testing. Test your app with Postgres just as lite as SQLite. (⭐577)

Django Postgres Migration Tools

Wed, 07 Jan 2026 12:00:00 +0000

“Django Postgres Migration Tools - add-on for safer and more scalable migrations in django.

Thoughtworks - Technology Radar vol 33

Mon, 22 Dec 2025 12:00:02 +0000

Notes from Thoughtworks - Technology Radar vol 33

text-to-sql solutions aren’t working as expected
pnpm, langGraph, and pydantic recommended for adoption

VectorChord

Fri, 19 Dec 2025 12:00:00 +0000

VectorChord is a faster pgvector alternative.

BM25 Ranking in PostgreSQL with pg_textsearch

Fri, 19 Dec 2025 00:00:00 +0000

Today I learned about pg_textsearch, a new PostgreSQL extension by Timescale that brings BM25 relevance-ranked full-text search to Postgres.

What is BM25?

BM25 (Best Matching 25) is the industry-standard ranking algorithm used by search engines like Elasticsearch and Lucene. Unlike simple term frequency counting, BM25 provides smarter relevance scoring through:

Term Frequency Saturation: Mentioning a word 10 times isn’t 10x better than once—diminishing returns kick in
Inverse Document Frequency (IDF): Rare terms get higher weight than common ones
Document Length Normalization: Short and long documents are compared fairly

The formula uses two tunable parameters:

k1 (default 1.2): Controls term frequency saturation
b (default 0.75): Controls length normalization (0 = none, 1 = full)

Basic Usage

-- Create a BM25 index
CREATE INDEX docs_idx ON documents USING bm25(content)
 WITH (text_config='english');

-- Search with the <@> operator (returns negative scores—lower is better)
SELECT title, content <@> 'database search' as score
FROM documents
ORDER BY content <@> 'database search'
LIMIT 10;

pg_textsearch vs PostgreSQL Built-in FTS

Aspect	PostgreSQL FTS	pg_textsearch (BM25)
Ranking	Basic term frequency	Probabilistic with IDF
Operator	`@@` with `tsquery`	`<@>`
Boolean queries	Rich (`&`, `\|`, `!`, `<->`)	Simple (implicit AND)
Phrase search	Yes	No
Highlighting	`ts_headline()`	No
Relevance quality	Basic	Industry-standard
Maturity	Battle-tested	Prerelease (v0.1.x)

When to Use Each

Use PostgreSQL FTS when you need:

Complex boolean queries ((postgresql | mysql) & !oracle)
Phrase search ('full text search')
Highlighting/snippets
Prefix matching (data:*)

Use pg_textsearch when you need:

High-quality relevance ranking (like Google/Elasticsearch)
Simple search box UX (type words, get ranked results)
Fair comparison across varying document lengths
Proper rare-term weighting

Hybrid Approach

You can combine both—use FTS for filtering and BM25 for ranking:

SELECT title, body <@> 'database performance' as score
FROM articles
WHERE body_tsv @@ to_tsquery('english', 'database & !mysql') -- FTS filter
ORDER BY body <@> 'database performance' -- BM25 ranking
LIMIT 10;

Taking step further with Semantic Search

Pure vector search might miss exact matches; pure keyword search misses synonyms. Hybrid gets the best of both — a search for “ML algorithms” would find documents about “machine learning techniques” (semantic) and those with the exact phrase (keyword).

This article outlines with code sample how to combine BM25 with Vector Search in PostgreSQL.

Key Takeaways

BM25 is what makes modern search engines feel “smart”—it’s not just counting words
pg_textsearch brings this to Postgres without needing Elasticsearch
The <@> operator returns negative scores (more negative = more relevant)
For production search, consider the hybrid approach: FTS for filtering, BM25 for ranking
The extension is still prerelease—not yet recommended for production

Resources

Django Native PostgreSQL Connection Pooling

Wed, 18 Jun 2025 00:00:00 +0000

Opening PostgreSQL connections adds overhead. Django 5.1 (released August 7, 2024) added native connection pooling¹, which lets an application reuse connections. Measure connection overhead and response times before and after enabling it; the benefit depends on your workload.

If you’re on Django 6+ and running async/ASGI, also see the async pool notes in How Django’s ORM Went From sync_to_async Threads to Native psycopg3.

Native Pooling vs External Pooling

PgBouncer runs as a separate pooling server. Django’s native pooling runs in the application process, so it does not require a separate server. It does require psycopg’s pool package. Third-party Django pooling packages and manual connection management are other options, with their own maintenance and configuration needs.

Configure Native Pooling

Requirements: Django 5.1+ and PostgreSQL (this feature does not work with psycopg2)²

Install the correct package:

pip install "psycopg[binary,pool]"

Critical: The package is called psycopg (not psycopg3). The pool functionality requires a separate package that this command installs automatically³.

Add these lines to your settings.py:

DATABASES = {
 "default": {
 "ENGINE": "django.db.backends.postgresql",
 "NAME": "your_db_name",
 "USER": "your_db_user",
 "PASSWORD": "your_db_password",
 "HOST": "your_db_host",
 "PORT": "5432",
 "CONN_MAX_AGE": 0, # Required - pooling fails without this
 "OPTIONS": {
 "pool": True,
 },
 }
}

Deploy this change. Your app now reuses connections instead of creating new ones for every request.

Prevent the Configuration Error That Breaks Production

Must set CONN_MAX_AGE = 0. Without this, Django throws ImproperlyConfigured: Pooling doesn't support persistent connections and your app crashes⁴. The pool manages connection lifetimes, making Django’s persistence redundant.

Scale for High-Traffic Production

Replace "pool": True with specific parameters for production loads:

DATABASES = {
 "default": {
 "ENGINE": "django.db.backends.postgresql",
 "NAME": "your_db_name",
 "USER": "your_db_user",
 "PASSWORD": "your_db_password",
 "HOST": "your_db_host",
 "PORT": "5432",
 "CONN_MAX_AGE": 0,
 "OPTIONS": {
 "pool": {
 "min_size": 4, # Keeps connections warm
 "max_size": 16, # Handles traffic spikes
 "timeout": 10, # Fails fast under extreme load
 "max_lifetime": 1800, # 30 minutes maximum connection age
 "max_idle": 300, # Close idle connections after 5 minutes
 },
 },
 }
}

Start with max_size equal to expected concurrent users divided by 10. Scale up if you see timeout errors in logs.

Monitor Your Performance Gains

Enable pool monitoring to track your improvements:

# In settings.py
import logging
logging.getLogger('psycopg.pool').setLevel(logging.INFO)

Watch pool utilization, connection wait times, timeout errors, and response times. Record median and 95th percentile latency before and after the change under comparable load. Use those measurements to size the pool and judge whether pooling helps your application.

Critical Fix for Threading and Celery Applications

Essential for background processing: This isn’t a pooling bug — it’s expected Django behavior that pooling simply makes painful. Any connection opened outside Django’s request-response cycle (for example, in a thread you spawn yourself) stays open until you explicitly close it. Without a pool you might not notice; with one, those un-returned connections exhaust max_size and you start seeing PoolTimeout errors⁵. Add explicit cleanup:

from django.db import connection
from threading import Thread

class DataProcessingThread(Thread):
 def run(self):
 # Your database operations here
 User.objects.filter(active=True).update(last_seen=timezone.now())

 # Critical: returns the connection to the pool
 connection.close()

For code that may touch multiple database aliases, Django’s documented helper django.db.close_old_connections() is the more robust choice — it closes all old or unusable connections across every configured database:

from django.db import close_old_connections

class DataProcessingThread(Thread):
 def run(self):
 try:
 User.objects.filter(active=True).update(last_seen=timezone.now())
 finally:
 close_old_connections()

Without this cleanup, each background thread permanently holds a connection from your pool.

Deployment Scenarios That Require Different Approaches

ASGI applications: On Django 5.1–5.2, Django’s documentation recommends against native pooling with ASGI — use PgBouncer or similar external poolers instead. This changed in Django 6.0, which ships an async-aware AsyncConnectionPool that works correctly under ASGI. I cover how that fits together with the truly-async ORM in How Django’s ORM Went From sync_to_async Threads to Native psycopg3.

Serverless deployments: Connection pools don’t persist across serverless invocations. Skip this optimization for Lambda/Cloud Functions.

Multi-tenant applications: Each tenant database needs its own pool configuration in DATABASES.

When to Evaluate Pooling

Evaluate pooling when connection establishment is a measurable part of request latency, concurrency is growing, or the application is approaching PostgreSQL connection limits. Compare measurements before and after rather than assuming a fixed latency or cost saving.

Django 5.1 Release Notes - Official Django documentation announcing native connection pooling support ↩︎
Django Database Configuration - Complete guide to Django database settings including pooling requirements ↩︎
psycopg Installation Guide - Official psycopg documentation for package installation and pool dependencies ↩︎
Stack Overflow: Django Pooling Configuration Error - Community discussion of the CONN_MAX_AGE requirement ↩︎
Django Ticket #35672 - Closed as invalid: connections opened in threads must be returned manually. See also the Databases → Caveats docs, which recommend django.db.close_old_connections() for connections created outside the request-response cycle. ↩︎