Circa 2022
At my first job we didn’t use an ORM. We used a thin Postgres driver and wrote our own data layer on top of it.
That sounds like masochism, but it bought us something (and was a learning experience for me). Every table had a corresponding data model in the application: a class, with compile-time types and runtime validation, that knew what it was. A policy document was a PolicyDocument. It had an issuer, an effective date, a status, a version, and other metadata. The actual PDF lived in a cloud bucket and the model held a link to it. We were writing something close to a rich domain model, and the database was where those objects went to sleep.
Any field in the model that would be needed to filter, sort, or join on became a column in the table. Issuer, status, effective date, the foreign keys. Typed, indexed, cheap. Everything else went into a single jsonb column.
The reasoning there was about access patterns rather than effort. Every one of those remaining fields was only ever read as part of loading the whole object, and nobody was ever going to write WHERE <jsonb_col>->>'reviewer_note' = .... They existed to be rehydrated into a PolicyDocument and handed to business logic, and they had no independent life as query targets.
So the row was a small, sharp, queryable index (single, composite, whatever was needed without index bloat) into an object, with the object’s long tail folded up behind it.
I liked this model, it was just...clean. I guess it scratched an itch (haha). Not that I’ll use it any more because there’s a lot of overhead that any decent ORM library takes care of anyway, and I don’t want to be reinventing the wheel.
Fast forward to 2026.
There’s a system indexes an org and produces a report. The report is a JSON. It is a large JSON. It’s stored in a jsonb column on a table whose other columns are an id, an org id, a status, and some timestamps.
That’s the entire design, and it had been running in production for a while, eventually spiking the database CPU (to a number big enough to raise eyebrows) once large enough orgs had started getting indexed.
After a lot of Codex-enabled code sifting to get to the cause, I landed on the schema, found the column, and got that specific “oh no...” feeling you get when the bug you’re chasing turns out to be the architecture. So I did what you do: days of metrics, storage measurements, controlled reproductions, a read-only RCA doc.
And that doc is why this post exists, because it answered “why is the CPU pinned?” a long time before it answered the question I actually got stuck on, which is how something this wrong stays invisible for weeks. That answer lives about four layers below the SQL, in machinery you never interact with directly and that Postgres works very hard to make sure you never have to think about.
So let’s go look at it. Everything after this section is going to cash out. Or you can straight jump to “The story”.
Before we go further: what Postgres is actually doing under the hood
Pages:
Postgres has no mechanism for reading a single row off disk. What it actually works in is the page: 8 KB, the atomic unit of heap tables, indexes, all of it.
A table on disk is an array of 8 KB pages in a file. When you ask for one row, Postgres finds the page holding it, reads all 8 KB into memory, and picks your row out of it. There is no smaller unit of I/O, and no such thing as fetching half a page.
Inside a page you get a small header, then an array of line pointers growing forward from the front, then tuples (row versions) growing backward from the end, with free space in the middle. The line pointers are a level of indirection, so a row can shuffle around inside its page without anything that points at it having to care.
That 8 KB number is going to explain almost everything else, keep it in your mind’s L1 cache.
Shared buffers:
Reading a page from disk every time the database has a query is absurd. A memory access lands in something like a hundred nanoseconds; the same page off a local SSD costs tens of microseconds, and on a cloud instance, where the “disk” is almost always network-attached storage rather than a physical device bolted to the machine, you’re into milliseconds. That’s two to four orders of magnitude, and the backend pays it synchronously, blocked, doing nothing else while it waits. So Postgres keeps a cache of pages in memory called the shared buffer pool, an array of slots each exactly one page wide, sized by shared_buffers.
When a backend wants a page it checks the buffer pool first. On a hit it uses the copy in memory and does no I/O at all. On a miss it has to find a free slot, read the page off disk into it, and then use it.
There’s a second cache underneath. Postgres reads through the filesystem, so the OS keeps its own page cache of the same data, which means a page you read often typically exists twice in RAM: once in Postgres’s buffer pool, once in the kernel’s. This is why the standard advice is to give shared_buffers about a quarter of your memory and not all of it, you’re sharing that RAM with the kernel’s copy of the same pages whether you like it or not.
When a backend modifies a page, it modifies the cached page in memory rather than the physical location of the data on disk, and that page then becomes dirty, meaning the change now exists in memory and nowhere else. Nothing goes to storage yet. Your UPDATE returning successfully does not mean your data page was written anywhere. Before that dirty page can be evicted, it has to be flushed to disk, or the change goes with it (and if the machine dies before that flush ever happens, what saves you is the WAL, which is next).
Eviction:
The buffer pool is finite, so when every slot is full and you need one more page, something has to go. Postgres picks the victim with a clock sweep: each buffer carries a usage counter, a hand rotates around the pool decrementing counters, and the first buffer it finds with a count of zero and nobody currently using it gets evicted.
Here is the part that matters. If the victim is clean, eviction costs nothing, because you can overwrite the slot and move on. If the victim is dirty, it can’t simply be dropped, because that memory holds the only copy of the change. The backend that just wanted to read something must now stop and write that page out before it can have its slot.
Which means a plain SELECT can find itself flushing a page it never touched, on behalf of a transaction it has nothing to do with.
And notice what eviction really costs you, which isn’t the memory. The page you evicted was useful, and the next query that needs it now pays a disk read it used to get for free. A workload that drags a large volume of pages through the pool isn’t merely using cache, it’s destroying cache, for everyone, including queries that have nothing to do with it.
The write-ahead log (WAL):
So dirty pages sit in memory. Before any change to a data page is allowed to reach disk, a record describing that change must already be durably on disk, in a sequential log. Log first, then data, which is the whole name: write-ahead.
Every write you do gets recorded twice, once as a WAL record and once eventually as the data page itself. Such is the price of not losing your data (and why write volume is never merely the size of what you wrote).
The first time a page is dirtied after a checkpoint, Postgres writes the entire 8 KB page into the WAL, not just your change. It’s called a full page image, and it exists because a crash mid-write can leave a page half-updated on disk, and a diff applied to a torn page is garbage. So a two-byte change to a fresh page can cost you 8 KB of log.
The WAL is also the reason dirty pages get to be lazy. The change is already safe in the log, so Postgres can flush the data page whenever it feels like it, in bulk, at a checkpoint.
MVCC (you never actually update anything):
Postgres is MVCC, which means concurrent readers never block on writers. It buys that with a specific trick: an UPDATE does not modify a row. It writes a new version of the row somewhere and stamps the old version as expired as of your transaction. Both versions physically exist, and which one you see depends on your snapshot.
Physically, then, an update is an insert plus a tombstone. The old version stays on disk, taking up space and getting read into buffers along with everything else on its page, until VACUUM comes along, determines that no live transaction can still see it, and reclaims the space.
So updating a row costs more than writing the new data. It costs the new tuple, WAL for it, index entries pointing at it, and a corpse that somebody has to clean up later. Update a row often enough without adequate vacuuming and the table bloats, spreading the same live data over more and more pages, each read dragging more dead weight through your buffer pool.
There’s an optimization called HOT (heap-only tuple). If your update doesn’t touch any indexed column, and the new version fits on the same page as the old one, Postgres chains it in place and skips updating the indexes entirely. HOT is great. Remember its two conditions, because HOT is going to show up later wearing a disguise.
TOAST:
Which brings us to the question this has all been walking toward: what happens when a value doesn’t fit in an 8 KB page?
Postgres wants at least four tuples per page, which caps a tuple at roughly 2 KB. Go over that and TOAST engages: The Oversized-Attribute Storage Technique (yes, it’s a backronym).
TOAST tries two things in order. First, it compresses the oversized variable-length attributes, and if that gets the tuple under the limit the value stays in your row, compressed, and you’re done. If it’s still too big, TOAST moves the value out of line: out of your table entirely, into a separate companion relation, a different file on disk, created automatically the moment you declared a jsonb column and which you have probably never looked at.
That companion table has three columns: a chunk id, a chunk sequence number, and a bytea of chunk data. Your value gets sliced into chunks of just under 2 KB each, sized precisely so that four chunk rows fit in one 8 KB page. Same arithmetic as before, applied one level down.
So a value that is N bytes after compression becomes ceil(N / 2 KB) rows, each with its own tuple header, each with an entry in the TOAST table’s index, each living on a page that has to be located and read individually. Do that division for a value of any real size and the answer comes out in the thousands, which is the point at which you should stop picturing a report as one value and start picturing it as a small table.
The pointer isn’t a pointer:
The genuinely beautiful (horrible) part.
What’s left in your actual row, where the value used to be, is 18 bytes, and you can account for every one of them:
- 2 bytes: a header saying “I am a TOAST pointer”
- 4 bytes: the original size
- 4 bytes: the stored size after compression
- 4 bytes: a chunk id
- 4 bytes: the OID of which TOAST table to look in
That accounts for all eighteen, and notice what the list doesn’t contain anywhere: an address.
So this isn’t a pointer in the C sense, because there is nothing to dereference. What it actually is, is a foreign key. To get your value back, Postgres takes that chunk id, walks a B-tree index on the TOAST table, finds the chunk rows, reads each of their pages, concatenates the pieces in sequence order, and decompresses the result. Only then do you have a value. You never wrote that join, you don’t see it in your query, and it fires on every single read.
Your data isn’t adjacent to your row, or near your row, or even in the same file as your row. The next logical byte after that 18-byte stub is whatever unrelated column came next in the tuple, while your report sits somewhere else entirely, scattered across a side relation in an order nobody promised you.
And one more thing, since this is where the last section pays off. The planner keeps statistics on your column, and they report an average width of 18 bytes, because the on-row datum genuinely is 18 bytes. I assumed this was a bug for about ten minutes before I went and read the source, where ANALYZE carries a comment saying, in effect, if the value is toasted, we use the toasted width. It’s deliberate. The column that dominates your entire storage footprint is, to the query planner’s cost model, narrower than a UUID.
Stitching it together:
Now put the whole machine in one frame and watch what a single large jsonb write actually costs.
You replace one value in one row. Postgres compresses your payload, allocates a fresh chunk id, and inserts a thousand-ish chunk rows into the TOAST relation. Those chunk rows land on hundreds of pages, all of which get pulled into the buffer pool and dirtied. Every one of those inserts generates WAL, and every page dirtied for the first time since the last checkpoint drags a full 8 KB page image into the WAL along with it. The index on the TOAST table needs an entry per chunk, which dirties more pages, which makes more WAL.
And MVCC is still MVCC through all of this, so the previous version of your value is sitting right there, every chunk of it. TOAST values never get updated in place, only inserted and deleted, so the old chain stays on disk as garbage until vacuum gets to it. You didn’t overwrite a report, you wrote a second one and left the first one lying around.
Then all those dirty pages need slots, and the buffer pool only has so many. The clock sweep starts evicting to make room, and what it evicts is whatever was already in there, which is the small, hot, deeply boring index pages that make your ordinary queries fast. Those queries go to disk now, and on a small instance that’s the same disk and the same CPU your write is already saturating.
Reading it back is the same story with the arrows reversed: index walk, hundreds of page reads, decompress, parse into a JSONB tree, materialize. And critically, shared buffers cache the compressed chunk pages, not the parsed value. There is no shared, ready-to-use copy of your report anywhere in memory, so every backend that wants it rebuilds it from chunks, privately, in its own memory, from scratch, every single time.
None of this is Postgres doing something wrong. Every mechanism above is load-bearing, correct, and part of why Postgres is trustworthy, and TOAST in particular is a fantastic piece of engineering. All of it was simply designed for values that are occasionally too big, on rows that are otherwise normal rows.
The story
So: a report. One large JSON. One jsonb column. A row that is otherwise an id, an org id, a status, and some timestamps.
Go back to the first job for a second, because the difference isn’t “we used jsonb in both places.” There, the jsonb was the tail of an entity. The row was a real object with real columns, and the jsonb carried the leftovers that nothing ever queried. TOAST barely engaged, and when it did it was on the fat outliers, which is the exact case it exists for.
Here the jsonb is the entity. The scalar columns are vestigial (strip the report column out and there’s no object left, just a receipt saying a job finished). The table isn’t modelling a thing that has a report, it’s a filing cabinet with one blob per row and a label taped to the front.
What’s in that cabinet is a file, and you’ve just read eight sections on why Postgres is an expensive place to keep files.
Nobody sat down and decided this, which is the part that gets me. Nobody ever does. You get a jsonb column when the persistence layer is whatever the autocomplete suggested and there’s nobody in the room with enough database instinct to ask what happens when this grows. It worked in dev. It kept working for weeks, and by the time it stopped working the shape was load-bearing, which is its own problem.
Here’s the bill, in ratios (the ratios are the point, and the actual numbers aren’t mine to publish).
The table is not in the table. The TOAST relation is over eight thousand times the size of the parent heap it belongs to. Virtually the whole table, by volume, is in the annex. Read that as an architecture statement rather than a storage stat: Postgres looked at this data, decided it did not belong in a row, and quietly built a blob store to cope. We already had a blob store, it’s called a bucket, and it doesn’t attach MVCC and a write-ahead log to your object graph.
The control group. That same schema has two other jsonb columns (one holds per-issue metadata, the other a bit of config). Neither has ever crossed the toast threshold, so neither has been moved out of line in its life, and the TOAST relations sitting behind those tables are empty enough to round to nothing. Same database, same engine, same data type, same everything. The report column is four orders of magnitude bigger than either of them and it’s the only one that’s a problem, so whatever’s wrong here, it isn’t jsonb.
Reads. Same query, same rows, cache-warm, no disk I/O at all. Select the scalars and it’s instant, one buffer. Force it to materialize the report and it’s four orders of magnitude slower, touching about 200 times more shared buffers (in production, the count for the same three rows went from one to several hundred).
Four orders of magnitude with everything already in memory. All of that is the machinery from the last section, paid per backend, on every read: index walk, chunk gather, decompress, parse, materialize.
There’s a second multiplier hiding in that word “materialize”. What sits on disk is the compressed form. What crosses into your backend’s private memory is the logical JSON, which runs about seven times larger. So the size you’d get from measuring your storage isn’t the size that lands in RAM, and it lands there once per backend, per read, with no shared copy.
Writes. One completed cycle generated several times the report’s own stored size in WAL alone, and well over ten times its size in total disk writes. You know where all of it went now: chunk inserts, full page images, TOAST index maintenance, and a dead previous chain sitting there waiting on vacuum.
The one that stopped me was disk reads, at roughly seventeen times the report’s size, on a cycle whose entire job was to write. Index lookups to find things, pages fetched in so they could be dirtied, the read half of writes that nobody thinks of as reads.
And here’s HOT in its disguise. Almost every update to that table qualifies as HOT, because the indexed columns don’t change, so by the parent table’s statistics this workload looks clean. That’s the bit that took me longest to see. HOT only ever protected the parent’s indexes, it has no opinion at all about the thousand chunk rows being rewritten one relation over. The metric that was supposed to tell us the writes were cheap was measuring the one part of the write that was.
Mean CPU on that instance sits at about a sixth of capacity while the p99 is up at the ceiling. I’ll be straight about attribution here, because the RCA had to be: not every one of those spikes is the report’s fault. That database has a separate self-inflicted problem where migration logic runs on every container start, and that owns most of the long plateaus. But the short violent ones have a completely different signature, WAL and disk and memory all spiking together, and they line up with report finalization. They look like exactly what the last section says they should look like: an idle database, and then one write takes the whole machine hostage and drags every unrelated query down with it, through a buffer pool it just finished evicting.
The trap is that Postgres is too good at this
Every problem above has a Postgres answer. Value too big? TOAST handles it, automatically, silently. Compression too weak? Change the strategy, or switch to LZ4. Fetching too much? Project individual keys with -> instead of selecting the column. Need to search inside it? GIN index. Reads still slow? Redis. CPU too high? The instance resizes with one click.
Every one of those works, which is what makes it a trap.
Postgres will meet you at every step of a bad idea, so you never hit the wall that would have made you stop. You get a slope instead, and every individual step down it is defensible on its own. Nobody ever sees an error saying this value does not belong in a database. You get a slightly slower Tuesday, and then a slightly slower Tuesday, and eighteen months later your p99 is a cliff and your table is a rounding error inside its own annex.
A bucket is worse at all of this, except for the one thing that matters, which is that it makes you decide. Writing to a bucket forces you to say out loud that this is an artifact with a lifecycle and not a field, and once you’ve said it out loud the rest falls out for free: keep the scalars you actually query as columns, keep a reference and a hash and a schema version, let the big immutable thing live where big immutable things live.
Which is what we were doing at the first job with those PDFs. I didn’t realize at the time that it was a decision. It just looked like where the PDF goes.
So, should you put JSON in Postgres?
Frankly, even though I wrote the article, I’m still not comfortable with the question in that heading. The real one is:
Is this JSON a property of the thing you’re modelling, or is it the thing itself?
If the row is a real entity with real queryable structure, and the jsonb is a long tail of attributes that only ever get read as part of loading that entity, then yes, use it, that’s what it’s for. It stays small, it rarely toasts, and it rides along on a row that has its own reason to exist.
“It’s a bag of fields with no query pattern and I don’t want thirty nullable columns” is a real reason.
“I don’t want to write a migration” is a deferral.
And jsonb really is the most convenient “feature” that makes these deferrals seem like decisions. It can take you away from modelling your data the way it needs to be.
And if the JSON is the entity? Large, immutable once written, always consumed whole, sitting on a row that only exists to hold it? Then it’s a file, so treat it like one. Put it in a bucket, keep the queryable scalars in Postgres next to a reference and a hash and a schema version, and let the database go back to being a database. If you genuinely need to query deep inside those documents then you don’t want a JSON column in a relational store at all, you want a document store, and you should go get one on purpose instead of building a worse one out of TOAST chunks and Redis.
Postgres will let you store the report. It’ll compress it, chunk it, index it, cache it, serve it, and it’ll do every one of those well enough that you won’t notice for far longer than you should. It never once told us no, and I’m still turning that part over.
Postgres offers you flexibility. Some flexibility is good, but good lord, using too much of anything is bad.
More writing: From the Birthday Paradox to Consistent Hashing. · This piece is also on Substack.