If you run SQLite in WAL mode with a busy writer and a few readers, you may one day find a -wal file of several gigabytes next to a database that is much smaller. Autocheckpoint is on, nothing reports an error, and PRAGMA wal_checkpoint says it worked. I read the checkpoint code in SQLite 3.53.2, then built a small lab to watch it happen.

The short version: when read transactions overlap so that at least one is always open, the checkpoint can never finish and the writer can never go back to frame 1. In my runs the WAL grew by 11.9 MiB per second for as long as the overlap lasted, and when the readers went away it stayed at 357 MiB. A wal_checkpoint(TRUNCATE) every 5 seconds kept it under 60 MiB in 2 of 3 runs (119 MiB in the run where one call came back busy), at the price of stalling the writer for up to 1.8 s each time.

How WAL mode commits and checkpoints

In WAL mode a commit doesn't touch the database file. The writer appends the changed pages to the -wal file as frames (a 24-byte header plus one page), and a shared-memory index in the -shm file maps page numbers to the newest frame that holds them. Readers look there first and fall back to the database file.

A checkpoint copies frames back into the database file. Two counters in the shared index drive everything (wal.c L321-L401):

mxFrame is the last committed frame. nBackfill is how far the checkpoint has copied. When nBackfill reaches mxFrame, everything in the WAL is in the database, and the next writer can start again at frame 1 instead of appending. It overwrites the old file from the start. It doesn't truncate it.

By default SQLite runs a checkpoint after any commit that leaves the WAL at 1,000 frames or more (sqlite3WalDefaultHook, main.c L2470-L2481). That checkpoint is the PASSIVE kind: it never waits for anybody.

(Chart: see the original post.)

What a reader pins

A reader doesn't copy anything. When it starts a read transaction it pins one of four read marks (aReadMark[1..4], since WAL_NREADER is 5 and slot 0 is special) whose value is no greater than the mxFrame it saw. If a slot already holds that value it shares the slot with the readers already on it. Otherwise it writes its mxFrame into a slot it can lock exclusively, and if it can't get one, it settles for the largest older mark (walTryBeginRead, L3163-L3199). It holds a shared lock on that slot until the transaction ends.

Frames after its mark are newer than its snapshot. For pages changed in those frames, the reader still needs the old copy in the database file, so the checkpoint may not copy those frames over it until the reader finishes (L378-L381). Frames up to its mark are safe to copy.

If the WAL is fully copied when the reader starts, it takes slot 0 instead, which means "read the database file only, ignore the WAL" (walTryBeginRead, L3128-L3160). Remember that slot 0, it matters later.

Why the checkpoint stops at the oldest reader

walCheckpoint starts by assuming it can copy everything, then walks the four read slots (L2227-L2245):

mxSafeFrame = pWal->hdr.mxFrame;
mxPage = pWal->hdr.nPage;

for(i=1; i<WAL_NREADER; i++){
  u32 y = AtomicLoad(pInfo->aReadMark+i);
  SEH_INJECT_FAULT;

  if( mxSafeFrame>y ){
    assert( y<=pWal->hdr.mxFrame );

    rc = walBusyLock(
      pWal,
      xBusy,
      pBusyArg,
      WAL_READ_LOCK(i),
      1
    );

    if( rc==SQLITE_OK ){
      u32 iMark = (i==1 ? mxSafeFrame : READMARK_NOT_USED);

      AtomicStore(pInfo->aReadMark+i, iMark);
      SEH_INJECT_FAULT;
      walUnlockExclusive(pWal, WAL_READ_LOCK(i), 1);

    }else if( rc==SQLITE_BUSY ){
      mxSafeFrame = y;
      xBusy = 0;

    }else{
      goto walcheckpoint_out;
    }
  }
}

If it can lock a slot exclusively, nobody is using that mark, so it resets it: slot 1 to mxSafeFrame, the others to READMARK_NOT_USED (0xffffffff, unused). If it can't, a live reader is there, and mxSafeFrame drops to that reader's mark. The copy loop then skips every frame after mxSafeFrame (L2306), and nBackfill only advances to it (L2331).

Then comes the line that makes this hard to notice (L2340-L2345):

if( rc==SQLITE_BUSY ){
  /* Reset the return code so as not to report a checkpoint failure
  ** just because there are active readers. */
  rc = SQLITE_OK;
}

So PRAGMA wal_checkpoint(PASSIVE) returns busy = 0 even when it copied only part of the WAL. The signal is in the other two columns: log (frames in the WAL) keeps getting bigger than checkpointed.

Why the file grows, and why it stays big

Copying is only half of it. The WAL goes back to frame 1 in walRestartLog (L3869-L3909), which runs at the start of every write. It rewinds only if the writer's own snapshot is on slot 0 (pWal->readLock==0, so the WAL was fully copied when the writer started), and it can lock read slots 1 to 4 exclusively, meaning no reader is using the WAL.

Now put a steady writer next to readers that overlap. Some reader always holds a mark from a second or so ago. The checkpoint copies up to that mark and stops. nBackfill never catches up with mxFrame, so the writer is never on slot 0, so it never rewinds. Every commit appends. The -shm index grows with it, one 32 KiB block per 4,096 frames (HASHTABLE_NPAGE, L615).

The SQLite docs describe exactly this under checkpoint starvation: "if a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound."

And when the overlap ends, the file keeps its size. A rewind moves the write position to frame 1, but, as the same page says, a checkpoint "does not normally truncate the WAL file (unless the journal_size_limit pragma is set)". The file only gets smaller when:

• journal_size_limit is set, and the first commit after a rewind trims the file to that limit (L4210-L4221). The default in this build is -1, no limit.

• someone runs wal_checkpoint(TRUNCATE), which truncates it to 0 bytes (L2363-L2381).

• the last connection to the database closes, checkpoints and deletes the WAL.

The experiment

One writer process commits one row with a 1,000-byte blob per transaction, open loop at a fixed 2,000 commits per second, with synchronous=NORMAL. Reader processes each run BEGIN, a SELECT that takes the snapshot, hold it, look up 100 recent rows, COMMIT, then idle. A sampler records the file sizes every 100 ms.

Seven scenarios, 30 s each, 3 runs of each, interleaved:

• Scenario a writer only

• b 2 readers holding 0.4 s every 1.0 s

• c 3 readers holding 1.0 s, idling 0.2 s, staggered: at least 2 are always open

• d c, plus wal_checkpoint(TRUNCATE) every 5 s with a 5 s busy timeout

• e c, plus journal_size_limit of 4 MiB

• f b, plus journal_size_limit of 4 MiB

• g c, with wal_autocheckpoint=0 on the writer (the sampler's once-a-second probe still checkpoints, see below)

Here is what the readers actually did in 3 seconds of one run of b and one of c, straight from the logged open and close times:

(Chart: see the original post.)

And here is the WAL file:

(Chart: see the original post.)

With gaps (b), the WAL rewinds twice a second, once per gap, 60 times in 30 s, and peaks at 6.05 to 6.08 MiB. With overlap (c), it never rewinds once, and grows in a straight line at 11.91 MiB/s in all three runs. That is about 1.5 frames of 4,120 bytes per commit: the final checkpoint counted 90,873 frames for 60,000 commits. At the end the -shm file was 736 KiB, 23 index blocks.

Through all of c, every PASSIVE probe returned busy = 0, while log ran up to 2,751 frames ahead of checkpointed. That is a little under one second of writes, about one reader hold time.

After the readers leave

When the run ended I stopped the readers and ran one more PASSIVE checkpoint with nothing else open. It copied everything, returned 0 | 90873 | 90873, and the file stayed where it was:

(Chart: see the original post.)

journal_size_limit did nothing in e. Its trim runs on the first commit after a rewind, and under overlap there are no rewinds. In f, where there are, it held the file at 4.00 MiB, though the high-water mark still reached 6.01 MiB between trims. In one of the three e runs the file was already 4.00 MiB before that final checkpoint. Most likely the readers closed a moment before the writer's last commits during shutdown, so one commit found a fully copied WAL, rewound and was trimmed: the same mechanism, with the blocker gone.

What TRUNCATE costs

The non-passive modes behave differently (sqlite3WalCheckpoint, L4295-L4426). FULL, RESTART and TRUNCATE take the writer lock first, waiting with the busy handler. Because xBusy is set, walBusyLock then waits on each reader slot instead of giving up. RESTART and TRUNCATE also wait until no reader is using the WAL at all (L2358-L2362), and TRUNCATE then cuts the file to 0 bytes. Readers that start after the WAL is fully copied take slot 0, which doesn't hold the checkpoint up, and in my runs the readers' 100-row lookup was no slower in d than in c (226 against 224 µs median). The SQLite docs do warn that readers might block while it runs. The writer is blocked for sure, because the checkpoint holds its lock.

In d, 14 of 15 TRUNCATE calls reset the WAL and returned 0 | 0 | 0. They took 1.40 to 1.81 s (median 1.70 s) with readers that hold for 1.0 s. That is close to two hold times, and the code explains why: first the call waits for the readers pinned below mxFrame, but while it waits, readers that start see a WAL that isn't fully copied yet, so they take a real mark at the new mxFrame. The RESTART step then has to wait for those too. On all 14 calls, the checkpoint returned within 105 ms of the last close of a reader that had started before the copy finished.

Results

"Response time" is when a commit finished minus when the open-loop schedule said it should start, so a stall counts in full. Time inside execute() hides it: only one commit per TRUNCATE actually waits, and the thousands queued behind it run fast once it lets go. If you measure the effect of a checkpoint job on your writer, measure response time.

The one TRUNCATE that failed came back in 1.68 ms with 1 | -1 | -1. That is the WAL_CKPT_LOCK check at the very start (L4336): if another checkpoint is already running, every mode returns busy without calling the busy handler. My probe wasn't running at that moment, so the other checkpoint was almost certainly the writer's own autocheckpoint, which under starvation runs after every commit. That call missed, the WAL grew for another 5 s, and that run's high-water mark was 118.86 MiB instead of about 60.

Two smaller results

Checkpointing on the writer's connection costs it something under starvation. Once the WAL is past 1,000 frames, the autocheckpoint runs after every commit and copies whatever it can. In g the writer's autocheckpoint was off, and the only checkpoint was the sampler's PASSIVE probe once a second from another connection. That dropped the writer's p50 from 38 to 32 µs and its p99.9 from 2,653 to 694 µs. The WAL grew exactly as fast, and the probe kept nBackfill as close as the autocheckpoint did (at most 2,743 frames behind, against 2,751 in c), since the checkpoint wasn't the thing holding the WAL down anyway. Reads did not get slower, and the code says they shouldn't. A reader doesn't search the whole WAL index. walFindFrame walks hash blocks only from its mxFrame down to the block holding minFrame (L3568-L3569), and minFrame is nBackfill + 1 when the read transaction starts (L3253). So lookup cost depends on how far the checkpoint lags, not on the size of the WAL. Under starvation the lag stayed at most 2,751 frames, less than one 4,096-frame block, so a lookup searched 1 or 2 blocks. In c the median was 216 µs for transactions that started 5 to 12 s into the run and 218 µs for 22 to 30 s, at a WAL of up to 357 MiB. Reads would slow down only if nBackfill itself fell far behind, for example with autocheckpoint off and no other checkpointer. I didn't measure that.

What to do in your app

Keep read transactions short. The problem is overlap, and overlap comes from long snapshots: a cursor left open while you call an API, a BEGIN that a connection pool never closed. In Python, a SELECT whose rows you haven't fully fetched keeps its statement active, and an active statement keeps its read snapshot. Watch for it.

Poll PRAGMA wal_checkpoint(PASSIVE) and alert when log - checkpointed keeps rising, and watch the size of the -wal file. The busy column won't tell you.

Set journal_size_limit, for example:

PRAGMA journal_size_limit = 67108864

for 64 MiB. It doesn't stop starvation, but it gives back the disk the next time the WAL rewinds.

Run wal_checkpoint(TRUNCATE) or (RESTART) from a maintenance job with a busy timeout, when writes are quiet. Expect it to block the writer for up to about twice your longest read transaction. If it returns busy = 1, retry soon rather than waiting for the next period.

If you turn off autocheckpoint, checkpoint from another connection. In g, one PASSIVE checkpoint a second from a separate connection kept up as well as the autocheckpoint and took that work off the writer. With nothing checkpointing at all, nBackfill never reaches mxFrame, so the WAL never rewinds, gaps or not.

How I measured

Machine: Apple M5 Pro (18 cores, 48 GiB), macOS 26.6.2, APFS on the internal SSD.

Software: Python 3.14.4 and the SQLite it links, 3.53.2 (DEFAULT_WAL_AUTOCHECKPOINT=1000, DEFAULT_JOURNAL_SIZE_LIMIT=-1, from PRAGMA compile_options).

The source links above are pinned to the version-3.53.2 tag, because that is what Python links here. 3.53.3 and 3.53.4 are out.

Runs: 7 scenarios × 3 runs × 30 s, interleaved. The first 5 s of each run are left out of the latency statistics. Tables show the median of 3 runs.

The writer is open loop, so every run issued all 60,000 scheduled commits, and all of them completed with no write errors. That says nothing about stalls; the response times above do.

Caveats: The sampler's PASSIVE probe once a second is itself a checkpoint, and it runs in every scenario. Where autocheckpoint is on, it copies no more than the autocheckpoint would. In g it is the only checkpointer, so g means "one PASSIVE checkpoint a second from another connection", not "no checkpointing". --probe-every 0 turns it off.

The first full pass had a probe colliding with the TRUNCATE calls and a maintenance call firing after the run had ended. I fixed both and reran everything. All numbers here are from the second pass.

This is one machine and one filesystem. I did not run the Linux cross-check or a sweep of write rates. "MiB" is 2^20 bytes.

The scripts, the raw CSVs and how to rerun them are in the lab folder. python3 run.py runs the full matrix in about 12 minutes with nothing but the standard library.

Credit

Credit goes to D. Richard Hipp and the SQLite developers, who wrote and maintain the WAL code and whose WAL documentation already describes this failure and the "reader gaps" fix in plain words. The code is commented well enough that a stranger can follow a checkpoint line by line, which is how this post got written.

The same page also documents this year's WAL-reset bug, a rare corruption case when two connections write or checkpoint at the same instant, which one of the SQLite developers, Dan, found and fixed in 3.51.3.

The short version: on a 1 million row bulk insert, the foreign key check costs about 44% less time in Postgres 19. A second optimization that was pulled from 19 before release would have helped much more, but only when the foreign key values arrive in order.

The old path: a query per row

A foreign key is enforced by an internal AFTER INSERT trigger, RI_FKey_check in ri_triggers.c. In Postgres 18 that trigger builds this query once, saves the plan, and runs it through SPI (the server's internal SQL interface) for every inserted row:

SELECT 1 FROM ONLY parent x WHERE id = $1 FOR KEY SHARE OF x

FOR KEY SHARE is the important part. It locks the parent row so nobody can delete it or change its key while your transaction is open, but it still lets other sessions update the parent's non-key columns.

The plan is cached, but every row still pays for SPI setup, executor startup and teardown, snapshot handling and the permission checks, just to do one index lookup.

Same check, same lock. Postgres 19 removes everything between the trigger and the index.

The new path: probe the index directly

Amit Langote's commit 2da86c1ef9 adds a fast path. Before touching SPI, RI_FKey_check now asks two questions:

if (ri_fastpath_is_applicable(riinfo) &&
    ri_FastPathCheck(riinfo, fk_rel, newslot))
    return PointerGetDatum(NULL);

ri_fastpath_is_applicable says no for two cases: when the referenced table is partitioned (the probe would need to be routed to the right partition), and for temporal foreign keys (range containment needs more than a single lookup). Later fixes added more fallbacks: the referenced index has to be a btree, and its collation must match the column's.

If the answer is yes, ri_FastPathCheck does what the query used to do, by hand:

1. Opens the parent table and its unique index.

2. Takes a snapshot. In the shipped version this happens after the table lock is granted. Taking it earlier opened a window where a parent row committed while you waited for the lock would be invisible, and the check would fail for a key that exists (commit 1390182683).

3. Builds scan keys from the new row's values and probes the btree with index_getnext_slot.

4. Locks the row it found with table_tuple_lock(..., LockTupleKeyShare, ...) in ri_LockPKTuple. That's the same lock FOR KEY SHARE takes.

5. If the lock had to follow an update chain to reach the newest row version, recheck_matched_pk_tuple checks that the key still matches. That is the fast path's version of the recheck the executor does for SELECT ... FOR UPDATE.

It also switches to the parent table owner's user id for the lookup, the same way the SPI path does, so permissions and row-level security behave the same.

The benchmark

A parent table with 1 million integer primary keys, and a child table with a foreign key to it. Each run inserts 1 million child rows in one INSERT ... SELECT, after a TRUNCATE and a CHECKPOINT. The same insert into a table with no foreign key is the baseline. Six runs per version; the charts show the median.

Median of 6 runs, 1 million rows per insert. Lower is better.

With scattered keys, the foreign key check itself (the time above the no-foreign-key baseline) drops from 2.17 s in 18.6 to 1.21 s in 19. The whole insert is 1.66x faster.

The control: turn the fast path off

To check the gain really comes from the fast path, I ran the same test with a partitioned parent table. The fast path refuses partitioned parents, so 19 falls back to SPI:

Partitioned parent, scattered keys. The gap nearly disappears (about 5%).

Two things show up here. The gain disappears when the fast path is off, so the fast path is what made the difference. And a foreign key to a partitioned table is expensive either way: about 3.5x slower than the SPI path on a plain table, because every check has to find the right partition first.

The batching that was pulled

Three days after the fast path, a second commit went further. Instead of probing the index once per row, it buffered up to 64 rows and checked them together. For single-column keys it passed all 64 values as one array scan key (SK_SEARCHARRAY), so the btree sorts them and walks its leaf pages once instead of descending from the root 64 times. The commit reported about 2.9x in total over the old code.

That batching is in the 19 betas but not in 19.0. On September 10 it was removed from REL_19_STABLE. Buffered checks have to survive across trigger calls and be flushed at the right moment, and that state has to be right under nested triggers, subtransactions, deferred constraints and SET CONSTRAINTS. Miss one case and a buffered check never runs: the transaction commits a real foreign key violation without an error. The commit message says it plainly: with 19 close to release, there wasn't time to be sure every case was covered. Batching stays on master for a later release. That's a hard call to make that close to release, and it's the right one.

My first measurement disagreed with the 2.9x. With scattered keys, master (batching included) is only about 2% faster than 19. The array scan only saves work when the 64 values land on the same few leaf pages. My keys were deliberately spread over the whole index, so every value still needed its own leaf.

So I reran with the keys in order (parent_id = 1, 2, 3, ...), which is what you get when child rows are loaded in parent order. Now batching does what the commit said: 701 ms against 1,021 ms for 19, and 2.8x against 18.6, close to the reported 2.9x.

What this means if you run Postgres

• Bulk loads into tables with foreign keys get faster in 19 for free, as long as the referenced table isn't partitioned and the key has a btree index (primary keys and unique constraints do).

• Foreign keys to partitioned tables don't get this. If you bulk-load into a table that references a partitioned table, the check still costs about 7 µs per row.

• Load order matters more after batching lands. Once batching ships in a later release, loading child rows sorted by parent key could make the check nearly twice as cheap again.

• Only the insert-side check changed. ON DELETE CASCADE, SET NULL and the other action triggers still go through SPI, as the original commit explains.

How I measured

• Machine: Apple M-series laptop. Each server ran in Docker (a Linux arm64 VM) with 4 CPUs and 4 GB of memory, one server at a time.

• Settings: shared_buffers = 1GB, max_wal_size = 8GB and checkpoint_timeout = 30min on every version. Everything else was left at the defaults.

• Versions:

  • 18.6 and 19beta3 are the official Docker images.

  • "19" is REL_19_STABLE at 8fbbb338be, which already has batching removed, built from source with -O2.

  • "master" is master at 4545cee, built the same way.

  • The 19beta3 image still includes batching and measured 1,529 ms on scattered keys, close to both 19 and master.

• Caveats:

  • The source builds use a different compiler setup from the packaged images. That probably explains part of the small difference in the no-foreign-key baseline (332 ms against 299 ms).

  • These are single-session numbers. Concurrent inserts and lock contention are a separate test.

All scripts, Dockerfiles and raw results are in the lab folder. Run ./run.sh <label> <image> and you get the same table.

Credit goes to Amit Langote, who wrote the fast path, the batching and roughly thirty follow-up commits to this code this year, and to everyone on pgsql-hackers who tested the betas hard enough to find the problems before release.

Originally published at getnobody.org. Benchmark scripts and raw results: getnobody.org/labs/pg19-fk-fastpath.