SQLite writer health

Every call to sqlitex.WithTx opens a real SQLite write transaction via BEGIN IMMEDIATE. Only one such transaction can be in flight at a time across the whole daemon. This page records per-caller wall durations and SQLITE_BUSY counts so we can tell who is holding the writer slot and who is being starved.

pool_wait

Time spent waiting for a connection from sqlitex.Pool via Pool.WithTx / Pool.WithSave / Read[] / Write[]. Large values mean the pool was starved — every conn was in use by someone else and this caller waited for one to be returned. Distinct from begin_wait, which is the writer-mutex wait after this caller already had a conn. Zero for bare-conn callers that bypass the pool (they pass a *sqlite.Conn directly to WithTx / Save).

begin_wait

Time spent inside BEGIN IMMEDIATE before it returned (includes SQLite's internal busy-handler backoff). Large values mean the writer slot was held by someone else while this caller waited.

hold

Wall time from successful BEGIN IMMEDIATE to COMMIT/ROLLBACK. Large values mean this caller is doing too much work inside the transaction.

total

Caller-visible end-to-end latency: pool_wait + begin_wait + hold. The single column an operator can use to spot "who is slow" without summing three percentile groups. The per-caller table is sorted by total p99 for this reason.

begin_busy

The busy_timeout expired before BEGIN IMMEDIATE could succeed; the gRPC client sees this as SQLITE_BUSY: database is locked. Each row carries a held by snapshot of whoever owned the writer slot at the moment of failure and a drained during wait list of holders that finished during the wait. For the aggregate "who caused all these busys" view see the Begin-busy attribution section below — usually the most useful starting point. begin_busy rows live in their own collapsed ring (separate from completed slow writes) so they can't evict real slow commits via their synthesised hold. The synthetic caller sqlitex.WALCheckpointer represents the background PRAGMA wal_checkpoint(PASSIVE) goroutine — it doesn't go through WithTx/Save, so without explicit hooks it would be invisible here; if it dominates HeldBy or DrainedDuringWait the disk's fsync floor is the bottleneck.

begin_busy_timeout

SQLite returned SQLITE_BUSY_TIMEOUT — the unix VFS's fcntl-based busy_timeout path expired. A file-level lock byte was contested for the full busy_timeout. If held by is also empty for these rows, that means no instrumented WithTx/Save was holding the writer mutex; the kernel-level fcntl byte was nevertheless unavailable. Diagnostic: another process holds the same DB file (multi-process SQLite is not our intended mode), or a kernel-level contention pattern is in play.

begin_busy_snapshot

SQLite returned SQLITE_BUSY_SNAPSHOT — a wal_index snapshot held by a reader couldn't be advanced past for the writer. Look at the Read operations table for callers with high hold p99; a long-held read transaction on the same connection blocks new writers from acquiring a fresh snapshot.

begin_busy_recovery

SQLite returned SQLITE_BUSY_RECOVERY — a hot journal needs recovery from a prior unclean shutdown of a writer. Self-healing: should be one-shot at the next successful write.

begin_interrupted

BEGIN IMMEDIATE returned a non-busy error — almost always SQLITE_INTERRUPT from a context cancellation upstream. Not a lock-contention event; no held by snapshot. Common when a caller scopes a tx to a deadline that has expired, e.g. connect.go's per-attempt timeout.

savepoint_top

Top-level SAVEPOINT on an autocommit connection that actually wrote at least once. The first DML/DDL statement promotes it to the writer-slot active set, so its hold time counts the same as a WithTx commit and it shows up on begin_busy "held by" snapshots. Pre-instrumentation these rows bypassed the page entirely, which is why the actual offender used to be invisible. begin_wait is renders as "—" for these rows and is excluded from the per-caller wait percentile: the SAVEPOINT keyword itself doesn't queue for the writer mutex (the deferred transaction is upgraded lazily by the first DML), so the only thing we could measure here is the SAVEPOINT statement's own ~µs execution time — not real contention. The actual writer-mutex wait, when it exists, is folded into the first DML's duration and shows up in hold.

savepoint

Nested SAVEPOINT issued while an outer transaction was already holding the writer slot. The outer tx is the lock holder — this row is recorded for completeness but excluded from hold percentiles to avoid double-counting.

contender events

Distinct begin_busy events where this caller was itself stuck inside Exec("BEGIN IMMEDIATE ...") — i.e., in SQLite's busy_handler retry loop — at the moment of another caller's timeout. Different from events (which counts begin_busys this caller helped CAUSE via the writer-slot it held during a wait window). Contender events counts begin_busys this caller SHARED with the victim because both were stuck in their own retry loops. A caller dominating this column is the source of a contention storm: too many parallel BEGIN IMMEDIATEs from one origin starve each other inside the busy_handler, even when none individually holds the writer mutex for long.

If you see begin_busy events with held by empty and drained during wait near-zero, the writer mutex was NOT actually held by an instrumented caller that completed during the wait window — and SQLite's exhausted busy-handler ran out anyway. Expand the also stuck in BEGIN IMMEDIATE block on the row: it lists, by caller name, every other goroutine that was also in BEGIN IMMEDIATE's busy_handler retry loop at the moment of timeout. A single caller name dominating that list across multiple begin_busy rows is your offender — its parallel BEGIN IMMEDIATEs are starving each other (and the victim) inside sqlite3_step. The aggregate version of the same signal is the contender events column in the Begin-busy attribution table below.

If the contender list is ALSO empty, cross-reference the outcome column subkind to identify the physical cause: begin_busy_snapshot points at long-held readers; begin_busy_timeout points at kernel-level fcntl contention or another process holding the DB file; begin_busy_recovery points at a prior unclean shutdown. The plain begin_busy label (no extended subkind) with no contenders and no holders means "BUSY-class error of unknown subkind with no instrumented contender" — a true mystery, escalate to direct VFS-level inspection.

Read-only SAVEPOINTs (Read[], ListEvents, ListPeers, etc.) are suppressed from this page entirely. They only ever held the SHARED reader lock and cannot cause SQLITE_BUSY on anyone's BEGIN IMMEDIATE; listing them would only dilute the writer-slot signal.

Temp-table writes are a known false-positive — but the worst offenders now opt out. Statements like INSERT INTO rbsr_iris ... write to per-connection TEMP tables, which live in a separate SQLite database file and don't take the main DB's writer lock. The instrumentation can't tell main-DB writes from temp-DB writes without hooking the VFS xLock callbacks, so any plain Save scope that runs even one INSERT/DELETE — even purely against temp tables — promotes to savepoint_top and counts toward the writer-slot percentiles. Callers that know their scope only writes to TEMP can opt out via sqlitex.SaveTempOnly / Pool.WithSaveTempOnly; the RBSR machinery (syncing.(*Server).loadStore, syncing.(*Service).DiscoverObjectWithProgress) does this, which is why they no longer dominate the Aggregate writer-slot utilization section. They render under Read operations instead. Cross-reference the captured statements on any remaining savepoint_top row before treating its caller as a real lock holder, and add a new opt-out only after auditing that the scope truly never writes to the main DB.

Reading hold vs begin_wait: these phases are independent — begin_wait is contention from others, hold is this caller's own work. They have no reason to track each other; in a healthy daemon hold is much larger than begin_wait because most of the time the lock is free when you ask for it.

Color thresholds: yellow ≥ 100 ms, red ≥ 1 s. The busy_timeout itself is 10 s.