Seed daemon network health

peer_select

One DB query — which peers do we ask? Should be milliseconds.

connected_sync

Reconcile + download from already-connected peers in parallel. Waits for ALL of them. This is what dominates the user-visible "click → content" delay.

dht_discover

Only fires if connected_sync didn't find content. Asks the Kademlia DHT for providers of the requested CID.

dht_sync

If DHT returned providers, reconcile + download from them. Often empty in healthy operation.

connected

Content was found via an already-connected peer. The fast happy path.

dht

Already-connected peers didn't have it; DHT discovery + sync succeeded.

notfound

Nothing found anywhere. User sees a stuck spinner or "not found."

error

Discovery aborted with an error.

For each individual peer in a sync, four phases run in order:

dial

Open the gRPC-over-libp2p stream and run the protocol-version check. Fast on warm peers.

reconcile_rpc

Per-round timing of the RBSR set-reconciliation RPC. A single sync can run several rounds. This is observed once per round, not once per sync.

bitswap_fetch

After RBSR identifies which blobs we want, the bitswap engine pulls them from any connected peer.

putmany

Bulk-insert the downloaded blobs into our local SQLite.

How each bitswap download loop terminated. One increment per fetch.

complete

Channel closed naturally — bitswap delivered everything we asked for. The good outcome.

idle_timeout

40s passed with no new block arriving, so we gave up. Highlighted red if >20% of fetches hit this.

ctx_done

Context was canceled mid-flight (caller went away). Should be near zero.

Same per-call timing as the bitswap_fetch row above, but split by termination reason. Watch the complete row's p90/p99 — that's "real fetches, how long do they take" without the timeout cases skewing the picture.

For each fetch: time between the LAST block received and the moment the download loop exited. Tells us what each outcome actually means in practice.

complete

Near zero is healthy — bitswap closed the channel right after the last useful block. Larger values mean the channel lingered after delivery (wasted wait).

idle_timeout

Tautologically equals the idle-timer value (~40s).

ctx_done

How much time we'd already been idle when cancellation hit.

Per fetch: downloaded ÷ wanted. A row at ≤ 1.00 with most of the count means we got every blob asked for. Counts in lower buckets mean blobs are missing — either gateways don't have them or bitswap couldn't find them within the idle window. mean=1.00 healthy; mean=0.85 means we lose ~15% of asked-for blobs on average.

For each peer we sync with, RBSR computes "blobs the peer has that I don't" and asks bitswap to fetch them. This histogram is the size of that wantlist per peer-sync.

healthy

Mean near 0, with most observations in the ≤ 1 or ≤ 10 buckets — the diff is small because we already have most of what each peer has.

unhealthy

Persistent observations in ≤ 1000+ buckets. Each one means we asked bitswap to fetch hundreds-to-thousands of blobs from this peer. If the putBlock disposition → exists share is also high, RBSR is undercounting our local set: it tells every peer we're missing blobs we actually have, peers ship them, the blockstore drops them, and the next sync repeats. Combined with the 2-minute TimeoutPerPeer hard cap, large wantlists trigger the sync — late-cancellation waste path.

When OTHER peers call our ReconcileBlobs RPC, where does our time go? Same Go code runs on gateways, so this is a structural proxy for what they spend per request.

auth_resolve

Check which spaces the calling peer is authorized for. May trigger an HTTP fetch to a siteURL on cache miss (we just added stale-while-revalidate + persistent cache to mitigate this).

load_store

Build the per-filter RBSR set from our local blobs. Recursive CTEs over structural_blobs + blob_links. Scales with corpus size — a heavy gateway will spend a lot here.

rbsr_session

Allocate the RBSR session struct. Always trivial.

rbsr_reconcile

The actual set-reconciliation algorithm: fingerprint the set, compute the diff. O(n) over store size.

Whole-handler wall-clock for inbound ReconcileBlobs requests. Compare directly against the client-side reconcile_rpc row higher up. If client p99 ≫ server p99, the gap is in the network/stream layer between us, not on either CPU.

Backpressure in front of inbound ReconcileBlobs. Default limit is max(2, 2*GOMAXPROCS); callers wait up to 3s for a slot, then receive ResourceExhausted.

in_flight

Requests currently inside the expensive SQLite/RBSR handler. Red when it reaches the limit.

waiting

Requests queued for a slot right now. Any non-zero value means the limiter is saturated at this instant.

rejected_total

Requests that waited too long and were rejected. This should stay near zero; growth means the server is overloaded for its CPU.

queue_wait

How long accepted and rejected requests waited for capacity. A rising p90/p99 means saturation before outright rejects show up.

How many blobs the RBSR set ends up holding for one inbound request. Bigger store → more compute everywhere. A mean around 1-10 means most filters are tight; spikes into 1000+ mean a recursive filter pulled in a heavy account.

Per-round timing of our outbound ReconcileBlobs calls, split by whether the gRPC conn map already had a live connection to that peer at the start of the round.

new_conn

First RPC to this peer in this process — we paid the libp2p stream open + gRPC HTTP/2 setup cost before the request.

reused_conn

An existing gRPC ClientConn to this peer was already in hmnet.Client.conns, so this round skipped the dial and went straight to the request.

If new_conn p99 is dramatically higher than reused_conn p99, dial cost is the bottleneck. If they're comparable, time is being spent on the gateway side, not the wire.

Cumulative counter for how each sync-with-peer call ended.

ok

Success.

protocol_mismatch

Peer doesn't speak our Hypermedia protocol version. Expected on the public swarm; fast-fails at dial.

dial_failed

Couldn't establish a gRPC connection.

rpc_error

ReconcileBlobs returned an error.

preempted

Context was canceled mid-flight (e.g. scheduler killed the task).

putmany_failed

Couldn't write downloaded blobs to SQLite.

Highlighted red if any non-ok row is >5% of total. protocol_mismatch volume is fine; the others should be near zero.

Where the daemon's bytes are going since startup. Three layers are tracked independently — they don't double-count each other:

libp2p

Raw stream bytes on the libp2p transport (bitswap, kad-dht, hypermedia gRPC-over-libp2p, identify, ping, autonat, holepunch, relay). The big libp2p protocols are broken out below.

http server

Bytes through the daemon's HTTP listener — gRPC-Web from the local desktop/web frontend, the public file gateway (/ipfs/<cid>), debug pages. The desktop app's gRPC-Web traffic shows up here as loopback since the frontend connects to 127.0.0.1.

http client

Outbound HTTP we own — primarily the delegated DHT client. Counted at our wrapper, so libp2p-internal HTTP probes (which we don't wrap) aren't included.

scope: loopback means the remote endpoint is 127.0.0.0/8 or ::1. remote means anything else, including LAN. So if your bandwidth feels too high but the loopback row is the dominant chunk, the bytes never left the machine.

bitswap duplicate-block waste: bitswap broadcasts WANT messages to every connected peer. When several peers race to send the same block, we accept all copies. Duplicate data received is the share of the bitswap recv stream that was wasted on blocks we already had. A high value (red >20%) means most of the bitswap inbound traffic is amplification, not new content. Multiplied across a few hundred connected peers this is usually the dominant source of "why is my bandwidth so high".

putBlock disposition: every block that reaches our blockstore (delivered by bitswap or pushed via PutMany) goes through one of three branches in blockStore.putBlock: new writes a fresh row, update fills in a previously-known placeholder, exists means we already had a complete row and the data is dropped. A large exists share (red >20%) means we paid full network cost to receive content already on disk — i.e. RBSR or bitswap thought we were missing blobs we actually have. Cross-check against SQLite size — this session below: if the DB barely grew but bitswap recv was big, the exists row is almost certainly where it went.

codec mismatch: when a peer ships a block, the bitswap-delivered CID carries a codec (raw, dag-pb, dag-cbor, …). When that block reaches putBlock and we already have the same multihash stored under a different codec, we hit the exists branch and drop the data. RBSR's set diff identifies items by full CID — codec + multihash — so two valid encodings of the same content (e.g. small UnixFS files stored as raw by some peers and as dag-pb by others) look like different items and get repeatedly fetched and dropped. Each row is a (stored→incoming) codec pair we observed, with the count of redundant deliveries. Empty table means peers and us agree on codecs; non-empty means we're paying real bandwidth for content we already have under a different CID.

syncing pre-flight Has filter: after RBSR computes a wantlist of CIDs the peer has and we don't, we run a multihash-keyed blockstore.Has on each before handing it to bitswap. CIDs we already have (codec mismatch or scope/orphan reachability — see the codec-mismatch table above) get filtered out at zero network cost, sparing us the WANT_HAVE → HAVE → WANT_BLOCK → BLOCK round-trip and the block delivery itself. The counter shows lifetime saved fetches; expect it to grow at the rate putBlock disposition: exists would have grown without the filter. A green non-zero value is direct inbound bandwidth saved; zero on a freshly-started daemon is normal.

blob index drift: count of rows in blobs with size >= 0 that have NO matching row in structural_blobs. RBSR builds its local-set view from structural_blobs via collectBlobs, so these blobs are present on disk but invisible to the diff. Each one will be re-fetched from peers every sync cycle and dropped at putBlock as exists. Red when the dag-cbor count is non-zero — those should be indexed and aren't.

persist pipeline — failures: post-streaming-refactor, fetched blocks are pipelined into PutMany batches and persisted on a detached ctx. Two failure modes can show up:

late-cancel discard

Legacy code path that lost work when the per-peer ctx fired during fetch. Should stay at 0 with the streaming refactor — non-zero here means a path slipped through and is still using the old all-or-nothing persist.

PutMany batch rollbacks

A streaming batch's PutMany tx rolled back, most often because indexBlob hit a cross-blob reference whose referent isn't in the DB yet (e.g. a Change ahead of its genesis_blob, a Capability before its parent). Those blocks come back from peers next sync and the order resolves. A small steady value during initial big syncs is expected; a sustained high value points at a real ordering / dependency bug worth chasing.

SQLite size — this session: page_count × page_size sampled at daemon start vs now. Compared against bitswap unique-recv (data received minus duplicates). If vs bitswap unique recv is much less than 1, then unique blob bytes were received from peers but didn't reach disk — either still buffered, dropped by indexer validation, or never persisted. If it's roughly 1×, the on-the-wire bytes really did become disk bytes. The ratio assumes blob content is incompressible-ish; if your storage path applies compression this ratio drops accordingly without indicating a leak.

All counters reset on daemon restart; numbers are cumulative since then.