Callout Fleet

LIST = unauthenticated /callout/list/{wallet} per caller, cache-key cycling over fresh tunnels (hubs box3/4/8/9/10 as disjoint shards, since 2026-10-17; arrivals arrive via Redis from the hubs). POLL = REST poll of /following-positions/alerts (1,950 accounts, 39 /64s). USER-WS = per-account alertsFeed.user websocket on pump’s NATS cluster (presence + JWT; 2 per box on box11-15 + box1p). GLOBAL-WS = account-less calloutCreated websocket on the same NATS cluster (static creds; box1p + the six poll-box hubs, forwarded to the executor over the peer lane). Raw race-log labels: list / alerts / push / nats. The old KPI ignored GLOBAL-WS and read ~80 ms too high after pump moved its per-user paths behind the global publish (2026-10-15).

Load = Σ planned Hz of the worker’s callers; Util = load ÷ capacity (amber > 85 %, red > 100 %). Cached-like = share of responses that looked like a pump cache hit (a key reused inside its 30 s TTL). Red row = dead worker (no heartbeat for 30 s) or a status hash without any heartbeat. Hover a row for key period / effective Hz / loop lag.

The ~3,000 pump accounts behind POLL / USER-WS still follow these callers (since 2026-10-18 16:01Z they no longer get a 🏁 Telegram race card — cards are SlotShark’s callers only, executor knob CALLOUT_RACE_TG_SETS) (stale follows from older watch lists / revived accounts), so their callouts are detected and raced — but they are in none of our sets: never bought, never relayed to friends, and LIST does not poll them (that is why their cards show no LIST line). Add one to priority to have LIST poll it at 13 Hz, or to a SlotShark config / friend list to make it a real caller. Source: the executor race log (callout-races.jsonl), read incrementally every 60 s; addresses resolved from pump.fun names.

Source of truth = sniper:list:required (the LIST workers republish it every 60 s from the executor’s watched configs and the friend / admin / notify follow sets). The panel keeps a roster of it: First seen is when the panel first saw the wallet (callers present when the roster started on 2026-10-18 show as baseline); a caller counts as removed after 5 min absent, and one that returns keeps its first-seen. Names come from pump.fun’s public profile (looked up once, refreshed daily; “…” = not looked up yet). Click an address to copy it.

Each priority caller is read on every one of pump’s 400 cache keys (limit 1-100 × 4 windows) on a time-pinned grid, so each read is a fresh store read: 13.1 reads/s, mean sampling lag ≈ 38 ms vs ≈ 130 ms for a normal caller. Its grid is split over the 2 best eligible workers (each polls every other slot): a worker restart or a 403 pause halves the rate for a few seconds instead of stopping it, and a paused worker hands its callers over at once. Its budget is paid first on each worker — the other tiers share the rest. Health: OK = on the grid on every worker of its set at ≥ 12 Hz; amber = on fewer workers / lower rate / waiting for the fleet (a worker without the priority build keeps it as a normal top-tier caller); red = not polled. Cached-like = share of grid reads with a cache-hit-like TTFB (healthy ≈ 5-10 %; a real stale read problem shows ≥ 50 %). Cost ≈ 13 req/s per caller; the list is capped at 40.

Covered = a LIVE worker owns the caller and had a successful poll within 60 s. Uncovered required callers are always listed (red) with the reason: no owner (no worker has it), owner dead (only dead workers have it), never polled, no successful poll > 60 s. Tier: P0 = hand-picked priority (above), T1 = watched, T2 = planned (Hz from the planner), T3 = rest; extra = polled but not required. ×N = owned by N workers (duplicates waste budget; a P0 caller’s two grid workers are by design).

Shared races = callouts BOTH lanes saw. Lead = other lane’s first arrival − LIST’s (green = LIST ahead). Hub→executor hop = the executor’s stamp minus the hub’s own stamp for the same LIST detection (Redis peer lane + executor ingest; the hubs publish their lines to sniper:callout:race-lines).

Shadow = the on-chain scheduler observes and scores but does not steer the fleet. States: JUST_BOUGHT (eligible position younger than 60 s), PENDING (eligible, uncalled position), IDLE (none), UNCERTAIN (no history yet → baseline polling), DEGRADED (stream down → everyone at baseline). Eligible = holds the coin (≥ ~$1, pump’s rule) and has no live callout on it. Policies: A continuous (today), B strict event-driven, C hybrid (idle rate), D adaptive (rate by position age). “Modeled” lags are 500 ÷ rate for requests that were never sent — not measurements; the real LIST lag of the same callouts is in the summary.

Fleet cadence = one poll every (interval ÷ total pollers). Status: green healthy · amber degraded (pauses / few push) · red hold, poll dead, push down, or stale heartbeat.

USER-WS = connected / subscribed per-account alertsFeed websockets (GLOBAL-WS sockets live on the hubs and box1p and are not counted here). Type: poll (all shards poll) · poll+push (mixed) · push-only (dedicated push box). Amber = some websockets not connected (often >~100/·64 presence limit); red = stale heartbeat.