System manual

The funnel at a glance

TEJ (leads)        SCOUT (triage)          MAX (build)             machines
───────────        ──────────────          ───────────             ────────
knows the city ──► reads the site ───────► writes/points the  ──►  ingest.py  ─► events
add_venue_lead     GO: planned + doc       adapter, runs the       (+ series      │
add_source_lead    NO-GO: none + why       LIVE GATE,              derive,        ▼
connect_coverage                           build_source            cost parse)  RATER ─► event_scores
                                                                                  │
        STEWARD re-probes the fleet, re-derives drifted flags ◄───────────────────┘
                                                                   export.py ─► data.json ─► this site

Three independent axes, never conflated: discovery (how we learned a thing exists — provenance columns), extraction (sources + the coverage bridge — can we hear it?), monitoring (do we want it? — and an event's visibility is always derived from these, never stored).

The tables

SQLite (chicago.db), SQLAlchemy 2.0 models in models.py. Every status flag carries an honesty grade: LIVE has a writer and is current · FROZEN set once at seed, never recomputed · DEAD no writer, don't trust · DERIVED recomputed from other rows. The fourth failure mode we learned the hard way: DRIFTED — has a writer that never re-runs. The Steward exists to hunt those.

pipeline_sources — one row per scrapeable endpoint

source_coverage — the M:N bridge (source ⇄ venue/group)

venues / groups — the where and the who

events — the payload

series — recurring clusters, first-class

event_scores — append-only judgment

One row per (event, scoring run); newest wins on export; history is never deleted (the rows are the expensive asset). Six 1–10 dims + a weighted composite. Series inherit: the scorer rates a series once (recurrence noted in the prompt) and writes that row to every unscored instance — a 14-week knitting circle costs one LLM call. Re-weighting (weights.json) never requires re-scoring.

The plumbing — operators are the only writes

The durable layer is mutated only through operators.py — idempotent, never drop, no raw SQL, no JSON seeds. The allowlist a role gets is its job description:

A lead's life (the Empty Bottle worked example — real, 2026-06-09)

1  TEJ    add_venue_lead("Empty Bottle", …)  +  add_source_lead(platform_guess="squarespace")
          └ guessed wrong — and that's fine, because:
2  SCOUT  reads the site: the Squarespace is an empty shell; the real feed is
          Ticketmaster Discovery v2 (96 events). Corrects the source row, writes
          docs/sources/empty-bottle-….md, verdict GO (status=planned).
          (Same pass: Rainbo Club → honest NO-GO, "15 events ever, dead since 2024".)
3  MAX    ports scrape_ticketmaster into scrapers.py, smoke-tests live,
          build_source(…) → gate passes (77 valid) → built/ok, venue verified+monitored
4  INGEST ingest.py --source empty-bottle-… → 65 Event rows (cost parsed: $16.17, free, …)
          → series derived → "RATER HANDOFF: N unscored" printed
5  RATER  scores the new events (announced first — it spends money); series inherit
6  EXPORT export.py → data.json → the map & list you're using

The roles — a staff, not a script

A role = a charter (.claude/agents/<name>.md) + a scoped operator allowlist. The scoping is the safety: an AI employee's worst failure mode is confidently fabricating state, and here the fabrications are structurally impossible — Tej can't fake a built pipeline, Scout can't build, built is earned from a live probe, not a claim.

The missing manager is deliberately thin: leads_inbox (a SQL view of every un-built row) is the work queue, and coverage-per-hood is the KPI. Dispatch is a query, not an agent — for now.

Keeping it true

• Test suites — tests.py in the engine (in-memory DB, stub adapters exercising the real gate) and in thirdplace. Zero deps, no network. They caught three real bugs on their first runs (a built-but-failing source still monitored its venue; "#3" never clustering; "doors at 8" parsing as $8).
• The Steward — probe drift + flag drift on a schedule.
• The recall harness (thirdplace/evalrecall.py) — the denominator for "almost all events": manually sweep one neighborhood for one weekend, freeze the list, measure what the pipeline caught and which channel the misses live on (expected answer: Instagram — the next frontier).

Repo pointers (the deeper layer)

• reality/README.md — orientation; docs/vision.md — the thesis
• event_wizard_chicago/CLAUDE.md — the model + run commands
• docs/roles.md + docs/leads-pipeline.md — role charters context + the operator contract & flag audit
• docs/sources/<id>.md — one page per source: platform, endpoint, gotchas, verdicts, build log
• docs/plans/2026-06-09-engine-v3-spec.md — the rebuild spec; docs/research/2026-06-09-e2e-pipeline-audit.md — the live audit that drove it
• On this site: raw tables · roles board · the friendly version