Case 01 / 04 · 2026
Portfolio Data Platform
Two pipelines over this site's own data: a private knowledge base compiled into a 45k-token attention budget behind six gates, and a Databricks medallion that turns the traffic this site produces into an answer about who actually read it — governed, versioned and replayable on both sides.
- Databricks · Lakeflow
- Delta Lake
- Unity Catalog
- Supabase · Postgres + RLS
- GitHub Actions
- Astro · Vercel
01 · The problem
This site has two data problems, and they point in opposite directions. It has to KNOW things — enough for an assistant to answer a recruiter properly, without ever touching the private work journal those facts come from. And it has to OBSERVE things — enough to tell someone who read for five minutes from a crawler that fetched one page and left, without keeping anything that identifies either of them.
Neither is really a chat problem or an analytics problem. Both are pipeline problems with a privacy constraint bolted to the front: a private source that must not leak, a transformation that has to be governed, an artifact somebody can review in a diff, and a gate that is allowed to say no.
So they are built as two pipelines over one system, and they meet in the middle. The assistant answers from a compiled corpus; every visit and every conversation lands in Postgres; a Databricks medallion refines that into gold; and the questions the assistant could not answer come back as the next page written into the corpus. The toggle below switches between the two halves.
02 · Architecture
Know
the compiled corpus
Public wiki pages are the corpus by default, not by allowlist. Six gates decide whether a build may ship at all, and what ships is one committed artifact plus a lockfile that makes every corpus change reviewable in a diff.
- 21 sources
- 32.3k est. tokens
- 45k budget
Answer
the assistant on this site
The artifact IS the system prompt, so every answer is grounded in reviewed text. Thirty golden cases run against the exact prompt the site serves decide whether a corpus change may merge.
Observe
Supabase + KV counters
One beacon per page load and one per tab-hide carrying visible dwell; for chat, metadata only. A salted hash of the IP rather than the IP, and the full user-agent discarded at the edge.
- events
- chat_sessions
- chat_messages
RLS: insert-only
Refine
the Databricks medallion
- Bronze Auto Loader over an immutable landing zone · explicit schema · drift caught in _rescued_data
- Silver Typed · production-only · AUTO CDC dedupe · channel resolved once · erasure anti-joined
- Gold Sessionized visits, a bot judgement, daily and per-page rollups — five materialized views
Decide
what gets written next
Two loops off one lake: the questions the assistant could not answer become the next source page, and what readers actually reach decides what is worth writing at all.
→ back to what the site knows
The private journal never enters CI, and chat message text never enters the lake. On both sides the sensitive half is absent by architecture rather than protected by trust — which is also what keeps the deletion promise on /privacy mechanically satisfiable.
One site · two pipelines
The outbound half: a medallion lakehouse over this site's own production traffic — Supabase and Upstash to gold, Auto Loader and Lakeflow, deployed as an asset bundle and run every morning.
- Databricks Lakeflow
- Auto Loader
- AUTO CDC · SCD-1
- Unity Catalog Volumes
- Asset Bundles
- PySpark
Source
Supabase + Upstash KV
Three INSERT-only Postgres tables, plus three daily counters that expire on a timer. Each table is read on a watermark — an identity id where there is one, a timestamp where there is not — and every run re-fetches a deliberate overlap, because a watermark is exact against mutation but not against a transaction that commits after the cursor has already passed it.
Land
a Unity Catalog Volume
One immutable NDJSON directory per run — staged, then moved, so Auto Loader never sees a half-written file. The lake is the system of record here, because Postgres is not a durable home for this history.
Bronze
Auto Loader · explicit schema
Declared types, not inference. A string arriving in an INT column lands in _rescued_data instead of silently casting to null, and rescue is per field — one bad value never costs the rest of the beacon.
Silver
typed, judged, deduplicated
Production-only, owner-tagged from a config table rather than a constant, paths parsed into content, dwell joined onto its view, and AUTO CDC (SCD-1) absorbing the extract's duplicates. Channel is resolved once here — a tagged link wins over the referrer, because a click out of a CV PDF arrives carrying Google's.
Gold
five materialized views
Recomputed in full, on purpose: sessionization, burst detection and "never seen again" are judgements about a row given its neighbours, and the neighbours have not all arrived yet.
- page_views
- visits
- conversations
- daily
- content
→ a morning digest
One expectation — _rescued_data IS NULL — ties two repositories together. If this site adds, renames or retypes a column, the pipeline goes red the same day rather than quietly drawing a dashboard with a hole in it.
Deploy
a Databricks Asset Bundle
The job, the pipeline, the catalog and the alert are all declared as code. Secrets are referenced by name from a Databricks scope — the repository is public and holds names only.
Extract
two tasks, in parallel
Postgres and the KV counters share no watermark, no secret and no source, so serialising them buys nothing. The pipeline waits for both to FINISH, not to succeed: an Upstash outage must not strand the events history, and an un-ingested file is not a lost file.
run_if: ALL_DONE
Refresh
bronze → silver → gold
Triggered, never continuous — the pipeline wakes, consumes what is new, and shuts the compute down. A full refresh takes seconds, which is what makes "fix the definition and replay" the honest answer to almost every schema question.
Report
a morning digest
Yesterday's numbers, emailed once gold has refreshed; failures alert separately. The expectation counts are published to a queryable table rather than left in a UI tab, because a check nobody can read is decoration.
02:00 UTC job · 02:30 digest
Roughly 40 production rows a day — wildly over-sized for the data, deliberately and in writing. A working medallion over telemetry you own is worth more than a right-sized script, and nothing on the live site depends on it.
Telemetry · in detail
-
Derive at source only what is destroyed at source
The raw IP and the user-agent exist only at the edge, so a crawler token and the webdriver flag have to be captured there or they are gone for good. Everything else — bot scoring, sessionization, channel resolution — belongs downstream, where a tuned heuristic can be replayed across all history instead of being stamped into rows that can never be backfilled.
-
The landing zone is the system of record
Raw NDJSON is kept immutably, so declaring a column later and running a full refresh reconstructs history rather than leaving it null before the change. That property is exactly why the format is NDJSON and not Parquet: Parquet fixes a schema at write time, which would push type inference into the one component that must never lose data.
-
At-least-once extract, idempotent silver
The watermark commits after the write, so a run that dies re-lands rows. Rather than chase exactly-once extraction, silver absorbs it: AUTO CDC keyed on the identity id and sequenced by the run that observed it, so a replay is deterministic right down to which file a row came from.
-
Gold is a judgement, and it is allowed to change
A visit is only classifiable given its neighbours — "never seen again" is decidable only in hindsight, and a dwell beacon lands in a later batch than its page view. So gold recomputes in full, a visitor who returns tomorrow reclassifies yesterday, and the bot score is the count of signals that fired rather than a probability, because there is no labelled data to fit one against.
The inbound half: a private knowledge base compiled into a token budget, gated six ways, committed as a versioned artifact, and re-checked by thirty cases in CI before it can ship.
- Node
- Supabase · Postgres + RLS
- GitHub Actions
- Artifact promotion
- Evals
Ingest
Super Brain wiki
Opt-out, not an allowlist: every public page IS the corpus. Pages under own-use/ are dropped at sync and dropped again at build — the same rule enforced twice, on purpose.
Compile
build-knowledge
- Access tier public → corpus · own-use → dropped twice · owner-only [[links]] deleted, not rewritten — a title is itself a leak
- Quality tier refusal rules and FAQ first, then digests, then pages — the model meets the load-bearing text first
- Derived facts experience, certs, projects and a content index compiled straight from the site data, so the bot cannot contradict the page
Gate
six gates · exit 1
A leak denylist, unresolved [VERIFY] markers, prompt-boundary smuggling, a broken site-facts import, a missing source, and the token ceiling. Any one of them fails the build outright.
Serve
a committed artifact
One generated prompt module plus a byte-stable lockfile that makes every corpus change reviewable in a diff.
- 21 sources
- 113,045 chars
- 32.3k est. tokens
- stale flag at 180 days
→ the assistant’s system prompt
The private source never enters CI — it is not in the repository at all. What ships is a reviewed artifact, and CI validates that artifact against 30 cases using the exact prompt the site serves. The boundary is enforced by architecture, not by trust.
Ask
a visitor question
Logging is on by default and can be switched off in the Talk panel. When it is off, nothing at all is written for that conversation.
Log
Supabase
A salted SHA-256 hash of the IP, a country code from the network edge, a coarse device bucket, and token usage. The raw IP and the full user-agent are never stored.
- chat_sessions
- chat_messages
- events
RLS: insert-only
Mine
the gap report
Each answer is paired with its question and deduplicated, then gaps are found two ways: a fast path on the assistant’s own “I don’t have that detail” phrasing, and a cheap classifier for the rest. Read-only — it never writes to the database.
Close
a new source page
The output is a topic list ranked by how often it was asked. Each topic becomes the next page I write, and the next build folds it in.
→ back to the build
A purge function is defined with a six-month default and is run on request; no scheduler runs it automatically — and /privacy says exactly that, rather than claiming an automatic purge that does not happen.
Knowledge · in detail
-
An opt-out corpus, not an allowlist
Curating an allowlist means the corpus silently rots as the knowledge base grows. So the default inverts: every public page is in, and privacy is a property of where a page lives. Owner-only pages are excluded at sync and again at build, and links pointing at them are deleted rather than rewritten — because a page title is itself a leak.
-
Six gates that fail the build
A leak denylist covering ticket prefixes and colleague names, unresolved [VERIFY] markers, prompt-boundary smuggling, a broken site-facts import, a missing source with no committed fallback, and the token ceiling. Each one exits non-zero. The cheapest place to catch a leak is before it is ever committed.
-
Thirty cases gate every corpus change
Hiring, technical, curious, adversarial and multilingual questions, run against the exact prompt the site serves — all-or-nothing, so one regression blocks the merge. The wiki-link leak found in July 2026 is kept as a permanent assertion: that specific bug can never return silently.
03 · How it works
-
A private source that never crosses its boundary
Neither pipeline is allowed to move the sensitive half. The work journal is not in the repository at all, so it cannot reach CI; message text is never replicated into the lake, so a deletion request stays mechanically satisfiable instead of chasing copies. In both cases the protection is that the data is absent — not that something is trusted to guard it.
-
The artifact is the contract
Build once where the source lives, then ship something reviewable. The corpus compiles locally into a generated prompt module and a byte-stable lockfile, so a change is a diff a human can read. The lake keeps immutable raw files, so a schema declared today can be replayed over everything landed before it. Both make a rebuild reproduce rather than approximate.
-
A gate that is allowed to say no
Six build gates and thirty golden cases stand between a corpus change and production. A single silver expectation on _rescued_data stands between a column changing on this site and a wrong dashboard the next morning. Each exits non-zero rather than warning, because the cheapest place to catch a leak or a drift is before anyone can read the result.
04 · Outcome
Both loops now run themselves. A new fact is one page in the wiki, and the next build either folds it in or refuses to ship — today that is 21 sources compiled to roughly 32k tokens against a 45k attention budget, with 30 of 30 cases passing. A visit is one beacon, and by the next morning it has been sessionized, classified and rolled up, with a digest in my inbox saying what yesterday actually was.
The honest framing matters as much as the architecture: neither half is load-bearing. This site does not need a lakehouse for forty rows a day, and the page says so rather than implying a scale it does not have. What I wanted was a complete, small instance of the discipline the rest of my work runs on — a private source, a governed transformation, a versioned artifact, a gate that can say no, and a feedback loop that decides what to build next — built twice, in opposite directions, over data I own and can therefore talk about freely.
Read the thinking behind it: I Built My Chatbot's Knowledge Like a Lakehouse Ask the assistant about this