SE Intel — Build Log

The Problem

The rate limiter from Week 1 used Workers KV as a counter: read the current count, add 1, write it back. Under concurrent traffic, this is a textbook lost update: 20 requests all read 0, all compute 0 + 1 = 1, all write 1. The counter ends at 1, not 20. KV has no transactions, no compare-and-swap, no atomic increment.

The bug was invisible for 4 weeks because single-user testing at 1 request/second never triggers a read-modify-write race. Under deliberate concurrency testing (25 simultaneous requests), the counter only reached ~10.

The Fix

Moved the counter into a Durable Object (RateLimiterDO). One DO instance per user, keyed by userId. Because all checks for the same user route to the same DO instance, the read/compare/write sequence is serialized for that counter. No locks, no CAS, no distributed coordination for this access pattern.

Proof: 20 concurrent requests for a role with a 20/min limit — all returned 200. Request 21 returned 429. With the old KV implementation, 20 concurrent requests left the counter at ~10.

Property	KV	Durable Object
Consistency	Eventual	Strong (serialized)
Atomic increment	No	Yes
Global latency	Low-latency edge reads	One extra DO hop
Best for	Read-heavy caches	Counters, coordination

Interview Answer

"I found a concurrency bug in our KV-based rate limiter — KV doesn't support atomic read-modify-write, so under 25 concurrent requests the counter only reached ~10 instead of 25. Classic lost update problem. I moved the counter into a Durable Object, which serializes all requests to a given instance — the actor model eliminates the race without adding locks. Verified live: 20 concurrent requests, all 200; request 21, 429."

The Problem

Before this work, a Workers AI failure produced a 500 or a broken SSE stream. No fallback, no degraded mode, no informative error. Impossible to demo resilience without waiting for a real outage.

The Fix — 3-Tier Fallback Chain

Tier	Model	Latency	Metric Status
1 (Primary)	Llama 3.3 70B	4-15s	success
2 (Fallback)	Llama 3.1 8B	0.7-3.5s	fallback
3 (Canned)	None (static)	~0.2s	error

The X-Debug-Fallback: true header skips Tier 1 and goes straight to Tier 2. This lets you demo degraded mode on demand, test the fallback path in CI, and verify the canned response works — all without a real outage.

Both chat and streaming paths have the full chain. The streaming path emits a {type:"model", reason:"forced_fallback"} SSE event before the first token so the client knows which model is responding.

Bug Found During Testing

The initial fallback model name (@cf/meta/llama-3.1-8b-instruct) doesn't exist in the Workers AI catalog — the correct name is @cf/meta/llama-3.1-8b-instruct-fp8. When the wrong name caused an error, the chain correctly fell all the way to Tier 3 (canned response) — accidentally proving the full degradation path works end-to-end.

Interview Answer

"I built a three-tier fallback chain: primary 70B model, fallback 8B model, and a canned degraded response if both fail. There's a debug toggle — an HTTP header — that lets me skip the primary and demo degraded mode on demand. The metrics distinguish between 'worked normally,' 'worked via fallback,' and 'both failed,' so the health scorecard can measure provider reliability over time."

What We Built

Three pieces: (1) a metric written on every request via waitUntil() — latency, org, agent type, success/failure; (2) a health calculator (getOrgHealth()) that turns raw metrics into p50/p95 latency, error rate, and SLO pass/fail; (3) a deterministic probe (health-probe.sh, 4 checks) that proves the pipeline works without an LLM.

The audit_log and request_metrics tables are intentionally separate: one carries PII (message previews, user roles), one doesn't. Different access patterns, different retention governance — same principle as the short-term/long-term memory split.

The SLO Recalibration Story

The initial p95 target was 8000ms — an aspirational guess. Real production data from a 70B model with RAG (embedding query + vector search + generation) showed p95 at 11-13s. Leaving the target at 8s would mean a permanently red dashboard, which teaches everyone to ignore it. Recalibrated to 15000ms with the before/after numbers documented and a concrete improvement roadmap (KB answer caching, smaller-model routing for simple queries, reducing max_tokens).

Interview Answer

"Every agent request writes a structured metric — latency, org, status — via waitUntil so it never blocks the response. The health scorecard computes p50/p95 latency and error rate per org against documented SLOs. When the initial target was wrong, I recalibrated it against real data with the numbers documented, rather than shipping a dashboard that's always red."

What We Built

A Python eval harness with two layers: (1) an LLM-as-judge scorer (5 dimensions, 15-point scale) for subjective quality; (2) a deterministic faithfulness.py check that compares retrieved KB chunks against the response text — no LLM in the loop. The deterministic check caught a real bug the LLM judge missed: the KB chunk said "35% discount" but the agent answered "25%."

The harness runs as run_eval.sh --ci — exits 1 on any grounding failure, exits 0 clean. 15 test cases across both agents and all five roles. Score progression: 67% → 73% → 87% over three iterations.

Interview Answer

"I built a two-layer eval: an LLM judge for subjective quality, and a deterministic faithfulness check that compares retrieved chunks against the response. The deterministic check caught a real hallucination the LLM judge missed — the KB said 35%, the agent said 25%. The eval runs as a CI gate that blocks deploys on quality regression."

What We Built

Threaded orgId isolation across three layers: (1) RAG — Vectorize metadata filter at query time, "global" sentinel for shared docs; (2) Memory — DO keys scoped to orgId:userId (7 sites), KV keys to ltm:{orgId}:{userId}:{factId} (5 sites); (3) Audit — org-scoped reads, role-split (manager sees org, individual sees own).

Three deterministic admin probes (/admin/kb-probe, /admin/memory-probe, /admin/audit-probe) prove zero cross-org leakage — no LLM in the test path. End-to-end test: isolation-test.sh 3/3 pass.

Additional Chapters

Chapter 1 — The Problem. Five hard problems in enterprise AI assistants: role-gated access, conversation memory, retrieval quality, evaluation, and multi-tenancy.

Chapter 2 — Workers and Durable Objects. V8 isolates, single-threaded DO execution, embedded SQLite. Why DOs eliminate race conditions on conversation state.

Chapter 3 — Auth and Zero Trust. JWT verification via WebCrypto, defense-in-depth (three enforcement layers for RBAC).

Chapter 4 — Rate Limiting. Sliding window counter design, per-role limits, TTL-based cleanup. (Later replaced by DO in Week 4.)

Chapter 5 — RAG Pipeline. BGE-base embeddings, Vectorize cosine similarity, namespace filtering, parallel multi-namespace queries, eager tool calling.

Interview Answer

"Every user gets their own Durable Object — a physically separate SQLite database, not a filtered view of a shared table. At the org level, Vectorize queries are filtered by orgId metadata at query time. Three deterministic probes prove zero cross-org leakage across RAG, memory, and audit — no LLM in the test path."

Chapter 6 — Memory (Short-Term + Long-Term)

Two-tier memory: short-term (DO SQLite, last 20 turns per thread) and long-term (KV, extracted facts injected into every future system prompt). The critical lesson: state.waitUntil() is mandatory for fire-and-forget async work inside a DO — without it, the isolate terminates before LTM extraction completes, producing zero results with no error thrown.

Chapter 7 — Streaming SSE

AI.run({ stream: true }) returns a ReadableStream. The DO parses Workers AI's internal SSE format and re-emits three event types: tools (before first token, shows what was retrieved), token (one per token), done (latency + metadata). The Worker passes the DO's stream through without buffering the full model response, which is the design choice that keeps first-token latency low.

Chapter 9 — MCP (Model Context Protocol). Built an MCP server for SE Intel (research_account, get_enablement, get_memory tools). Stdio transport, thin bridge pattern — the MCP server is a protocol adapter, not a second copy of the business logic.

Chapter 10 — Failure Modes. Component failure map: what happens when each dependency (Workers AI, Vectorize, D1, KV) goes down. Trade-offs table for each failure mode — which ones are acceptable (metrics write fails silently) and which aren't (auth bypass).

Chapter 11 — Behavioral Prep. 60-second pitch, 5-minute deep dive, screening question answers. Each tied to specific chapters and deployed code, not abstract claims.