CurrencyTransfer call copilot on Cloudflare + Jev: architecture
Status: final synthesis, 2026-09-24. Feeds the otto-plan PRD. Built from the three proposals in docs/architecture/ (poc-fastest as the spine; production-shaped's guarantees and data-first's engine grafted on) and the three judges' verdicts. Citations are absolute paths under /home/stevan/dev/jev/; guide §n = docs/jev-guide.md, dissection §n = docs/research/reference-copilot-dissection.md, brief §n = docs/research/ct-domain-brief.md, corpus §n = docs/research/corpus-and-curation.md, stt §n = docs/research/stt-options.md, prior-art §n = docs/research/prior-art.md. Anything not cited is marked ASSUMPTION; numbers marked est. are estimates, not measurements.
1. Summary
- Milestone 1 is a replay copilot: pick a historical Aircall call or upload a recording, press play, and watch a dashboard update utterance by utterance with CT signals, stage, must-say checklist, risk flags, open concern and a next-best-move card drawn from pre-approved lines.
- One TypeScript engine,
step(state, utterance, answers, policy), pure and side-effect-free, runs in a Durable Object per replay/live session in M1 and in a Queue consumer for corpus scoring in M2. Same state, same questions, same gates in both. - Jev only judges: one speculative fan-out request per decision point (
env.AI.run('typesafe/jev')), ~9.6k input tokensest.at the measured ~3.5 chars/token (§6.2), soft budget 11k with a drop order, hard cap 12k against Cloudflare's 32k limit. All arithmetic, thresholds, memory and hero metrics are code. - Historical calls are ingested from the call-coach exports with no STT:
role internal|externalalready gives rep/client.@cf/deepgram/nova-3is used only for new uploads; Aircall recordings are mono, so diarisation plus a code/Jev/admin speaker map is the path (stt"Live verification"). - Policy (question banks, playbooks, weights, thresholds, checklist, risk rules) is versioned, immutable JSON with three separate hashes. Raw Jev answers are stored keyed by
bank_hash+state_hash; move/phrasing answers separately byplaybook_hash; derived decisions bypolicy_hash. Re-weighting is free; a playbook line edit never invalidates judging answers. - Hero metrics per Stevan's decision: Onboarding = must-say checklist completeness (rep-turn locks only) + latched risk flags; Customer Success = resolution state machine (reported → owned → client-accepted) + a secondary call-health composite. No closing probability anywhere.
- Rep-facing text is a typed union: an approved playbook line addressed by
text_id+ policy version, or a tailored rewrite that passed six single-condition Jev verifier nouls and a code denylist, delivered off the critical path on a droppable message. The policy loader hard-rejects any bundle containing[VERIFY]. - Curation starts in M1: the engine emits Jev-tagged moments (must-say locks, risk flags, concern episodes, uncertain-band decisions) and the replay page has a one-form model/acceptable/avoid mark. M2 adds corpus-wide scoring, the review queue ordered by information value, promotion to examples and playbook lines, and an enforced publish gate.
- Compliance: pseudonymise in code before D1, Jev or the browser (
[REP],[CLIENT],[IBAN], ...); LLM coach text never enters D1; dedicated AI Gateway with body logging off; Cloudflare Access from day one on the Worker and the docs site (email policy for Stevan, a service token for the scripts); audit log with the Access identity. Client-derived text goes totypesafe/jevonly for the four exported samples until Stevan answers the DPA question (§13). - Cost
est.$0.022 per replayed 7-minute call in Jev credits, ~$23 per 1,000 historical calls, ~$70-75 for the whole usable corpus; nova-3 adds ~$0.04 per uploaded 7-minute call. Latency per decision 0.44 s over REST measured today, ~0.8 s via the binding with cold start; target p95 utterance-to-card ≤ 1.5 s.
2. Design stance
The three proposals scored within three points of each other; the judges converged on the same synthesis: poc-fastest's M1 spine (replay first, browser-owned clock, minimal machinery, three non-negotiables), production-shaped's enforced guarantees (hash split, budget drop order, typed rep-facing text, [VERIFY]-rejecting loader, gated publish, reconnect history, audit log) and data-first's engine and curation shape (pure step(), rep-turn-only must-say, decomposed verifiers, moment kinds, information-value review ordering, noul-first hero metrics, calibration-derived thresholds).
What this document optimises for, in order:
- Stevan's milestone 1 as written: upload or pick a call, transcribe if needed, replay it in sync with a dashboard. Everything that is not on that path is a named slot in M2 or M3, never a prerequisite.
- No corner-painting. Four things are non-negotiable even in the PoC because retrofitting them is expensive: (a) every raw Jev answer is stored with its hashes, model id and the exact pseudonymised state; (b) banks, playbooks, weights and thresholds are versioned data, not code constants; (c) transcripts are pseudonymised before they reach D1, Jev or the browser; (d) the engine is a pure function so batch scoring is configuration, not a rewrite.
- Compliance by construction, not convention. Where the judges found a badge, a prose
not_for, or a "will not block on verifier error", this design uses a type, a code rule, or a hard drop. - Jev hygiene from
guide §7: literal single-condition nouls with alignedtrue/falsecriteria; no counting, arithmetic or dates asked of Jev; small filtered state; anoneoption on every Choice paired with an absolute noul; thresholds never carried between question types;response.modellogged on every answer because Cloudflare does not pin versions (guide §8).
What it trades away in M1: no admin console beyond the mark-moment form, no corpus-wide scoring, no CT-DB pre-call brief, no Vectorize, no live audio, no weights editor in the UI, rewrite path built but off. One piece of M2-serving infrastructure is paid for early on purpose: the answers/move_answers hash split (§5, D3) exists so that an M2 playbook-line edit does not invalidate judging answers; M1 has no playbook editing, so nothing in M1 exercises the split, and reviewers should read it as a retrofit-cost decision, not as something M1 validates.
3. System overview
flowchart LR
subgraph devbox
EXP[("call-coach exports<br/>call_records / call_turns .jsonl.gz")]
IMP["scripts/import-call-coach.mjs<br/>select · stitch v1 · redact v1"]
EVAL["scripts/eval.mjs<br/>L1 L4 L5 L6 → eval/reports"]
EXP --> IMP
end
UP["Browser upload<br/>mp3 / m4a / wav"]
subgraph CF["Cloudflare account 694e…9ce9 — Cloudflare Access on every hostname"]
W["Worker jev-copilot<br/>HTTP /api/* · WS upgrade · static dashboard"]
DO[("Durable Object CallSession<br/>step() per utterance · WS · seek cache")]
D1[("D1 copilot<br/>calls · utterances · answers · move_answers<br/>decisions · policy_versions · moments · marks<br/>labels · eval_runs · audit_log")]
R2[("R2 copilot-raw<br/>raw/ audio · aircall json · nova-3 json<br/>eval/ runs · golden/")]
AI[["Workers AI<br/>typesafe/jev · @cf/deepgram/nova-3"]]
GW["AI Gateway jev-copilot<br/>body logging off · spend limit"]
PG["Pages ct-copilot-docs<br/>architecture · demo script · eval reports"]
Q[["Queue score-calls<br/>M2: batch scorer = same step()"]]
end
IMP -->|"POST /api/calls/import"| W
UP -->|"PUT /api/calls/upload"| W
W -->|"audio → REST"| AI
W --> D1
W --> R2
W -.->|M2| Q
Q -.-> DO
DB["Browser dashboard<br/>owns the clock: audio element or virtual timer"] <-->|"WebSocket /ws/calls/:id"| DO
DB -->|"GET /api/calls/:id, /audio"| W
DO -->|"one request per decision point"| GW
GW --> AI
DO -->|"answers · decisions · moments"| D1
DO -->|"read policy + call once"| D1
EVAL -->|"POST /api/evaluate"| W
EVAL --> PG
Data flow for one replayed utterance: the browser passes utterances[i].t_end and sends {type:'utterance', i}; the DO enqueues it on its serial chain (§4), builds the filtered state, and looks up (call_id, i, bank_hash, state_hash) in answers and (call_id, i, playbook_hash, state_hash) in move_answers. Cache lookup is per hash. Both hit: no Jev call. Both miss: one fan-out request. Judging hit + move miss (the normal case after an M2 playbook edit): a smaller moves-only request carrying next_move plus the code-allowed phrasing:: Choices against the same state_json (~2k tokens est.), stored in move_answers. Move hit + judging miss is treated as a full miss, because allowedMoves() depends on the judging answers. Then: apply the token budget, store the raw answers, run step(), persist SessionState, the decision and any moments, and push one decision message. A rewrite, if enabled, is enqueued and arrives 1-3 s later as a separate rewrite message or not at all.
4. Components on Cloudflare
Worker jev-copilot. Stateless front door, TypeScript, plain fetch router (Hono optional). Responsibility: Cloudflare Access JWT check (Cf-Access-Jwt-Assertion validated against the team's certs and the application AUD; the identity, email for a person or common_name for a service token, becomes audit_log.actor and sessions.actor), POST /api/calls/import (canonical rows from the import script), PUT /api/calls/upload (raw binary body streamed to R2 → nova-3 → stitch → redact → D1, §7), GET /api/calls, GET /api/calls/:id (pseudonymised utterances from D1), GET /api/calls/:id/audio (from R2 with HTTP Range support: parse Range, call env.R2.get(key, { range }), answer 206 with Content-Range, Accept-Ranges: bytes, Content-Length and the stored httpMetadata.contentType, 416 on an unsatisfiable range, HEAD supported; a plain new Response(obj.body) is a 200 without ranges, which means no seeking in Chrome/Firefox and no playback at all in Safari), GET /ws/calls/:id (WebSocket upgrade forwarded to the DO), POST /api/evaluate (the existing jev-lab shape in src/index.ts, kept for the eval script), POST /api/moments, GET /api/policy/:scenario, and the dashboard as Worker static assets (assets binding). Why: the only compute primitive with the AI binding; one origin for HTTP, WebSocket and static files means one Access application and no cross-site cookie work in M1 (§15, D9). That Access application has two policies: Allow (email = Stevan) and Service Auth (service token jev-scripts), because scripts/import-call-coach.mjs and scripts/eval.mjs run from devbox as non-browser clients and an email-only policy answers them with the login page; the scripts send CF-Access-Client-Id / CF-Access-Client-Secret, and the Worker accepts the resulting service-token JWT (common_name, no email). The token pair lives in ~/.config/jev/cloudflare.env next to the API token, never in the repo. Authorisation in M1 is the Access gate alone: every Access identity can read the whole corpus, which is acceptable while the only identity is Stevan's. Before Access is extended to RMs or managers (§13 Q5), GET /api/calls, GET /api/calls/:id and GET /ws/calls/:id must filter on calls.rep_ref (already in the schema) against the identity, or Stevan records full-corpus read as the tenancy model; the route list has no per-call authorisation today and this document says so rather than implying one. Limits that matter: 100 MB request body bounds uploads (stt §3.3); 128 MB memory per isolate, which is why uploads stream and are never buffered (§7); 30 s CPU default per invocation, so nothing loops over Jev calls in the Worker; wrangler dev bills real AI usage (guide §4.2), so unit tests stub env.AI.
Durable Object CallSession (SQLite-backed). One per (call_id, session_nonce), so two tabs replaying the same call do not share state. Holds the rolling window, persisted facts and checklist, concern state, resolution machine, EMA'd health, stage hysteresis state, card state, the pinned policy version, and the WebSocket. Serialisation is explicit, not assumed. A Durable Object's input gate closes only while a storage operation is pending; while the object awaits env.AI.run or a fetch (0.4-0.8 s) further webSocketMessage events are delivered and their handlers interleave. At 5-20× the browser sends an utterance every 0.1-0.6 s, so without a queue step i+1 would build its state and state_hash before step i's answers and persisted facts existed, corrupting the feedback chain, the seek cache and the "catching up" count. The DO therefore keeps an in-object promise chain (this.tail = this.tail.then(() => this.process(msg))) and a pending counter: utterance, seek and set_weights are enqueued on the chain; ping, ack_card, dismiss_card, mark_moment and flag_false_positive bypass it; pending drives the queue message; seek clears the chain before it runs; nothing else in the DO awaits Jev outside the chain. (The reference is serial only because one asyncio loop awaits handle_message per received frame, server.py:236-243; poc-fastest's "processed serially" was not a DO guarantee.) Hibernation loses memory, so state is stored per step. With the WebSocket Hibernation API the object is evicted from memory whenever it is idle for more than a few seconds while the socket stays open, and the constructor runs again on the next message; none of the in-memory fields above survive that. SessionState (≤ ~50 KB) and the session's decision list are written to ctx.storage (SQLite) at the end of every step(), in the same turn as the D1 write is issued, and the constructor reloads them inside blockConcurrencyWhile; the session nonce, call_id and pinned policy version are attached to the socket with serializeAttachment so a hibernated socket resumes without a new load. Because the browser owns the clock there is no server-side timer chain and no alarms, so nothing conflicts with hibernation and it costs nothing (the judges flagged this conflict in production-shaped and data-first). On reconnect it replays the decision timeline from the same storage so a tab reload never loses the session (dissection §7a fixed). Limits: 10 GB SQLite per object (we keep < 5 MB), soft 1,000 req/s per object, 32 MiB received WS messages (corpus §5); irrelevant at one decision every 2-6 s.
D1 copilot. Pseudonymised system of record for everything queryable across calls (§5). Why: cross-call SQL for the M2 review queue, calibration, rankings and label joins. Limits: 10 GB per database, 2 MB per row (answer rows ~3-4 KB), 100 KB per statement, 100 bound parameters per statement, which means at most floor(100 / columns) rows per multi-row insert: 9 rows for the 11-column utterances and answers tables (all three proposals copied "12 rows" from an 8-column example; a helper chunkForD1(rows, cols) enforces the real limit). Use batch() for the rest.
R2 copilot-raw. Private bucket, write-once blobs: raw/{call_id}/aircall.json (un-redacted normalised transcript), raw/{call_id}/coach.json (the LLM coach output including summary and quotes[].reason, which name clients), raw/audio/{call_id}.{ext}, raw/stt/{call_id}.json (nova-3 output), eval/{run_id}/…, golden/{policy_version}.json. Lifecycle rule on raw/ with a retention Stevan sets (§13). Why: PII blobs do not belong in a query store; unbounded size; cheap.
KV. Not used in M1. Policy versions are D1 rows; the DO reads the published row once per session and pins it. M2 adds policy:{scenario}:v{n} (immutable) and policy:{scenario}:current (pointer) as a read-hot mirror when the batch scorer reads the policy per invocation; KV's up-to-60 s eventual consistency is harmless for version-addressed keys (corpus §5).
Queues. Not used in M1. M2 adds score-calls (one message per call, consumer runs the same step() sequentially, idempotent on (call_id, i, bank_hash, state_hash), concurrency ≤ 8 until the Workers AI rate limit for Jev is measured, dead-letter after three failures) and rewrite-jobs (keeps the generator off the DO's critical path). The judges flagged the unmeasured Queue-consumer CPU assumption (~70 sequential fetches per invocation); the fallback is one message per 20 utterances carrying the engine state, and M1's cost/latency report measures the numbers that decide it.
Pages ct-copilot-docs. Hosts the deliverables per Stevan's decision: this document, the demo script with utterance numbers, eval reports, the cost/latency report. The project sits behind its own Cloudflare Access application (same email policy) from the first deploy, confirmed before deliverable 9 publishes anything: the demo script and eval reports name calls and rep tokens, and a Pages project is public by default. Nothing published here carries hosts, paths, keys or credentials; operational access details live in an ops runbook kept outside the Pages build. The dashboard itself is served by the Worker in M1 (one origin for WebSocket + Access); the M2 admin console moves the SPA to Pages under a custom domain with a Worker route on /api/* and /ws/* once a CT hostname is chosen (§13 Q6). Pages Functions are not used: the AI binding cannot be declared in wrangler config for Pages (guide §4.2).
Workers AI models. typesafe/jev via the binding with { gateway: { id: 'jev-copilot' } } as the third argument (guide §4.3); verified REST 400-550 ms and 0.44 s again today, binding ~800 ms including cold start; REST evaluation sits at result.result, so the client unwraps defensively (guide §4.7). If the binding turns out not to honour gateway selection or the per-request cf-aig-* controls (binding reference gap, guide §4.3), the DO switches to REST with fetch and an account token provisioned with wrangler secret put CF_AI_TOKEN (never in wrangler.jsonc, .dev.vars committed to git, or source), which scripts/jev.mjs already does from ~/.config/jev/cloudflare.env; tested on day 1. @cf/deepgram/nova-3 over REST with binary body and ?diarize=true&utterances=true&punctuate=true&smart_format=true&language=en-GB&numerals=true (verified: results.utterances[] with speaker/start/end/confidence, 8.6 s for a 9:46 mono Aircall call, 94% speaker agreement with Aircall labels, ~473 neurons/min ≈ $0.31/h; stt "Live verification"); add keyterm (CurrencyTransfer, GBP, EUR, SWIFT, IBAN, mid-market, forward, drawdown, safeguarding, beneficiary) and mip_opt_out=true (effect on the Cloudflare proxy unverified). Whisper is not used: no speakers (stt §3.1). Rewrite generator: ASSUMPTION @cf/meta/llama-3.3-70b-instruct-fp8-fast as the CF-native default, a policy field, swappable for a Haiku-class model through AI Gateway; behind a flag, off in M1.
AI Gateway jev-copilot. Dedicated gateway, never default: request/response body logging off (the default gateway stores Jev bodies regardless of ZDR, guide §4.3, §4.11), caching off (identical states are rare and a cached answer hides version drift), per-gateway spend limit set before any batch run, cf-aig-metadata {call_id, policy_version, mode} on REST calls. Billing: Unified Billing prepaid credits, 5% purchase fee, $0.042/M input pass-through, output free (guide §4.6, §4.10; keySource: Unified confirmed today). HTTP 402 code 2021 = empty credits: surfaced as an operator alert on the dashboard and in the Worker log, never retried (guide §4.8).
5. Data model
Principle: raw judgments, move/phrasing picks and derived decisions are three tables keyed by three different hashes, so each kind of policy edit invalidates only what it must.
-- corpus (pseudonymised text only; nothing from the LLM coach's prose)
CREATE TABLE calls(call_id TEXT PRIMARY KEY, source TEXT CHECK(source IN ('call-coach','upload')),
scenario TEXT, scenario_source TEXT CHECK(scenario_source IN ('call-coach','jev','admin')), scenario_conf REAL,
rep_ref TEXT, direction TEXT, recorded_at TEXT, duration_s REAL, engine TEXT,
stitch_version INTEGER, redact_version INTEGER, n_utterances INTEGER, n_decision_points INTEGER,
role_map_json TEXT, role_map_conf REAL, audio_r2_key TEXT, raw_r2_key TEXT,
coach_scores_json TEXT, -- numeric only: six 0-5 dims, call_value_score, quotes[].turn_idx (no reason text)
pd_ct_id TEXT, -- opaque join key for the M2 outcome join; ASSUMPTION: acceptable in D1
created_at TEXT);
CREATE TABLE utterances(call_id TEXT, i INTEGER, t REAL, t_end REAL,
speaker TEXT CHECK(speaker IN ('rep','client','unknown')), kind TEXT CHECK(kind IN ('turn','backchannel')),
text TEXT, words INTEGER, member_ids TEXT, redactions_json TEXT, decision_point INTEGER,
PRIMARY KEY(call_id, i)); -- 11 columns → ≤ 9 rows per INSERT
-- policy: immutable bundles, one row per version per scenario
CREATE TABLE policy_versions(scenario TEXT, version INTEGER, policy_hash TEXT, bank_hash TEXT, playbook_hash TEXT,
weights_hash TEXT, policy_json TEXT, status TEXT CHECK(status IN ('draft','evaluated','published','retired')),
eval_run_id TEXT, model_expected TEXT, created_by TEXT, created_at TEXT, note TEXT, PRIMARY KEY(scenario, version));
CREATE TABLE policy_pointers(scenario TEXT PRIMARY KEY, published_version INTEGER, published_by TEXT, published_at TEXT);
-- what Jev said (free recompute), split by what invalidates it
CREATE TABLE answers(call_id TEXT, i INTEGER, bank_hash TEXT, state_hash TEXT, model TEXT,
state_json TEXT, answers_json TEXT, input_tokens INTEGER, output_tokens INTEGER, latency_ms INTEGER,
cf_request_id TEXT, budget_json TEXT, created_at TEXT, PRIMARY KEY(call_id, i, bank_hash, state_hash));
CREATE TABLE move_answers(call_id TEXT, i INTEGER, playbook_hash TEXT, state_hash TEXT, model TEXT,
answers_json TEXT, created_at TEXT, PRIMARY KEY(call_id, i, playbook_hash, state_hash));
-- what code did with it (audit + timeline)
CREATE TABLE sessions(session_id TEXT PRIMARY KEY, call_id TEXT, mode TEXT, policy_version INTEGER, actor TEXT,
started_at TEXT, ended_at TEXT, model_drift INTEGER DEFAULT 0);
CREATE TABLE decisions(session_id TEXT, call_id TEXT, i INTEGER, policy_hash TEXT, decision_json TEXT,
shown_json TEXT, created_at TEXT, PRIMARY KEY(session_id, i));
CREATE TABLE rewrites(rewrite_id TEXT PRIMARY KEY, session_id TEXT, call_id TEXT, i INTEGER, move_id TEXT,
candidate TEXT, shown INTEGER, rejected_reason TEXT, verification_json TEXT, llm_model TEXT, policy_hash TEXT, created_at TEXT);
-- curation (M1 creates and seeds; M2 fills)
CREATE TABLE moments(moment_id TEXT PRIMARY KEY, call_id TEXT, scenario TEXT,
kind TEXT CHECK(kind IN ('must_say','risk','concern','next_step','uncertain','trainer_pick','coach_seed')),
start_i INTEGER, end_i INTEGER, topic TEXT, jev_json TEXT, policy_hash TEXT, priority REAL,
review_state TEXT DEFAULT 'queued', created_at TEXT);
CREATE TABLE marks(mark_id TEXT PRIMARY KEY, moment_id TEXT, verdict TEXT CHECK(verdict IN ('model','acceptable','avoid')),
note TEXT, use_as_example INTEGER, admin TEXT, created_at TEXT);
CREATE TABLE labels(label_id TEXT PRIMARY KEY, call_id TEXT, i INTEGER, question_id TEXT, gold_json TEXT, admin TEXT, created_at TEXT);
CREATE TABLE eval_runs(run_id TEXT PRIMARY KEY, scenario TEXT, policy_version INTEGER, bank_hash TEXT, model TEXT,
passed INTEGER, l1_pass_rate REAL, l4_pass INTEGER, l5_max_std REAL, l6_max_tokens INTEGER, l6_p95_ms INTEGER,
cost_usd REAL, r2_key TEXT, created_at TEXT);
CREATE TABLE audit_log(id INTEGER PRIMARY KEY, at TEXT, actor TEXT, action TEXT, subject TEXT, detail_json TEXT);
The policy bundle (one per scenario):
interface Policy {
scenario: 'onboarding' | 'customer_success';
version: number;
model_expected: 'jev-1.13.0';
bank: Record<string, JevQuestion>; // judging questions only (nouls, scores, stage, concern type)
playbook: { moves: Move[] }; // {id, title, what, not_for, must_say?: string[], requires?: string[], blocked_by?: string[],
// rewrite_allowed: boolean, lines: [{text_id, text, status:'approved', approved_by, approved_at, source_moment_id?}]}
checklist: ChecklistItem[]; // {id: 'said_who_holds_funds', applies: 'always' | {when: 'client_type', is: 'corporate'} | {when_fact: 'docs_status_known'} | ...}
risk_flags: RiskRule[]; // {id, kind: 'noul'|'persisted'|'code', question_id?, min?, code_rule?, label}
weights: Record<string, { kind: 'noul'|'fact'|'score'|'code'; w: number }>; // CS health composite only
thresholds: { persist_fact: 0.70; signal_on: 0.60; concern_open: 0.60; concern_clear: 0.60; concern_max_age: 8;
concern_type_min_conf: 0.40; next_move_min_conf: 0.35; next_move_min_prob: 0.30; phrasing_min_conf: 0.30;
switch_margin: 0.12; confirm_updates: 2; card_cooldown_utts: 3; uncertain_band: [0.30, 0.70];
risk_flag: 0.60; stage_switch_margin: 0.05; stage_confirm: 2 };
ema_alpha: 0.40;
token_budget: { soft: 11000; hard: 12000; window: 12; chars_per_token: 3.5; // divisor measured, not assumed (§6.2)
drop_order: ['phrasing_beyond_top4', 'all_phrasing', 'window_to_8', 'skip'] };
ingest: { stitch_version: 1; redact_version: 1 };
}
Hashing rules (scripts/hash-policy.mjs, shared with the Worker):
bank_hash = sha256(canonical(bank))where canonical sorts top-level question ids but serialises everycriteriaobject and array in declared order, so a reordering of Choice options changes the hash (option order affects results,guide §3.5, §8). Changing one word of one criterion is a new hash (the consistency cookbooks'_rubric_fingerprintdiscipline,corpus §6b).playbook_hash = sha256(canonical(playbook)): covers move ids, order,what/not_for, line ids and text. Thenext_moveChoice and thephrasing::<move>Choices are generated from the playbook, so their answers live inmove_answersunder this hash. Editing an approved line, the main M2 curation action, invalidates onlymove_answers; every judging answer inanswersstays valid.weights_hashcovers weights, thresholds, checklist applicability rules, risk rules, EMA alpha.policy_hash = sha256(bank_hash + playbook_hash + weights_hash + ingest versions).state_hash = sha256(state_json). Stored on everyanswersrow, together with the exactstate_jsonsent, so any answer is reproducible and the seek cache never serves an answer to a different state (the judges' fatal flaw in poc-fastest's cache).
Recompute semantics, stated honestly: recompute(policy') re-runs step() over stored answers with zero Jev calls. It is exact for weights, EMA, display thresholds, card gates and hysteresis. It is approximate for thresholds that feed back into the state Jev saw (persist_fact changes known_facts/checklist_done; concern_open changes open_concern), because the stored answers were given against the old state. The engine counts decisions where the rebuilt state differs from state_json; M2's weights editor shows it as "state drift: N of M decisions; replay to re-ask", and in M1 the count exists only in the engine's unit tests and the debug drawer. All three proposals claimed free recompute without this caveat; the reference has the same limitation in a worse form (it recomputes from features, dissection §3e).
6. Per-utterance decision loop
6.1 Utterance stitching from Aircall fragments
Evidence from data/samples/onboarding-strong-3339895706.{txt,json}: 159 fragments in 587 s, 118 overlapping the previous one (median gap −0.17 s), 71 of two words or fewer; shards like REP: to / REP: to that, / REP: It's it's not a pay even it's three like a to pay four. cut. interleaved with CLI: Mhmm.. The call-coach call_turns.data carries members and stitched_chain_len but chains are still short. A second, deterministic pass (stitch.ts, pure, versioned as stitch_version, unit-tested on the four samples) runs at import for historical calls and in the Worker for nova-3 output; it is never re-run live:
- Sort by
start; maprole internal → rep,external → client; anything else →unknown(masks speaker-specific nouls to 0 instead of mislabelling, dissection §10b.7). - Backchannel = ≤ 2 words after stripping punctuation, every word in
{yeah, yes, yep, no, mhmm, mm, uh-huh, okay, ok, right, sure, gotcha, exactly, absolutely, cool, fine}, sitting inside the other speaker's run (start < prev.t_end + 1.5 s). Stored askind='backchannel', shown dimmed in the transcript, counted on the enclosing utterance, kept in the state window (they carry acceptance signal), never a decision point. - Merge consecutive same-speaker non-backchannel fragments when
next.start − prev.t_end < 1.0 s(overlap counts), or< 2.5 swhenprev.texthas no terminal punctuation; merged length capped at 120 words;t_end = max(member ends);member_idskept for traceability. - Split anything > 120 words at sentence boundaries into sub-utterances sharing
t, sequentialt_end. - Decision point = a stitched
turnwith ≥ 4 words, or ending in?, or the last utterance of a speaker run. Everything else updates the transcript only. - Garbled cross-talk is a transcription artefact, left as is; L5 measures whether Jev is robust to it, and re-transcription with nova-3 from Aircall audio is the fallback.
STITCH_V1 = { mergeGapS: 1.0, mergeGapUnfinishedS: 2.5, backchannelMaxWords: 2, backchannelGapS: 1.5, maxWords: 120 } are ASSUMPTION, tuned by eye on the samples in M1 and on 20 calls in M2. Judge-verified expectation: a naive stitch of the strong sample yields ~94 turns (59% of fragments, ~18 words each), so 40-80 decision points for a 6-10 minute call is realistic.
6.2 State sent to Jev and the token budget
Small, filtered JSON object; only fields a question names by backticked path (guide §7). Numbers that only code needs (talk ratio, minute, word counts) stay out of the state; they are code signals, not Jev inputs.
interface JevState {
call_facts: {
scenario: 'onboarding' | 'customer_success';
client_type: 'personal' | 'corporate' | 'unknown'; // M1: 'unknown' unless obvious; M2: from CT DB pre_call
stage_history: string[]; // last 6 distinct, code-compressed
known_facts: string[]; // persisted *_known ids
checklist_done: string[]; // persisted said_* ids
open_concern: { open: false } | { open: true; type: string };
resolution?: 'none' | 'reported' | 'owned' | 'resolved'; // CS only
};
recent_transcript: { t: string; speaker: 'rep' | 'client' | 'unknown'; text: string }[]; // last 12 utterances incl. backchannels
latest_utterance: { t: string; speaker: 'rep' | 'client' | 'unknown'; text: string };
uid?: string; // eval/stability runs only (cache buster, guide §4.3)
}
Token estimate at 3.5 chars/token, not 4: on Cloudflare the reference's 37-question request averages ~27.9k chars for ~7.9k usage.input_tokens (eval/cf-parity.json: 380,577 tokens over 48 requests), and guide §10 records the same ratio (~25 KB → ~7.5k). Over compact JSON (no pretty-printing; the judges measured the brief's pretty-printed onboarding bank at ~9.2k on its own), plus the ~300-token fixed overhead the guide infers (guide §4.10):
| Part | Onboarding est. |
CS est. |
|---|---|---|
| Overhead | 300 | 300 |
call_facts |
170 | 170 |
recent_transcript (12 × ~35 words) + latest_utterance |
800 | 800 |
| Shared nouls (18) + scores (3, levels flattened to strings) | 2,400 | 2,400 |
| Scenario nouls (23 / 12) | 2,630 | 1,490 |
| Stage Choice (10 + none / 9 + none) | 690 | 630 |
| Concern-type Choice (8 + none / 7 + none) | 400 | 340 |
next_move Choice (14 / 13 moves, full fixed list) |
1,140 | 1,090 |
phrasing::<move> for code-allowed moves only (≤ 8 × 3 lines) |
1,030 | 970 |
| Total (fan-out request) | ~9,600 | ~8,200 |
moves-only request (judging hit + move miss, §3): overhead + call_facts + transcript + next_move + phrasing:: |
~2,100 | ~2,000 |
Budget enforcement in code before every request (budget.ts): estimate = ceil(JSON.stringify(input).length / policy.token_budget.chars_per_token) + 300, with chars_per_token = 3.5 in policy v1 and re-derived from Σ chars / Σ usage.input_tokens of the first replays (deliverable 9); a divisor of 4 under-reports by ~14%, so a request the estimator passed at 11.9k would really be ~13.5k and the "hard cap never exceeded" claim below would rest on a miscalibrated instrument, which is why the divisor is policy data and L6 checks measured usage.input_tokens, not the estimate. Soft budget 11,000: over it, apply the drop order (1) drop phrasing:: questions for allowed moves beyond the top 4 by prior/stage, (2) drop all phrasing:: questions, (3) shrink the window 12 → 8, (4) mark the decision skipped_budget, hold the previous snapshot and log. Hard cap 12,000 is never exceeded; a 4xx from an oversized request is a bug, not a retry (guide §2.5). The degradation is written to answers.budget_json and the decision message. Dropping whole phrasing:: questions is hash-safe: each such Choice has a fixed option set, questions are evaluated in isolation (guide §1), and omitting one changes no other answer. The next_move Choice always carries the full move list in fixed order, so its probabilities are comparable turn to turn and its answers all live under one playbook_hash (this repairs data-first's per-turn shortlist, which broke its own hash discipline).
6.3 Question-bank structure per scenario
The bank is the DRAFT in brief §4.3-4.5, kept in Stevan's format, assembled at build time by scripts/build-policy.mjs from four files: shared.json, onboarding.json or customer-success.json, the scenario playbook, and verify.json (rewrite verifiers, sent only in the verification request). One transformation is applied at build time: every Score level is flattened to one string ("Disengaged: one-word or one-line answers; deflecting ('just email me'); trying to end the call"), exactly as cf_safe_questions in scripts/replay_via_cf.py does, because Cloudflare's typesafe/jev wrapper returns HTTP 500 code 2002 ("Failed to parse model output") when Score criteria are {summary, signals[]} objects, and the brief's three Scores (engagement, urgency, trust, brief §4.3) are all written that way (guide §10). Nouls and Choices keep their structured instructions/criteria. bank_hash is computed over the flattened bank, and a unit test asserts that no built bank contains an object-valued Score level, so the first full request does not fail and the hash does not change on day 1. Nothing in the engine knows question names except through a signals map of id → role:
type Role = 'client_turn' | 'rep_turn' | 'either' | 'window_fact' | 'must_say' | 'risk' | 'score' | 'stage' | 'concern_type' | 'next_move' | 'phrasing';
interface SignalSpec { id: string; role: Role; lock_on?: 'rep' | 'client' | 'any'; weight?: number }
| Group | Onboarding ids | CS ids | Rule in code |
|---|---|---|---|
| Shared client-turn nouls | client_objecting, client_accepts, client_disengaging, client_confused, client_ready_to_book, next_step_agreed |
same | masked to 0 on rep turns |
| Shared either-speaker nouls | client_asked_about_{rate,safety,timing,fees,documents}, client_mentioned_alternative_provider, client_mentioned_deadline, rep_proposed_next_step |
same | unmasked |
| Shared rep-turn nouls | rep_asked_open_question, rep_explaining_at_length |
same | masked on client turns |
| Risk nouls | rep_made_rate_prediction, rep_made_guarantee (rep-turn) + window funding_from_third_party, jurisdiction_concern |
rep_made_rate_prediction, rep_made_guarantee |
≥ 0.60 on the turn / ≥ 0.70 persisted; latched |
| Window facts | purpose_known, pair_known, timing_known, frequency_known, funding_source_known, beneficiary_known, docs_status_known, current_provider_known, decision_maker_known |
issue_reported, upcoming_payment_known |
persist ≥ 0.70 on any turn |
| Must-say checklist nouls | said_who_holds_funds, said_rate_transparency, said_quote_lifetime, said_settlement_timing, said_fund_from_own_account, said_booking_is_binding, said_minimum, said_docs_needed, said_payment_reason, said_rm_contact |
issue_resolved_or_owned, rep_checked_status_with_specifics, said_forward_deposit_and_liability (conditional) |
persist ≥ 0.70 on rep turns only |
| CS-only nouls | — | no_upcoming_need, client_wants_to_wait_for_rate, client_asked_about_{forwards,alerts_or_orders,recurring,access}, rep_offered_tool |
per role |
| Scores (3 levels) | engagement, urgency, trust |
same | centred, health only |
| Stage Choice | onboarding_stage (10 + none: "Too little context yet") |
cs_stage (9 + none) |
hysteresis |
| Concern-type Choice | onboarding_objection_type (8 + none) |
cs_objection_type (7 + none) |
consulted only when client_objecting ≥ 0.60 |
next_move Choice |
14 moves, {what, not_for} |
13 moves | full list, code filter after |
phrasing::<move> |
≤ 8 × 3 lines | ≤ 8 × 3 lines | options are line ids with null description + the text, so choice is a verbatim id |
Two rules the judges insisted on, both now in code and in the L1 suite: said_* items lock only when latest_utterance.speaker === 'rep' (a client paraphrase such as "so you never hold my money?" reads the same window and would otherwise tick a compliance item; a one-question REST probe today returned 0.02 for that utterance when the instruction names the rep as speaker, but the code mask is the guarantee, not the wording); and the objection-type Choice is only read when the absolute noul fires, because a Choice always crowns a winner (guide §3.1). Every stage Choice gains an explicit none option absent from the draft. Amounts, dates and counts are never asked of Jev (guide §7); a regex amount extractor in code feeds the checklist's said_minimum applicability rule.
6.4 The step() engine and hero metrics in code
// engine/step.ts — pure; identical in the DO (M1) and the Queue consumer (M2). No I/O, no clock, no randomness.
export function step(s: SessionState, u: Utterance, a: Answers, m: MoveAnswers | null, p: Policy): { s: SessionState; snap: Snapshot; moments: Moment[] } {
const f = extractFeatures(a, u.speaker, p); // speaker masks; score/(levels-1); persisted facts clamp to 1
s = persistFacts(s, f, u, p); // *_known on any turn; said_* only if u.speaker === 'rep'; never un-persist
s = updateConcern(s, f, a, u, p); // open ≥ .60 on client turn; type if conf ≥ .40 else parent bucket ('money'|'process'|'unknown');
// `since` refreshed on every re-raise; clear on accept ≥ .60 && disengaging < .60, or 8 decisions after last raise
s = updateStage(s, a[p.stageQuestion], p); // switch only if challenger leads by stage_switch_margin for stage_confirm decisions
s = updateResolution(s, f, u, p); // CS: none → reported (client, issue_reported ≥ .60) → owned (rep, issue_resolved_or_owned ≥ .70) → resolved (client, client_accepts ≥ .60)
const risks = riskFlags(s, f, u, p); // latched booleans with the utterance index and value
const hero = p.scenario === 'onboarding' ? onboardingHero(s, f, p, risks) : csHero(s, f, p);
const card = pickCard(s, a, m, p); // allowedMoves() in code, then Jev's relative pick within them, then hysteresis
s = pushWindow(s, u);
return { s, snap: { i: u.i, hero, risks, stage: s.stage, concern: s.concern, checklist: checklistView(s, f, p), card, signals: f, health: s.health, budget: a.__budget }, moments: emitMoments(s, f, u, p) };
}
// Onboarding hero: a fraction and a list, never a probability.
function onboardingHero(s: SessionState, f: Features, p: Policy, risks: RiskFlag[]): Hero {
const applicable = p.checklist.filter(item => applies(item, s)); // e.g. said_docs_needed only while docs incomplete; said_minimum only if amount unknown or < 5k mentioned
const done = applicable.filter(item => s.persisted[item.id] !== undefined);
const uncertain = applicable.filter(item => !s.persisted[item.id] && inBand(f[item.id], p.thresholds.uncertain_band));
return { kind: 'onboarding', completeness: done.length / Math.max(1, applicable.length), done, uncertain, applicable, risks };
}
// CS hero: resolution state + secondary call-health composite with EMA (reference maths, CT weights; brief §4.6 draft).
function csHero(s: SessionState, f: Features, p: Policy): Hero {
const inst = clamp(0.5 + Σ(p.weights, f), 0.03, 0.97); // scores centred by (x − 0.5)·2; nouls raw; code signals: concern_open, talk_penalty
s.health = s.health === null ? inst : p.ema_alpha * inst + (1 - p.ema_alpha) * s.health;
return { kind: 'customer_success', resolution: s.resolution, health: s.health, moved: topDeltas(s, 3) };
}
Both hero functions are noul-first: nouls calibrate best on the independent test (ECE 0.012 vs 0.086 Choice vs 0.254 Score, prior-art §5), so no hero number is derived from a Score magnitude; Scores only enter the secondary health bar, centred so the middle level is neutral (guide §3.4).
6.5 Gates, thresholds, anti-flicker
All values live in policy.thresholds; defaults are the reference's, which are starting points, not tuned values (dissection §5a, §9d). M2 replaces them per question from calibration at a target precision (§9).
| Decision | Rule | Source |
|---|---|---|
| Persist a fact / tick a checklist item | window noul ≥ 0.70; said_* only on rep turns; never un-persists in a session |
reference FACT_PERSIST_THRESHOLD; data-first §6.3 |
| Light a signal | noul ≥ 0.60; 0.30-0.70 rendered "uncertain", not off | consistency cookbook band (guide §6) |
| Open concern | client_objecting ≥ 0.60 on a client turn; type from the Choice if confidence ≥ 0.40, else the parent bucket (money for rate/fees, process for documents/funding, unknown); since refreshed on every re-raise |
dissection §4b; classification-with-confidence cookbook |
| Clear concern | max(client_accepts, client_ready_to_book, next_step_agreed) ≥ 0.60 AND client_disengaging < 0.60; or 8 decisions since the last raise |
reference; "a disengaging turn never clears" |
| Stage | switch only if the challenger leads the current stage by ≥ 0.05 for 2 consecutive decisions; none never displaces a real stage |
call-coach-ai/decide.js via prior-art §2.1 |
| Show next-move card | next_move.confidence ≥ 0.35 and max(prob) ≥ 0.30, else "listening… (leaning X)" |
reference; 0.35 on a 14-option Choice ≈ 0.39 peak |
| Switch the card | challenger must lead by ≥ 0.12 or stay on top for 2 decisions; 3-decision cooldown after a switch; a rep-acked card is hidden for the rest of the call | decide.js |
Suppress a move (code, not not_for prose) |
move must_say already persisted; requires facts missing; blocked_by facts persisted; agree_next_step while a concern is open; check_serviceability_first forced to the top when jurisdiction_concern ≥ 0.60; concern open → shortlist restricted to that topic's moves. A suppressed top pick is shown greyed with "rule: X hidden because Y" |
dissection §5b gap; prior-art §2.1 |
| Highlight a line | phrasing::<move>.confidence ≥ 0.30 else the move's first approved line |
reference |
| Risk flag | noul ≥ 0.60 on that turn (guarantee, rate prediction) or persisted ≥ 0.70 (third-party funding, jurisdiction); code rule: booking discussed while said_booking_is_binding unlocked; flags never auto-clear; admin can mark a false positive, which writes a label |
brief §4.6 |
| Jev error, timeout (10 s race) or 5xx after 2 retries | hold the previous snapshot, mark the decision unknown, never render zeros as signal |
guide §4.8 |
| 402 code 2021 | stop the session with an operator alert; no retry | guide §4.6 |
Retry policy mirrors the SDK defaults: 408/429/5xx (529 included), max 2 retries, 500 ms doubling to 5 s with 25% jitter, honour Retry-After ≤ 60 s; fail fast on 400/401/402/403/422 (guide §4.8).
6.6 Tailored-rewrite path (off the critical path, droppable, flag per scenario, off in M1)
Specified and built behind policy.playbook.moves[].rewrite_allowed plus a global REWRITE_ENABLED flag; enabled only after Stevan and compliance accept §10's policy.
- Trigger (reference
personalize.pyrules): card un-gated; moverewrite_allowed; ≥ 2 decisions since the last rewrite; and (move changed, or a new fact locked, or 8 decisions on the same move). Moves whosemust_saycovers safeguarding, binding or fees are neverrewrite_allowed: those lines stay verbatim-approved (data-first §6.6). - Generator (
ctx.waitUntilin M1,rewrite-jobsQueue in M2; 3 s timeout; ≤ 40 words): prompt = move title/what, its approved lines as tone anchors, the last 6 pseudonymised utterances,known_facts; instruction "use only details in the transcript; never state numbers, rates, dates, partner names, schemes or promises that were not said". Code cleans and length-checks the line. - Code denylist before spending a Jev call:
/guarantee|always cheaper|will (go|come) (up|down|back)|FSCS|risk[- ]free|definitely|protected by/i→ drop, reasondenylist. - Jev verification: a second, small request (state
{candidate_line, move:{title,what}, recent_transcript, known_facts}, ~1.5k tokens) with six single-condition nouls,true= the bad case, aggregated with max (guide §7, SDE-cascade rule):invents_fact(drop ≥ 0.50),off_move(drop ≥ 0.50),makes_promise_or_guarantee(≥ 0.30),predicts_rate_direction(≥ 0.30),implies_scheme_protection("states or implies that funds are protected by a named compensation scheme such as FSCS", ≥ 0.30),names_payment_partner(≥ 0.30 until Stevan says which partners reps may name,brief §6 Q10). Verification unavailable → drop (the reference passes on error, dissection §6d; inverted here because the line is a compliance surface). - Stale check (move changed meanwhile → drop), then a
rewriteWebSocket message carrying{kind:'rewrite', rewrite_id, verified:true}; the card renders it under the approved lines, labelled "tailored (verified)". Every candidate, shown or dropped, is arewritesrow with its six verifier values; the dashboard's debug drawer shows dropped candidates struck through with the reason so Stevan can audit the filter.
7. Ingestion & STT
Historical Aircall calls (no STT). scripts/import-call-coach.mjs reads the three exports in data/call-coach/*.jsonl.gz (psql COPY escaping: unescape backslashes before JSON.parse), selects category ∈ {Onboarding, Customer Service}, duration ≥ 180 s, ≥ 20 turns (1,895 + 1,226 usable calls), and for each: maps role internal → rep, external → client; drops external_phone and all Aircall metadata; runs stitch() then redact() (§10); writes the un-redacted normalised transcript and the coach record to R2 raw/{call_id}/; writes calls + utterances rows through POST /api/calls/import in ≤ 9-row statements; keeps only the coach record's numeric fields in calls.coach_scores_json (six dims, call_value_score, quotes[].turn_idx as hints for where to look, not as labels). scenario = category with scenario_source='call-coach' (LLM-assigned; M2 adds a Jev scenario Choice with disagreement flags, corpus §1e). M1 imports the four exemplars; further calls follow the DPA answer (§13 Q1).
New audio uploads (nova-3). Browser → PUT /api/calls/upload with a raw binary body and Content-Type/Content-Length (no multipart: request.formData() buffers the whole file and forwarding it makes a second copy against the 128 MB isolate limit; Aircall's 32 kb/s MP3 is harmless at ~14 MB per hour, but a 40-minute WAV is ~77 MB and would OOM) streamed to R2 raw/audio/{call_id}.{ext} with env.R2.put(key, request.body, { httpMetadata: { contentType } }) → the Worker reads the first 16 bytes back (R2.get with a range) and sniffs the container: ID3 or an MPEG frame sync for mp3, ftyp at offset 4 for m4a, RIFF…WAVE for wav; anything else, or a mismatch with the declared type, deletes the object and returns 415 rather than forwarding an unvalidated blob to a third party → the object body is streamed from R2 to nova-3 over REST with the verified query string plus keyterm and mip_opt_out=true → raw JSON to R2 raw/stt/ → results.utterances[] mapped to fragments {start, end, speaker: 0|1, text} → speaker map → stitch() → redact() → D1. Accepted content types in M1: audio/mpeg, the only one verified through the proxy (a 2.3 MB file, Content-Type: audio/mpeg, stt "Live verification"); audio/mp4 (m4a) and audio/wav are advertised only after the day-1 probe shows they survive the proxy (§13), and WAV above 25 MB is rejected with "convert to mp3". Size cap 100 MB (request body limit). Verified on a real 9:46 mono Aircall MP3: 8.6 s, 111 utterances, speaker fields survive the proxy. The body-size ceiling on the Cloudflare proxy beyond ~2.3 MB is unverified; M1 measures it with a 30-40 min file and, if it fails, returns a clear error rather than chunking (chunking at silence boundaries is M2).
Mapping diarised speakers to rep/client. Three layers, all in the Worker: (1) code heuristic: the speaker whose first 60 s contain "Currency Transfer"/"CurrencyTransfer", "calling from", or a known rep first name is rep (the strong sample shows the client answering first, so "first speaker" alone is insufficient); (2) one Jev request with one Choice per diarised speaker over that speaker's first 8 turns, options {rep, client, other} with contrastive descriptions, accepted at confidence ≥ 0.80 (corpus §1b); (3) a swap toggle on the replay page that rewrites role_map_json, re-labels utterances.speaker and writes an audit_log row. Below 0.80 the call is flagged role_map_conf low and speakers stay unknown until the toggle is used. Aircall recordings are mono (verified), so multichannel=true is not available; M3 prefers per-leg audio over diarisation wherever the transport allows.
Audio for historical calls. Recordings are fetchable with GET https://api.aircall.io/v1/calls/{id} → recording signed URL (verified, stt "Live verification"). Where the Aircall credentials live and how the call-coach host is reached are recorded in the ops runbook, which is not published; this document is (§4 Pages) and deliberately carries no hosts, paths, key or credential details. data/samples/call-3339895706.mp3 already exists, so the audio-synced demo runs on a real call with zero nova-3 cost: the Aircall transcript is the text, the audio only drives the clock.
8. Dashboard and Admin console
8.1 Dashboard (Worker static assets; TypeScript, no framework; Chart.js as in the reference)
| Panel | Shows | Source |
|---|---|---|
| Player | <audio> element (uploads and calls with audio; seeking needs the Range-enabled audio route, §4) or a virtual clock; play/pause/seek; speed 0.5-20×; call picker; scenario badge; "catching up (N queued)" badge driven by the DO's pending counter (§4) at high speed |
client |
| Transcript | stitched utterances with time and speaker colour, backchannels dimmed, decision points marked, talk-share bar and rep-monologue warning (code) | GET /api/calls/:id + decision.talk |
| Hero (onboarding) | checklist: each applicable must-say item as said / uncertain / unsaid / n/a with the rep utterance it locked on; completeness x/y; risk flags in red with the offending utterance and value | decision.hero |
| Hero (CS) | resolution chip none → reported → owned → resolved with timestamps; health bar + EMA line; "what moved it" top 3 | decision.hero |
| Stage strip | history chips; current stage with confidence | decision.stage |
| Open concern | type or parent bucket, confidence, since, what would clear it; the topic's approved lines | decision.concern |
| Next best move | title, what, up to 3 approved lines with the highlighted one marked "say:", tailored line if a rewrite arrives, "rule: … hidden because …" note, "listening…" state; ack/dismiss buttons |
decision.card, rewrite |
| Signals | grid of lit / uncertain / off signals, lock icons on persisted facts, score levels | decision.signals |
| Weights & thresholds (M2) | editable → set_weights → recomputed (zero Jev calls) with the state-drift count; belongs with corpus calibration (§6.5, §9), not in Stevan's M1 ask; in M1 the messages exist only behind the debug drawer and are not an acceptance criterion |
|
| Jev telemetry | model id (with a drift warning if ≠ model_expected), latency mean/p95, tokens, budget degradation, cumulative cost, last request/response drawer, 402 alert |
decision.jev, decision.budget |
| Mark moment (M1) | select an utterance span → model / acceptable / avoid + tag + note → moments + marks |
WebSocket protocol (reference's hello / update / recomputed / phrasing / error extended, dissection §7c; every message {v:1, type, seq, ts, …}; Access JWT on the upgrade):
type ClientMsg =
| { type: 'load'; call_id: string; policy_version?: number }
| { type: 'utterance'; i: number } // the browser clock passed utterances[i].t_end
| { type: 'seek'; i: number } // DO resets and re-evaluates 0..i; cached rows cost no Jev call
| { type: 'set_weights'; weights: Policy['weights']; thresholds: Partial<Policy['thresholds']> } // experiment, not publish; M2 panel, M1 debug drawer only
| { type: 'ack_card'; card_id: string } | { type: 'dismiss_card'; card_id: string }
| { type: 'mark_moment'; start_i: number; end_i: number; verdict: 'model'|'acceptable'|'avoid'; tag: string; note?: string }
| { type: 'flag_false_positive'; kind: 'risk'|'must_say'|'concern'; i: number } // writes a label
| { type: 'set_speaker_map'; map: Record<string, 'rep'|'client'|'other'> } // uploads only
| { type: 'ping' };
type ServerMsg =
| { type: 'hello'; session_id: string; call: CallMeta; policy: { scenario; version; policy_hash; bank_hash; playbook_hash }; model_expected: string; history: Decision[] }
| { type: 'decision'; d: Decision } // one per decision point; d.i, hero, risks, stage, concern, checklist, card, signals, health, talk, budget, jev
| { type: 'rewrite'; i: number; move_id: string; rewrite_id: string; text: RepFacingText | null; rejected?: string; verification: Record<string, number> }
| { type: 'recomputed'; timeline: Decision[]; policy_hash: string; state_drift: number }
| { type: 'queue'; pending: number } // length of the DO's serial chain (§4); drives the "catching up" badge
| { type: 'alert'; code: 'credits_empty'|'model_drift'|'budget_skipped'; message: string }
| { type: 'pong' } | { type: 'error'; code: string; message: string };
// Rep-facing text is typed so the UI cannot render anything else.
type RepFacingText = { kind: 'approved'; text_id: string; policy_version: number }
| { kind: 'rewrite'; rewrite_id: string; verified: true };
The seek cache is keyed (call_id, i, bank_hash, state_hash) for judging answers and (call_id, i, playbook_hash, state_hash) for move answers; scrubbing back and forth costs nothing after the first pass as long as the rebuilt state matches, and re-asks (paid, logged) when it does not. A judging hit with a move miss sends the moves-only request (§3); the DO never re-pays for judging answers it already holds, and a full request is sent only when answers misses.
8.2 Admin console
M1 ships the seed: the mark-moment form on the replay page, moments emitted by the engine at each session (kinds and triggers below), and a read-only GET /api/moments?call_id= list. M2 ships the console on Pages behind Access, with the review queue ordered by information value (corpus §3c): uncertain-band moments first, then unlabelled top-composite calls, then bottom-composite, then disagreements between trainer_pick and the atomic composite.
| Moment kind | Emitted when | Admin sees |
|---|---|---|
must_say |
a said_* noul first crosses 0.70 on a rep turn |
the rep turn ± 2, which item, the value |
risk |
rep_made_guarantee / rep_made_rate_prediction ≥ 0.60, third-party funding, booking-without-binding |
the rep turn, the value |
concern |
concern opened → cleared/expired (episode cut, cap 8 decisions) | opening client turn, rep turns, resolution turn |
next_step |
next_step_agreed ≥ 0.70 |
the exchange |
uncertain |
any gate value in 0.30-0.70 | the turn; most valuable for criteria repair |
trainer_pick (M2) |
per-episode exemplar_candidate ≥ 0.70 from the batch scorer's episode request |
the episode |
coach_seed |
coach_scores_json.quotes[].turn_idx |
the old coach's pick, as a weak hint only |
Actions and effects (M2): mark model / acceptable / avoid (+ "use as example") → marks row; a model rep turn becomes a candidate playbook line (verbatim, tokens cleaned, editable, source_moment_id set, status='draft' until approved_by) and a candidate examples snippet for the relevant criterion; avoid → not_for text and a false example; all three → gold labels for L1/L2. Edit weights/thresholds → new draft policy version, corpus re-scored from answers, before/after diff (0 Jev calls). Edit question wording → new bank_hash, harness re-scores the labelled set only (~$0.30). Approve lines → compiled into the next playbook version (playbook_hash changes; judging answers untouched). Promotion: draft → eval run → diff view (flipped L1 cases, moved exemplars, ranking deltas) → publish (D1 pointer + KV mirror + audit_log row) → background backfill under the new hashes; old rows kept for rollback. Question wording changes are drafted by an LLM from Stevan's intent in the house style and gated by the harness (corpus §3b option B); nobody edits a live prompt directly.
9. Evaluation harness & promotion gate
scripts/eval.mjs runs against POST /api/evaluate on the deployed Worker (same model path as production) and, from M2, from an admin "Run eval" button; results go to R2 eval/{run_id}/ and a row in eval_runs. The golden set is snapshotted to R2 golden/{policy_version}.json with a copy committed to the repo so CI runs without D1.
| Layer | Gold | Check | Gate |
|---|---|---|---|
| L1 labelled utterances | eval/labelled/{onboarding,cs}.json: 25-40 cases per scenario, hand-written from the four exemplars plus synthetic edge cases: negations ("we're not worried about the rate"), an injected instruction inside client speech, "is that guaranteed?" (must not tick rep_made_guarantee), and the client paraphrase "so you never hold my money?" (said_who_holds_funds < 0.40, checklist unchanged). Coach quotes are hints for where to look, never gold: two LLMs agreeing is not a promotion gate |
per-case threshold checks as data (client_objecting >= 0.60, onboarding_objection_type in {rate}, …), the reference's report format (dissection §9b) |
≥ 90% pass: reported in M1a, the M1b exit target, a hard gate from M2 (as L5); no case regressing across a threshold vs the published version |
| L4 call shapes | the four exemplars (+ 2 per scenario in M2) | onboarding-strong ends with completeness ≥ 0.5, next_step_agreed persisted, 0 risk flags; onboarding-weak ends below strong's completeness; CS-strong reaches resolved; CS-weak never reaches owned; every replay: zero Jev errors, requests == decision points, one model id; a card shown on 50-100% of decision points |
hard |
| L5 stability | L1 × 5 with a fresh uid in state |
per-question std ≤ 0.03; list of threshold-crossing flips (each is a criterion to rewrite first) | report in M1, gate in M2 |
| L6 budget | every run | every usage.input_tokens ≤ 12,000 (hard), p95 ≤ 11,000 (report), p95 Jev latency ≤ 1.5 s, cost per call ≤ $0.05, model == model_expected |
hard; a model change fails L6 and opens a drift run |
| L2 marked episodes (M2) | marks |
`mean(handling_quality | model) > acceptable > avoidby ≥ 0.3;exemplar_candidate ≥ 0.70on ≥ 80% ofmodel` |
| L3 ranking (M2) | admin-ranked ~30 calls + outcomes via pd_ct_id (coverage 82% Onboarding / 64% CS, selection bias reported) |
Spearman(hero, admin rank) ≥ 0.6; AUC(completeness → activated within 30 days) reported | soft |
| Calibration (M2) | ≥ 30 labels per noul | reliability plot and ECE per noul; thresholds chosen per question at a target precision (risk flags ≥ 0.9, checklist ticks balanced), replacing the uniform 0.60/0.70 | report → thresholds |
Promotion gate. M1: a policy version's status becomes evaluated only when scripts/eval.mjs writes a passing eval_runs row whose created_at is later than the policy's, and the loader refuses to mark published otherwise; the loader also refuses any bundle containing [VERIFY] or a line without approved_by. Because the DO pins only a published row and the replay (deliverables 5-6) must run before the harness gate exists (deliverable 8b), M1a has one explicit, audited bypass: scripts/publish-policy.mjs --bootstrap marks a draft published with note='bootstrap' and an audit_log row, is refused as soon as a passing eval_runs row exists for that scenario, and is deleted in M1b when the gate switches on. The [VERIFY]/approved_by refusal has no bypass at any stage. M2: the same rule enforced on the publish route with the diff view in front of it. Model drift: the DO compares response.model with model_expected on every answer, sets sessions.model_drift, shows the warning, and the nightly harness run opens a drift eval_runs row with model as the changed variable. Answers for eval are cached by (state_hash, bank_hash, model), so re-running an unchanged bank is free.
10. Compliance & PII
Pseudonymise in code, at import/upload, before anything reaches D1, the DO, Jev, R2 eval/ or the browser (redact.ts, versioned redact_version): IBAN (mod-97), UK sort code + account number, card numbers (Luhn), phone numbers (UK/intl formats and spoken digit runs ≥ 6, "four two one one oh nine" normalised first), emails, postcodes, dates of birth, passport-like tokens → typed tokens [IBAN], [PHONE], [EMAIL], [POSTCODE], [DOB]; the rep's name from call_records.agent and a rep first-name list → [REP] (employee names are personal data; said_rm_contact works on the token); client names from pd_first_name/pd_last_name → [CLIENT], plus the capitalised-token-after-Hi|Hello|Thanks|Bye heuristic for spoken names not in metadata ("Hi, Karen" in the exemplar). Amounts and currencies are kept (they drive pair_known and the amount extractor; ASSUMPTION that amounts alone are not identifiers, to confirm). utterances.redactions_json records what was replaced; no mapping is stored in M1. M2 adds a Jev pii_present noul sweep over the pseudonymised text with ≥ 0.50 routed to a redaction queue (corpus §1d).
What never enters D1: the coach's summary, quotes[].reason, risks, action_items (LLM prose that names clients) stay in R2 raw/{call_id}/coach.json; only numeric scores and turn indices are copied. Raw audio, un-redacted transcripts and nova-3 JSON live only under R2 raw/ with a lifecycle rule.
Data processing. typesafe/jev on Workers AI is a "Third-party" model; whether TypeSafe's DPA / no-training / ZDR commitments cover traffic proxied through Cloudflare is unresolved (guide §4.11), and prior processing by the OpenAI coach pipeline is not a lawful basis for a new processor (struck from production-shaped on the judges' reading). Rule: until Stevan confirms (§13 Q1), client-derived text goes to Jev only for the four exported, pseudonymised samples and the synthetic L1 cases; every deliverable in §11 is written to be acceptable on that corpus. Gateway body logging is off on jev-copilot; REST calls also send cf-aig-collect-log: false. Deepgram mip_opt_out=true. Cloudflare Access with Stevan's email in front of the Worker and the Pages docs project from the first deploy, and a service token for the scripts (§4): a workers.dev or pages.dev URL with call transcripts or call-level reports is never open.
Approved-text policy. Every string a rep can see is a RepFacingText: an approved playbook line by text_id + policy version (status='approved', approved_by, approved_at), or a rewrite whose rewrites row holds verified=1 and six verifier values. The policy loader hard-rejects any bundle containing [VERIFY] or a line without approved_by; the brief's v0 playbooks carry ~10 [VERIFY] lines, so M1 needs a 30-minute red-pen sitting with Stevan (as compliance stand-in) or those lines are dropped from v1. No badge, no "draft shown in yellow".
Audit log. audit_log rows, actor = Cloudflare Access identity, for: policy upload / eval / publish / retire, role-map override, label and mark writes, weight experiments (set_weights with the hashes, M2), bootstrap publish (M1a only), rewrite shown / dropped, session start / end with policy_version and model. decisions.shown_json records what was on screen at each decision point from which hashes; answers.state_json records what Jev saw. Together they reconstruct any moment of any session.
Recording disclosure. Whether calls must be announced as recorded is a Stevan/compliance question (brief §2.3); the copilot does not change what Aircall records in M1.
11. Milestones
M1: replay PoC (target 10-12 working days of agent-driven development, in two halves)
Ships on the four exported samples; scales to more calls the day Q1 in §13 is answered.
Order and cut line, so the PRD can plan against it. Summing the deliverables at their low ends gives 10.5-14.5 days (engine + DO alone is 2.5-3.5 d with the explicit queue and per-step persistence), and risk 1 says the first L1 pass will be well under 90%, so the original 8-10 days were credible only with a cut. M1a (days 1-7) is the committed half: deliverables 1-6, 8a and 9 on the four samples, demo at the end of M1a on call-3339895706.mp3 with its Aircall transcript and audio, which needs no upload path. M1b (days 8-12): deliverables 7, 8b and 10 (stretch). M1b slips before M1a does. Rows are listed in build order.
| # | Phase | Deliverable | Acceptance criteria |
|---|---|---|---|
| 1 | M1a | Infra: Worker jev-copilot (from wrangler.jsonc of jev-lab) with AI, D1, R2, DO and static-assets bindings; D1 schema v1 (§5) as migrations; AI Gateway jev-copilot with body logging off and a spend limit; Cloudflare Access on the Worker hostname with two policies (Allow: Stevan's email; Service Auth: service token jev-scripts) and on the Pages project; Jev REST token as a wrangler secret; vitest with a stubbed env.AI |
wrangler deploy from clean; migrations apply; an unauthenticated request to /api/* and /ws/* is rejected; a request carrying the service-token headers is accepted and its audit_log.actor is the token's common_name; git grep for the token strings finds nothing; the gateway logging setting and spend limit are screenshotted into the runbook; day-1 probe records whether the binding honours {gateway:{id}} (else REST fallback is switched on) |
| 2 | M1a | stitch() v1 + redact() v1 as pure modules with fixtures from data/samples/ |
the four samples stitch to 40-80 decision points each with no cross-speaker merge on hand-check; a grep over utterances.text for phone/IBAN/email/postcode patterns, the agent name and pd_* names finds nothing; stitch_version and redact_version recorded on calls |
| 3 | M1a | Importer scripts/import-call-coach.mjs + POST /api/calls/import, authenticating with the service token; --audio <file> puts a local recording in R2 raw/audio/ for the M1a demo |
the four samples import with scenario_source='call-coach', coach_scores_json numeric-only, raw transcript + coach JSON in R2 raw/; call-3339895706.mp3 lands in R2 with audio_r2_key set; inserts use ≤ 9 rows per statement; re-import is idempotent |
| 4 | M1a | Policy v1 per scenario from brief §4-5 via scripts/build-policy.mjs (Score levels flattened to strings, bank_hash over the flattened bank, token_budget.chars_per_token=3.5, + none on stage Choices, [REP]/[CLIENT] in examples, must_say/requires/blocked_by on moves, rewrite_allowed=false on safeguarding/binding/fees moves), three hashes, loader, scripts/publish-policy.mjs --bootstrap (§9) |
loader rejects a bundle with [VERIFY] or an unapproved line (unit test); no built bank contains an object-valued Score level (unit test); v1 loads only after the red-pen sitting; v1 is made published for M1a by --bootstrap with an audit_log row, and the command refuses once a passing eval run exists; one representative request through the binding returns jev-1.13.0 and an answer for every question id, including the three Scores; estimated tokens ≤ 11k at 3.5 chars/token on the longest window in the samples |
| 5 | M1a | step() engine + CallSession DO: masks, persistence (rep-turn-only said_*), concern lifecycle, stage hysteresis, CS resolution machine, hero metrics, risk flags, allowedMoves, card hysteresis, budget drop order, retry/timeout policy, explicit serial chain with pending (§4), SessionState written to ctx.storage per step and rehydrated in the constructor, serializeAttachment on the socket, answers/move_answers/decisions/moments persistence, per-hash seek cache with the moves-only path (§3), reconnect history, 402 alert |
unit tests with stubbed env.AI cover masks, rep-turn-only locks (client paraphrase does not tick), concern open/clear/expire with since refresh, resolution transitions, recompute over stored answers with zero Jev calls and a correct drift count (engine-level), budget degradation, cache hit/miss on state change, judging-hit + move-miss issuing a moves-only request and no full request; 10 utterance messages sent in a burst produce decisions in order with each state_json containing the previous decision's persisted facts; a forced eviction (or 30 s pause) mid-replay resumes with the same persisted facts and no reset; replay of onboarding-strong locks ≥ 5 must-say items and persists next_step_agreed; onboarding-weak ends lower; reload mid-replay restores the timeline |
| 6 | M1a | Dashboard (§8.1) as Worker static assets, browser-owned clock, all M1 panels for both scenarios (weights editor is M2), Range-enabled GET /api/calls/:id/audio, mark-moment form, telemetry with drift and credit alerts |
the four samples replay end to end at 1×/5×/20× with pause/seek; call-3339895706 replays in sync with the <audio> element from the R2 copy; seek to 5:00 in Safari and Chrome starts playback within 1 s; "catching up" badge appears at 20× and clears; hero panels differ by scenario; every rendered rep-facing string is a RepFacingText (type-level test); a mark writes moments + marks + audit_log |
| 8a | M1a | Eval harness scripts/eval.mjs (service-token auth against the deployed Worker) with eval/labelled/*.json (25-40 hand-written cases per scenario incl. the negative and injection cases), L4 shapes, L6 budget; eval_runs rows; reports only |
L1 pass rate reported for both scenarios (no gate yet); L4 holds on the four samples; L6: every measured usage.input_tokens ≤ 12k; report JSON committed under eval/reports/ |
| 9 | M1a | Docs on Pages ct-copilot-docs behind Access: this document, a 5-minute demo script with utterance numbers, the eval report, and a measured cost/latency report from the first replays (tokens, latency, credit-balance delta reconciled against Σ usage, effective $/Mtok, whether output tokens are charged, Σ chars / Σ usage.input_tokens written back as token_budget.chars_per_token) |
published Pages URL that answers with the Access login page when unauthenticated; the demo script reproduces on the deployed Worker; §12's estimates replaced by measured numbers; no host, path, key or credential in any published file (grep in CI) |
| 7 | M1b | Upload path: PUT /api/calls/upload (raw body streamed to R2, container sniffed, §7) → nova-3 REST from the R2 object → speaker map (heuristic + Jev Choice + toggle) → stitch → redact → D1; audio-synced replay |
data/samples/call-3339895706.mp3 uploads, transcribes, maps speakers at ≥ 0.80 or flags for the toggle, and replays in sync with the <audio> element; a non-audio file and a mislabelled file are rejected with 415 before any Deepgram call; a WAV above 25 MB is rejected with the convert-to-mp3 message; neurons per minute, the m4a/wav proxy result and the body-size result for a 30-40 min file are recorded |
| 8b | M1b | Promotion gate: L5 stability runs; policy status='evaluated' only from a passing run; --bootstrap removed |
policy v1 reaches L1 ≥ 90% on both scenarios (M1b exit target; hard gate from M2); L5 per-question std reported; a deliberately broken draft (a true/false swap) fails and cannot be marked evaluated; --bootstrap no longer exists |
| 10 (stretch) | M1b | Rewrite path behind the flag, default off | when on: a candidate quoting a rate not on the call is dropped with invents_fact ≥ 0.50 recorded; a candidate containing "guaranteed" is dropped by the denylist without a Jev call |
Out of M1: KV, Queues, corpus-wide scoring, admin console beyond the mark form, the weights/thresholds editor and recomputed UI (M2, §8.1), per-call authorisation beyond the Access gate (§4), CT-DB pre_call join, Vectorize, live audio.
M2: curation console + corpus scoring (2-3 weeks)
score-calls Queue consumer running the same step() with a batch bank (no phrasing::, no rewrites) over all usable Onboarding (1,895) and CS (1,226) calls after the DPA answer, plus one episode request per concern (resolved, response_specific, response_compliant, handling_quality, exemplar_candidate, corpus §2 Layer 2); measured rate limit and consumer CPU recorded; dead-letter queue. Admin console on Pages: review queue by information value, moment view with model/acceptable/avoid, "use as example", "promote to playbook line" with source_moment_id and approved_by; weights editor with corpus recompute; bank editor via LLM-drafted wording gated by the harness; enforced publish gate with diff view; KV mirror; audit. Jev scenario Choice with disagreement flags; speaker-role Choice for unlabelled calls; pii_present sweep; L2/L3/calibration; per-question thresholds; outcome join via pd_ct_id to broker_accounts.verified_at and trade_bookings (read access needed, §13 Q8) and logistic-regression weights after Stevan's review; pre_call block in state from clients.*; rewrite path on for CS if compliance agrees; nightly harness on model drift; R2 retention rules.
M3: live audio
Transport decision first (Aircall media stream, browser softphone capture, per-leg audio; stt §5.3); nova-3 WebSocket from the DO with interim_results, endpointing=300, utterance_end_ms=1000, KeepAlive; a complete noul + silence timer for interim commits (prior-art §2.5); finals become the same utterance messages into the unchanged engine; latency budget end-of-utterance → card ≤ 1.5 s; rep-side overlay with alert-fatigue controls (one card per turn, cooldowns, no repeat within 4 minutes); observer mode for managers; rep feedback (ack_card, thumbs) feeding labels.
12. Cost & latency budget
Unit prices: Jev $0.042/M input tokens pass-through, output free, +5% on credit purchases (verified, guide §4.6, §4.10; today's probe: 400 input tokens, keySource: Unified); nova-3 ~473 neurons/min ≈ $0.31/h (verified); a Haiku-class rewrite generator ~$1/$5 per Mtok (ASSUMPTION); D1/R2/DO/Pages inside the Workers Paid plan at this scale est.. Call shape est.: a usable call averages ~7 min (229 h / 1,895 Onboarding), ~55 decision points, ~9.6k tokens per request at the measured 3.5 chars/token (§6.2; the earlier 8.4k assumed 4 chars/token and was ~15% low).
| Item | Per replayed call | Per 1,000 historical calls |
|---|---|---|
| Jev decisions (55 × 9.6k) | 528k tokens ≈ $0.022, $0.023 with fee | ≈ $23 |
moves-only re-asks after a playbook edit (55 × 2.1k), M2 |
— | ≈ $5 per re-scored 1,000 |
Batch bank (no phrasing::, 55 × 8.3k), M2 corpus scoring |
— | ≈ $19 |
| Episode requests (~4 × 1.5k) | — | ≈ $0.25 |
| Rewrite verification, if enabled (~8 × 1.5k Jev + ~1k generator) | ≈ $0.01 | ≈ $10, only if on |
| nova-3 (uploads only, 7 min) | ≈ $0.036 | ≈ $36 for 117 h, not needed for the corpus |
| PII noul sweep (M2, ~70 utterances × 400) | — | ≈ $1.2 |
| Total | ≈ $0.023 (transcript) / ≈ $0.06 (upload) | ≈ $20-24 Jev-only |
Full usable corpus (3,121 calls, 356 h) ≈ $60-75 in Jev credits. Harness run (~150 cases × 5 for L5) ≈ $0.30. Reconcile on day one: note the credit balance, replay the four samples, compare the balance delta with Σ usage.input_tokens × $0.042/M to learn the effective rate and whether output tokens are charged on Cloudflare (guide §4.10); M1 deliverable 9 publishes the result.
Latency per decision point: Jev 0.44 s over REST measured today (400-550 ms range), ~0.8 s via the binding including cold start, steady-state binding latency unmeasured (ASSUMPTION ~0.5 s); stitching is pre-computed; decision code < 5 ms; D1 write < 20 ms est.; WebSocket push < 50 ms est.. Target p95 utterance-to-card ≤ 1.5 s, gated in L6. At 1× replay a decision every 2-6 s leaves the loop idle; at 20× the DO's explicit serial chain (§4) falls behind by up to ~40 s over a 7-minute call, shown as "catching up"; second passes hit the seek cache and are instant. Upload transcription: 8.6 s for 9:46, so est. ~30 s for a 30-minute file. Batch scoring of 1,000 calls at 8 concurrent requests and ~0.6 s each ≈ 1.2 h; if the Workers AI Text Generation default of 300 rpm applies to Jev (undocumented, guide §4.9) ≈ 3 h. Rewrites arrive 1-3 s after the card and are never awaited.
13. Risks, unknowns, and the questions only Stevan can answer
Risks
- Messy speech vs literal criteria. The v0 questions were written against tidy phrases; Aircall fragments are garbled. Expect the first L1 pass well under the reference's 100% and expect
uncertainto be common. Mitigation: stitching quality, criterionexamplesdrawn from real turns (which is whatmomentsandmarksexist for), L5 on real turns, nova-3 re-transcription where audio exists. - No version pinning on Cloudflare (
guide §8). A silentjev-1.14moves every threshold. Mitigation:model_expectedon the policy,modelon every answer, drift alert in the session, L6 fails and opens a drift run; the harness costs ~$0.30. - 32k context and undocumented overflow. Design point 9.6k at 3.5 chars/token, soft 11k, hard 12k with a drop order; L6 hard-fails any measured request over 12k. Adding
pre_call, value-extraction Choices or more moves in M2 must go through the same gate. - Data-processing status of a third-party model on Workers AI (Q1). Everything beyond the four samples waits on it; M1 is written to be complete on those four.
- Draft must-say list and
[VERIFY]lines. Half the onboarding hero depends on wording compliance has not approved; the loader will refuse to ship the draft as is, so the red-pen sitting is on the critical path (Q3). - Diarisation on mono narrowband audio degrades on overlap (94% agreement on one call is one data point); the Jev role Choice and the toggle are the mitigations; per-leg audio in M3 removes the problem.
- Unmeasured Workers AI rate limit, steady-state binding latency and Queue-consumer CPU for Jev: all measured by M1 deliverable 9 before M2 is sized.
- LLM-assigned categories and coach scores:
categoryis treated as a label to verify, and coach quotes as hints, never gold. - Recompute approximation: threshold edits that feed back into state are not exactly replayable from stored answers; the drift count keeps this honest and a paid re-ask is one click.
- Choice option order is part of the question (
guide §3.5): a playbook diff that reorders moves changesplaybook_hashand must pass the harness like any other edit.
Unknowns to test in the first two days: whether the binding honours {gateway:{id}} and per-call log suppression (else REST); the binding request shape for nova-3 (REST is verified); the proxy body-size ceiling for a 30-40 min file; whether audio/mp4 and audio/wav survive the nova-3 proxy (only audio/mpeg is verified); real usage.input_tokens for the full bank and the chars-per-token ratio (3.5 assumed from the reference bank); whether mip_opt_out changes nova-3 billing; the cold-start profile of the DO.
Questions only Stevan can answer
- DPA / processor: may pseudonymised, client-derived transcript text be sent to
typesafe/jevthrough Workers AI (and audio to Deepgram via Cloudflare) under CT's data-processing obligations, and who signs that off? Until answered, M1 runs on the four exported samples only. Also: retention period forraw/audio and transcripts in R2. - Corpus selection for M1+: once Q1 is cleared, which calls (a rule such as top/bottom deciles by
call_value_scoreper category, or a hand-picked list)? - Must-say wording and applicability: the compliance-approved way to explain safeguarding, who holds funds, how CT is paid, the FSCS question, the £5k minimum, binding bookings; which
[VERIFY]lines inbrief §5are true as written; which items apply per client type (personal vs corporate) and per stream (PFX/CFX/SAR/IL); which payment partners reps may name. - Rewrite path: acceptable at all in onboarding, CS only, or neither? Default stays off.
- Hosting: keep the Worker on
workers.devbehind Access with Stevan's email for M1, or acurrencytransfer.comsubdomain from the start (needed before the SPA moves to Pages in M2)? Who else gets Access (RMs, managers)? M1's tenancy model is full-corpus read for every Access identity, which is fine while that identity is Stevan alone; extending Access needs either therep_reffilter on the call routes (§4) or an explicit decision, recorded here, that all Access users read the whole corpus. - CS hero: resolution state alone, or resolution plus the numeric health bar (this design shows both, health secondary)?
- Recording disclosure: must calls be announced as recorded, and is that itself a must-say item?
- Outcome join (M2): read access to CT
broker_accounts/trade_bookings(replica, nightly extract, orct-sql-skill queries by hand), and the activation window N (days) for "activated within N days of the call". client_typeper call: available from Pipedrive/CT for the checklist's applicability rules, or inferred from the transcript in M1?- Token and key housekeeping: the
cfut_token pasted today is active and reaches Jev with credits (verified);~/.config/jev/cloudflare.envalready holds a token. Should the new one replace it, and should it be an Account token rather than a user token for the deployed Worker (guide §4.5)? The token is unrelated to host access. Host access for the call-coach server is documented only in the unpublished ops runbook; because access details for that host were pasted into planning contexts before this document was scrubbed, the keys it references should be audited and rotated, and the rotation date recorded in the runbook.
14. Sources
/home/stevan/dev/jev/docs/jev-guide.md(§1 model, §2 contract and derived types, §3 semantics and thresholds, §4 Cloudflare integration: gateway, billing, envelope, errors, rate limits, terms, §5 patterns, §6 cookbooks, §7 jaggedness and prompt rules, §8 gotchas)/home/stevan/dev/jev/docs/research/reference-copilot-dissection.md(§1 state, §2 questions, §3 maths and recompute limit, §4 facts and objection lifecycle, §5 playbook and gating, §6 personalize and verifier pass-through, §7 protocol and process model, §8 video sync, §9 eval, §10 transfer notes)/home/stevan/dev/jev/docs/research/stt-options.md(§1, §3.1, §3.3, §5, "Live verification": mono Aircall audio, nova-3 REST result, speaker agreement)/home/stevan/dev/jev/docs/research/prior-art.md(§2.1decide.jshysteresis, §2.3 commitment-risk noul, §2.9 curation prior art, §4.4 cookbooks, §5 primeline calibration, §6 UI vocabulary, §8 ranked ideas)/home/stevan/dev/jev/docs/research/ct-domain-brief.md(§1 CT and journey, §2.3 must-say, §2.4 objections, §4 question banks, §4.6 composite drafts, §5 playbooks with[VERIFY], §6 questions)/home/stevan/dev/jev/docs/research/corpus-and-curation.md(§0 constraints, §1 ingestion and PII, §2 rubric layers, §3 curation actions and queue ordering, §4 playbook selection, §5 storage limits and schema, §6 harness, §8 open questions)/home/stevan/dev/jev/docs/architecture/proposal-poc-fastest.md,proposal-production-shaped.md,proposal-data-first.mdand the three judges' verdicts (task input, 2026-09-24)/home/stevan/dev/jev/docs/vendor/typesafe/model-jaggedness__jev-1.13.md,concepts__how-to-build-with-system-one.md,cookbooks__consistency_noul_cookbook.md,cookbooks__sde_cascade.md,cookbooks__classification_using_confidence.md/home/stevan/dev/jev/reference/call-coach/README.md,config/rubrics.json(onboarding and customer_service dimensions and checks),scripts/download_call_audio.py/home/stevan/dev/jev/reference/jev-sales-copilot/copilot/{engine.py,constants.py,personalize.py,server.py}via the dissection/home/stevan/dev/jev/data/samples/onboarding-strong-3339895706.{txt,json},call-3339895706.mp3,call-3339895706.nova3.json,customer-service-{strong,weak}-*.txt,onboarding-weak-3485591407.txt/home/stevan/dev/jev/data/call-coach/{call_records,call_turns,call_features}.jsonl.gz(row shapes;pd_ct_idcoverage measured by data-first)/home/stevan/dev/jev/src/index.ts,/home/stevan/dev/jev/wrangler.jsonc,/home/stevan/dev/jev/scripts/jev.mjs- Verified facts supplied with the task (2026-09-24) and re-verified in this session: REST Jev call 0.44 s,
jev-1.13.0,result.resultnesting,keySource: Unified; token active; Aircall recording download;eval/cf-parity.json(380,577 input tokens over 48 requests, the basis of the 3.5 chars/token divisor)
15. Decision log
| # | Decision | Alternatives considered | Why |
|---|---|---|---|
| D1 | poc-fastest is the M1 spine; production-shaped's guarantees and data-first's engine are grafted on | build production-shaped's M1 (RBAC, KV, Queue, Workflow eval before replay); build data-first's M1 (batch scorer and a 60-moment marking sitting before replay) | Both alternatives invert or overload the milestone Stevan defined ("step by step", replay first); two of three judges ranked poc-fastest first on fit and deliverability. The grafts cost days, not weeks |
| D2 | Pure step() engine shared by the DO now and the Queue consumer in M2 |
port the reference CallSession as a stateful class inside the DO |
Makes corpus scoring configuration rather than a rewrite; unit-testable without a DO; the judges' most-cited structural idea (data-first §2) |
| D3 | Three hashes: bank_hash for judging answers, playbook_hash for move/phrasing answers, policy_hash for decisions; state_hash on every answer |
poc-fastest's single bank hash including the playbook; production-shaped's bank/policy split without separating move answers | A playbook line edit is the main curation action; folding it into the bank hash invalidated every stored answer. state_hash fixes the seek cache serving answers to a different state. M1 has no playbook editing, so the split has no M1 consumer; it is M2 infrastructure paid for early (§2) |
| D4 | Budget with drop order (phrasing beyond top-4 → all phrasing → window 8 → skip), soft 11k, hard 12k | hard reject at 12k (poc-fastest); assert at 16k (data-first) | A 120-word window can trip a hard reject; 16k leaves less headroom under an undocumented overflow. Dropping whole fixed-option questions is hash-safe; a per-turn shortlist inside next_move is not |
| D5 | said_* locks on rep turns only, with an L1 negative case |
window nouls lock on any turn (poc-fastest, production-shaped) | A client paraphrase would tick a compliance item; the hero metric would record a must-say as delivered when it was not |
| D6 | Six single-condition verifier nouls + code denylist, verification failure = drop; safeguarding/binding/fees moves never rewritable | one compound makes_promise_or_prediction noul (poc-fastest, production-shaped); four nouls (data-first); reference pass-through on error |
guide §7: one condition per noul, decompose and aggregate with max; a regulated line must never reach a rep unverified; scheme implication and partner naming were missing from all three |
| D7 | Browser owns the clock; DO only evaluates; "catching up" badge | DO setTimeout chain with alarm watchdog (production-shaped); DO sleep loop (data-first) |
Timers conflict with WebSocket Hibernation and drift at 10-20×; the audio element gives true audio sync for uploads; live STT sends the same messages in M3 |
| D8 | Typed RepFacingText, loader hard-rejects [VERIFY] and unapproved lines, red-pen sitting on the M1 critical path |
yellow "unverified draft" badge (poc-fastest); status='draft' lines loaded and rendered (data-first) |
A badge is a convention, not a control; the compliance judge ranked this the decisive difference |
| D9 | Dashboard as Worker static assets in M1, docs on Pages; SPA moves to Pages under a CT hostname in M2 | Pages SPA + Worker API from day one (production-shaped, data-first) | One origin = one Access application and no cross-site cookie/WebSocket work; Stevan's decision names deliverables, which do go to Pages; the move is a hosting question (§13 Q5), not an architecture change |
| D10 | Curation seeded in M1 (engine-emitted moments + one mark form), full loop in M2 | curation entirely M2 (poc-fastest, production-shaped); full loop in M1 (data-first) | Honours Stevan's curation decision inside M1 at near-zero cost without depending on a marking sitting or on the DPA answer |
| D11 | Hand-written L1 labels + synthetic edge cases, 90% reported in M1a, M1b exit target, hard gate from M2; coach quotes are hints only | bootstrap labels from coach quotes[].reason, gate 85% (poc-fastest); labels only from Jev-tagged moments (data-first); 90% as an M1 hard gate (this document's first draft) |
Two LLMs agreeing is not a gate; Jev-tagged-only labels never contain false negatives; 90% matches the reference and the other proposals; as an M1 hard gate it is an open-ended criteria-tuning loop (risk 1), not a build task |
| D12 | [REP] token; coach prose stays in R2 raw/; numeric coach fields only in D1 |
keep the rep's real name in Jev traffic (production-shaped); coach_json in D1 (all three) |
Employee names are personal data; coach summaries name clients; a "pseudonymised" store must be pseudonymised |
| D13 | DPA gate explicit: four samples + synthetic cases until Q1 is answered; "already processed by OpenAI" is not a basis | proceed on ~20-100 calls on the assumption it is fine (all three, to varying degrees) | guide §4.11 says do not send private data until resolved; the plan must not stall on a gate it raises itself, so M1 is complete on the four samples |
| D14 | Recompute declared exact for weights, approximate for feedback thresholds, with a drift count | "recompute is free" without caveat (all three) | Stored answers were given against the old known_facts/checklist_done; honesty in the UI beats a silent approximation |
| D15 | D1 batch size = floor(100 / columns) via a helper | "≤ 12 rows" (all three) | 12 rows × 11 columns = 132 bound parameters > D1's 100 |
| D16 | 402 code 2021 = operator alert, no retry; SDK-mirroring retry policy otherwise | generic 4xx fail-fast (poc-fastest) | An empty credit balance mid-replay must not look like a bad request |
| D17 | Binding with {gateway:{id}} first, REST fallback decided on day 1 |
REST only (jev.mjs shape); binding only | The binding needs no token secret, but per-request log suppression on it is unverified (guide §4.3); REST is verified at 0.44 s |
| D18 | Stage Choices get none; objection type read only when client_objecting fires; parent-bucket fallback below 0.40 |
draft stage Choices without none; type shown whenever the Choice names one |
A Choice always crowns a winner (guide §3.1); the classification-with-confidence cookbook lifts unsure cases by reporting the parent |
| D19 | Hero metrics noul-first; Scores only in the secondary health bar; no closing probability, no "activation likelihood" bar | brief §4.6's activation-likelihood composite as a bar | Stevan's decision; nouls calibrate best (prior-art §5); a probability-shaped bar invites the wrong reading |
| D20 | Mono recordings, diarise + code/Jev/admin speaker map; historical corpus needs no STT | ask Stevan whether recordings are dual-channel (all three) | Already answered by the live verification in stt-options.md; Aircall's own participant_type is the better speaker source for history |
| D21 | Explicit serial chain in the DO + SessionState stored per step + socket attachment |
rely on "DO messages are processed serially" (all three proposals); keep state in memory between messages | Input gates only close during storage ops, so awaiting Jev lets utterance events interleave at 5-20×; hibernation evicts in-memory state after a few idle seconds. Both were correctness bugs in the text, not in any proposal's code |
| D22 | Access service token for scripts; Pages docs behind Access; no hosts, paths or credentials in published docs; Jev REST token only as a wrangler secret |
email-only Access policy (scripts blocked by the login page); public docs site; access details inline in this document (first draft) | Non-interactive clients need Service Auth; the docs name calls and rep tokens; a design document that lists a working privileged path into a PII store must not be publishable |
| D23 | M1a / M1b cut line with an audited bootstrap publish; eval harness reports in M1a, gate in M1b | single 8-10 day M1 with a 90% L1 hard gate and deliverables that summed past the budget; dashboard listed before the harness its policy load depended on | The DO pins only a published row; without a bypass the replay demo was blocked on the harness (a dependency inversion in the table); the bootstrap is refused once a passing run exists and removed in M1b |
| D24 | set_weights/recomputed panel moved to M2; messages kept behind the M1 debug drawer |
required M1 dashboard feature (first draft) | Not in Stevan's milestone-1 ask; its payoff is corpus calibration, which is M2 (§6.5, §9); on four calls the dials have little to show |
| D25 | Token estimator divisor is policy data (3.5 chars/token, measured), re-derived from the first replays | 4 chars/token constant (all three proposals and the first draft) | eval/cf-parity.json gives 3.52; a divisor of 4 under-reports by ~14% and would let the estimator pass requests that breach the 12k hard gate it exists to enforce |