#Corpus + curation layer for the CurrencyTransfer call copilot

Research memo, 2026-09-24. Scope: how to turn hundreds of mixed-quality historical call transcripts into (a) an admin-curated definition of "good", (b) grounded playbook lines, objection responses and must-say checklists for the live copilot, and (c) a scorer that can evaluate any call, historical or live, against that standard. Everything runs on Cloudflare with Jev via the Workers AI binding (env.AI.run('typesafe/jev', …)).

Sources are cited inline as file paths (local vendor docs under /home/stevan/dev/jev/docs/vendor/…, the reference app under /home/stevan/dev/jev/reference/jev-sales-copilot/…) or URLs. Anything not backed by a source is marked ASSUMPTION. Numbers from Cloudflare limits pages were fetched live on 2026-09-24 and are not in the local vendor folder; re-check before relying on them.

#0. Constraints that shape every decision

Design consequences used throughout:

1. Jev classifies and ranks; code stores, counts, weights, and decides. Every number an admin can tune is a coefficient in code/DB, never a prompt rewrite (patterns__composite-scoring.md; reference copilot/constants.py header: "Change a coefficient here, not a prompt").
2. Small, filtered state. Retrieve and filter in code before asking (model-jaggedness__jev-1.13.md §"Large state full of irrelevant detail"). Never send a whole 40-minute call as one state.
3. Store raw answers. Re-weighting, re-thresholding and re-ranking must be free (reference: set_weights → "recomputed with no extra Jev calls"; cookbooks__classifying_rag_passages.md: "re-routing every passage costs no API calls").
4. Pin the model version (jev-1.13.0, not jev-latest) and log response.model (models.md "Aliases"; reference constants.py MODEL).

#1. Transcript ingestion + normalisation

#What we are ingesting

ASSUMPTION: the "call coach" server holds one file per call (text or JSON), some with speaker labels and timestamps, some without; audio may or may not be retrievable. The scenario (onboarding vs customer success) is probably not labelled. Everything below tolerates missing timestamps and missing speaker labels; it does not tolerate missing text.

#Target canonical format

One JSON document per call, compatible with the reference loader (reference/copilot/replay.py accepts {"id","title","utterances":[{"t","speaker","text"}]}), extended with what the corpus layer needs:

{
  "call_id": "cc-2026-03-14-0173",
  "scenario": "onboarding",             // onboarding | customer_success | unknown (Jev-assigned, admin-confirmable)
  "scenario_confidence": 0.93,
  "source": {"system": "call-coach", "path": "…", "audio_r2_key": "raw/audio/…wav|null"},
  "rep_id": "rm-07", "client_ref": "ct-acct-hash-…",   // pseudonymised
  "recorded_at": "2026-03-14T10:02:00Z",
  "transcription": {"engine": "call-coach-original|@cf/deepgram/nova-3", "diarised": true},
  "utterances": [
    {"i": 0, "t": 0.0, "t_end": 4.1, "speaker": "rep",    "text": "…", "words": 18, "pii_redactions": []},
    {"i": 1, "t": 4.3, "t_end": 9.0, "speaker": "client", "text": "…", "words": 27, "pii_redactions": ["IBAN"]}
  ],
  "outcome": {"first_trade_within_30d": true, "traded_gbp": 12000}   // joined from CT DB, optional
}

#Decision 1a — where transcripts come from

Recommendation: C. Keep transcription.engine on every call so rubric scores can be segmented by engine; if call-coach transcripts score systematically differently, that is a transcription artefact, not a coaching signal.

#Decision 1b — speaker labels (rep vs client)

Diarisation gives speaker 0/1/…, not roles. Options:

When the source already has role labels, keep them but still run B and flag disagreements.

#Decision 1c — utterance segmentation

• Unit = one diarised turn (utterances: true from nova-3, or one line from call-coach). Merge consecutive same-speaker turns separated by < 1.0 s (ASSUMPTION threshold; tune by eye on 20 calls).
• Split any turn > 120 words at sentence boundaries into sub-utterances that share t. The reference flags a rep monologue at 70 words (constants.py REP_MONOLOGUE_WORDS); keeping utterances short also keeps per-utterance questions literal and cheap.
• Missing timestamps: synthesise as the reference does (replay.py: max(3.0, words/2.5) seconds).
• Segments for the rubric = sliding windows of 12 utterances (matches reference RECENT_WINDOW = 12), plus episodes = code-detected spans starting at a client concern and ending at the next client turn that accepts/moves on (see §2). Episodes are the unit admins tag.

#Decision 1d — PII handling

CurrencyTransfer calls will contain IBANs, sort codes, account numbers, card numbers, addresses, DOBs, passport numbers, beneficiary names and transfer amounts. ASSUMPTION: UK GDPR applies and the company has a lawful basis to process recordings for training/QA; confirm with compliance before anything leaves the call-coach box.

Recommendation: A → B → C in that order; only pseudonymised text is ever sent to Jev, stored in D1, or shown in the admin UI. Raw audio/transcripts live in a private R2 bucket with a retention rule (§5). Jev's own posture ("not trained on customer requests", models.md) is reassuring, but on Workers AI the model is listed as "Third-party" — check Cloudflare's third-party-model terms before sending even pseudonymised text (open question §8).

#Decision 1e — scenario classification (onboarding vs customer success)

One Choice per call, state = first 15 utterances + last 5 (not the whole call: context rot): {onboarding: "a first or early call: explaining how the service works, account setup, verification/KYC documents, first transfer walkthrough", customer_success: "an existing client: reviewing recent transfers, rate check-in, complaint, retention, upsell", other: "…"}. Gate on confidence like cookbooks__classification_using_confidence.md (≥ 0.9 accept, else unknown → admin). Store scenario on the call; the rubric and playbook are keyed by it.

#2. Jev-based call-quality rubric

#Structure: three layers, all atomic, composite in code

The reference app scores closing probability. We score call quality against the CT standard, which is a different composite but the same machinery (patterns__composite-scoring.md: "break the judgment into independent dimensions, score each one separately, and combine them with weights you control in code").

Layer 1 — utterance level (one request per utterance, speculative fan-out, all questions every turn, masked by speaker in code as the reference does; patterns__fan-out.md).

State per request (mirrors reference; ~5–8k tokens):

{
  "call_facts": {"scenario": "onboarding", "minute": 6.2, "talk_ratio_rep": 0.58,
                 "known_facts": ["fee_structure_explained"], "open_concern": {"topic": "rate", "since_utt": 41}},
  "recent_transcript": [ …last 12 utterances… ],
  "latest_utterance": {"i": 47, "speaker": "client", "text": "…"}
}

Questions (illustrative wording; the concrete must-say list is ASSUMPTION until Stevan / compliance supply it — see §8):

• Client-turn Nouls: client_raises_concern, client_accepts ("acknowledges the rep's last answer as satisfactory"), client_confused, client_disengaging, client_asks_to_proceed (buying/booking signal), client_asks_price (rate/fee question).
• Client-turn Choice concern_topic over {rate, safety_of_funds, settlement_timing, fees, process_or_documents, competitor, none} with what / not_for / examples per option (concepts__how-to-build-with-system-one.md "define contrastive Choice criteria"). Asked speculatively; only opens a concern when client_raises_concern ≥ 0.6 (reference pattern: objection_type is speculative, prospect_objecting gates it).
• Rep-turn Nouls: rep_answered_the_concern ("does latest_utterance.text directly address the concern in call_facts.open_concern?"), rep_asked_open_question, rep_monologuing (also computed in code from word count), rep_quoted_specific_figure ("states a rate, fee, margin, or timeframe as a concrete number"), rep_overpromised ("guarantees a future rate or outcome nobody can guarantee" — ASSUMPTION this is a compliance no-no), rep_confirmed_understanding.
• Rep-turn Score answer_quality (3–4 concrete levels: "ignores or deflects the concern" / "answers vaguely without specifics" / "answers with a specific, correct-sounding explanation" / "answers with specifics and checks the client is satisfied"). Only meaningful when a concern is open — code masks it otherwise.
• Window Nouls for must-say facts (durable, persisted in code once ≥ 0.7 like the reference's FACT_PERSIST_THRESHOLD): e.g. safeguarding_explained, fee_structure_explained, settlement_timeline_explained, rate_lock_explained, next_step_agreed. Each references recent_transcript and lists true/false examples.
• Scores over the window: rapport, client_engagement, clarity_of_rep (levels as concrete situations with signals, as in reference constants.py).
• Choice stage for the scenario (onboarding: opening, needs_discovery, service_explanation, verification_docs, first_transfer_walkthrough, concern_handling, next_steps; CS: its own list).

Layer 2 — episode level (code + one request per episode). Code cuts an episode at each opened concern: from the client utterance that opened it to the first later client turn with client_accepts ≥ 0.6 or client_asks_to_proceed ≥ 0.6, capped at 8 utterances (reference OBJECTION_MAX_AGE_UTTERANCES = 8). Then one request per episode with state = {concern: {topic, client_text}, rep_response_turns: [...], resolution_turn: {...}} and questions:

• resolved Noul ("did the client end the exchange satisfied, judging by resolution_turn?")
• response_specific, response_accurate_sounding, response_compliant Nouls (the last with the compliance rules pasted into the question as structured fields, not the state — api.md "instructions can be an object … refer to data fields by name in backticks")
• handling_quality Score (4 levels, concrete)
• exemplar_candidate Noul ("would a sales trainer show this exchange to new reps as a model of how to handle this concern?") — deliberately a trainer's judgment, kept separate from the atomic ones so the admin can see when they disagree.

Episodes are what admins tag ("great rate objection handling"), what feeds the playbook (§4), and what the eval harness regression-tests (§6).

Layer 3 — call level (code only, no Jev). Composite:

quality = Σ_k w_k · x_k
  x from Layer 1: mean(answer_quality | concern open), mean(client_engagement), 1 − mean(client_confused),
                  1 − mean(rep_overpromised), 1 − talk_ratio_penalty(code), fact_coverage (code: fraction of
                  scenario must-say facts persisted), episodes_resolved / episodes_opened (code)
  x from Layer 2: mean(handling_quality), share of episodes with response_compliant ≥ 0.7

Weights start as the admin's guess (a table in D1, versioned) and are tuned on labelled calls (§3, §6). Everything is recomputable from stored raw answers.

#Decision 2a — rubric granularity

#Decision 2b — flagging candidate "good" calls and "bad" moments

• Candidate good calls: quality ≥ P80 of the corpus and (if outcomes are joined) a good outcome, and at least one episode with exemplar_candidate ≥ 0.7. Ranked list to the admin, top 30 first. Outcome join: onboarding → client traded within N days; CS → client traded again / no churn. ASSUMPTION: feasible via the CT production DB (the ct-sql skill's clients / trade_bookings tables) using a hashed client reference; the call-coach data must carry something joinable (email, account id, RM + date).
• Bad moments: any episode with resolved < 0.3, or rep_overpromised ≥ 0.7, or response_compliant < 0.3, or client confused ≥ 0.7 twice within a window. Surface as a second list ("what to avoid"); admins tag these too — negative exemplars become not_for text and false criteria examples (§3).
• Uncertain band: treat Noul 0.30–0.70 as "uncertain", don't auto-flag on it (cookbooks__consistency_noul_cookbook.md), and Choice top-probability < 0.60 as "uncertain" (cookbooks__consistency_choice_cookbook.md). Uncertain episodes are the most valuable for admin review: they are where the criteria are ambiguous.

#Cost of scoring the backlog

ASSUMPTION: 500 calls × 250 utterances = 125k utterance requests at ~7k tokens = 875M tokens ≈ $37 at the direct-API price, plus ~2k episode requests (negligible). At the direct API's 1,200 req/min this is ~2 h wall-clock if run flat out; the Workers AI binding has no typesafe/jev rate-limit row and the Text Generation default is 300 req/min, which makes it ~7 h — the real limit is unknown until measured (build step 1, §8). Run it as a Cloudflare Queue consumer or a Workflow with concurrency ≤ 10 and idempotent per-utterance writes. Windows overlap, so window-level facts could be asked every 4th utterance in batch mode to cut cost ~3×; the live path keeps every-turn.

#3. Admin curation loop (no retraining; state + instructions + criteria + weights)

#What the admin sees

1. Ranked calls (composite, outcome, scenario, engine, RM) with a per-call timeline identical to the live dashboard (reuse the reference's eval/timelines.json shape per call).
2. Episode browser: every concern episode with topic, Jev's handling_quality, resolved, exemplar_candidate, and the transcript excerpt.
3. Semantic find inside one call: "where does the rep explain safeguarding?" → Choice over utterance ids (≤ 255 per request, so a 300-utterance call is split into two windows) plus an exists Noul (cookbooks__semantic_find.md — exactly this recipe).

#What the admin can do, and what it changes

#Decision 3a — how exemplars steer Jev

Recommendation: A + C. Judging questions get 2–4 examples per option drawn from tagged exemplars (admin ticks "use as criterion example"); suggestion text comes from C.

#Decision 3b — criteria authoring workflow

#Decision 3c — ranking the admin's review queue

Order the queue by information value, not by score: (1) episodes in the uncertain band, (2) top-composite calls without any admin label yet, (3) bottom-composite calls, (4) disagreements between exemplar_candidate (trainer's-eye Noul) and the atomic composite. Reviewing ~60 episodes gives the first labelled set (the autoresearch cookbook reads 60 examples per proposal round; the reference's integration test uses 19 labelled utterances / 56 checks and reaches 100%).

#4. From curated exemplars to the live copilot

#Playbook shape

Same JSON contract as the reference (reference/data/playbook.json: moves[] with id, title, what, not_for, phrasings[]), keyed by scenario and extended:

{
  "version": 12, "scenario": "onboarding",
  "moves": [
    {"id": "handle_rate_concern_with_transparency", "title": "Explain how the rate is set",
     "what": "The client says the rate looks worse than the bank/competitor or asks why it moved. Explain margin vs interbank plainly and offer the live quote.",
     "not_for": "Fee questions, safety questions, or a client who has already accepted the rate.",
     "topic": "rate",
     "phrasings": [
       {"id": "p1", "text": "…admin-approved line…", "source_exemplar": "cc-…#41-44"},
       {"id": "p2", "text": "…", "source_exemplar": null}
     ],
     "must_say": ["rate_is_indicative_until_booked"]}
  ],
  "checklists": {"onboarding": ["safeguarding_explained", "fee_structure_explained", "settlement_timeline_explained", "next_step_agreed"]}
}

Playbook phrasings are admin-approved text, ideally lifted verbatim from a tagged exemplar (with the [NAME]/[AMOUNT] tokens cleaned). Jev never writes them; it only picks (README.md: "Jev picks; code shows the text"). Optional generative tailoring stays exactly as the reference does it: off the critical path, verified by Jev invents_fact / on_move Nouls, dropped if stale (reference/copilot/personalize.py, constants.py VERIFY_QUESTIONS).

#Decision 4a — how the live path selects a move and a line

Recommendation: B for the live path; C for the admin/exemplar search tool and for a future "show me a real example" panel. Reasoning on the 32k limit: a curated playbook of 2 scenarios × 15 moves × 4 lines ≈ 120 lines ≈ 4k tokens, plus the ~5k-token window state and ~30 questions ≈ 15k — half the Workers AI budget, no retrieval layer needed. Vectorize's limits (20M vectors, 1536 dims, 10 KiB metadata, topK 50 with metadata — Vectorize limits) are far beyond what exemplars need; the reason to defer it is complexity, not capacity.

#Decision 4b — must-say checklist on the dashboard

Code-only over persisted facts: for the current scenario, show each checklist item as unsaid / said (locked ≥ 0.7) / uncertain (0.3–0.7); suggest the move whose must_say covers the oldest unsaid item once the stage is past discovery. This is the "what they should remember to say" bar from the brief and needs no extra Jev questions beyond the window Nouls in §2.

#Decision 4c — objection-specific responses

Per (scenario, topic) keep 3–6 approved lines. When a concern opens with concern_topic.confidence ≥ 0.4 (reference OBJECTION_TYPE_MIN_CONFIDENCE), the next_move shortlist is restricted to that topic's moves, and the phrasing Choice ranks its lines against recent_transcript. Below confidence, show the generic "acknowledge and clarify the concern" move (reference: "listening…" gate at NEXT_MOVE_MIN_CONFIDENCE = 0.35).

#5. Storage on Cloudflare

Limits fetched 2026-09-24 from the Cloudflare docs (not in the local vendor folder): D1 — 10 GB/database (Paid), 2 MB max row, 100 bound params/query, 100 KB statement, 1,000 queries/invocation; KV — 25 MiB value, 1 write/s per key; eventually consistent, changes visible in "up to 60 seconds or more"; R2 — 5 TiB objects, unlimited bucket size; Durable Objects — 10 GB SQLite storage per object, soft 1,000 req/s per object, 32 MiB WebSocket messages (received messages only); Vectorize — 20M vectors/index, 1536 dims, 10 KiB metadata, topK 50 with metadata.

Size estimate (ASSUMPTION 500 calls, 250 utterances each): utterances ≈ 125k rows × ~0.5 KB = 60 MB; raw Jev answers per utterance (37 questions with probabilities) ≈ 3 KB → 375 MB; episodes ≈ 2–5k rows. All well inside one D1 database.

Schema sketch (D1):

CREATE TABLE calls(call_id TEXT PRIMARY KEY, scenario TEXT, scenario_conf REAL, rep_id TEXT, client_ref TEXT,
  recorded_at TEXT, engine TEXT, utterances INTEGER, duration_s REAL, outcome_json TEXT, raw_r2_key TEXT);
CREATE TABLE utterances(call_id TEXT, i INTEGER, t REAL, t_end REAL, speaker TEXT, text TEXT, words INTEGER,
  pii_json TEXT, PRIMARY KEY(call_id, i));
CREATE TABLE answers(call_id TEXT, i INTEGER, rubric_hash TEXT, model TEXT, answers_json TEXT, input_tokens INTEGER,
  latency_ms INTEGER, PRIMARY KEY(call_id, i, rubric_hash));
CREATE TABLE episodes(episode_id TEXT PRIMARY KEY, call_id TEXT, topic TEXT, start_i INTEGER, end_i INTEGER,
  rubric_hash TEXT, answers_json TEXT);
CREATE TABLE call_scores(call_id TEXT, rubric_version INTEGER, quality REAL, features_json TEXT, PRIMARY KEY(call_id, rubric_version));
CREATE TABLE question_bank_versions(version INTEGER PRIMARY KEY, rubric_hash TEXT, questions_json TEXT, created_by TEXT, created_at TEXT, note TEXT);
CREATE TABLE rubric_versions(version INTEGER PRIMARY KEY, weights_json TEXT, thresholds_json TEXT, question_bank_version INTEGER, published INTEGER);
CREATE TABLE playbook_versions(version INTEGER, scenario TEXT, playbook_json TEXT, published INTEGER, PRIMARY KEY(version, scenario));
CREATE TABLE exemplars(exemplar_id TEXT PRIMARY KEY, episode_id TEXT, label TEXT, quality TEXT, note TEXT, use_as_example INTEGER,
  admin TEXT, created_at TEXT);
CREATE TABLE labels(label_id TEXT PRIMARY KEY, call_id TEXT, i INTEGER, question_id TEXT, gold_json TEXT, admin TEXT, created_at TEXT);
CREATE TABLE eval_runs(run_id TEXT PRIMARY KEY, rubric_version INTEGER, model TEXT, pass_rate REAL, spearman REAL, cost_usd REAL, r2_key TEXT, created_at TEXT);

Gotchas: D1's 100 bound parameters per statement means batch inserts of ≤ 12 utterance rows per statement (8 columns each); use batch() for the rest. answers_json must stay under 2 MB (it will be ~3 KB). KV rubric:current is a pointer, never the mutable document.

#6. Evaluation harness (regression-testing question-bank changes)

Modelled on the reference's two real-API test layers: tests/test_integration_jev.py (19 hand-labelled utterances, 56 threshold checks, ≥ 90 % pass gate, writes eval/integration_report.json) and tests/test_e2e_replay.py (replays whole calls, writes eval/timelines.json, asserts shapes: good ends ≥ 0.70, bad ≤ 0.40, mixed dips ≤ 0.30 then ends ≥ 0.65, p95 latency < 2 s, cost < $0.05/call, model pinned, zero errors).

#What is under test

Every promotion of a question_bank_version / rubric_version / model version, plus a nightly run. Inputs are frozen: the golden set is a snapshot (call ids + utterance ids + gold labels + expected shapes), exported from D1 to R2 and a small copy committed to the repo so CI can run without DB access.

#Decision 6a — test layers

#Decision 6b — where the harness runs

Cache every (state, questions, model) → answers like the cookbooks' JsonCache keyed by rubric_hash so re-running an unchanged question set costs nothing and so that a changed criterion "busts the cache instead of silently serving a stale answer" (cookbooks__consistency_noul_cookbook.md _rubric_fingerprint).

#Decision 6c — how a change is promoted

draft → scored on golden (L1–L6) → admin reviews diff (which cases flipped, which exemplars moved) → publish (D1 published=1, KV pointer) → backfill the corpus in the background (Queue) → old answersrows kept under theirrubric_hash for rollback. Model upgrades follow the same path with model as the changed variable (models.md: "pin that version's ID … move to the new one on your own schedule").

#7. Recommended build order

1. Ingest 50 calls (mixed engines), pseudonymise, classify scenario + speaker roles, store in D1/R2. Verify the 32k budget empirically on Workers AI with a 12-utterance window + ~35 questions, and measure the effective Workers AI rate limit for typesafe/jev (no documented row; the Text Generation default is 300 req/min).
2. Rubric v1 with placeholder must-say facts; score the 50; admin reviews the review queue (§3c) for one afternoon → ~60 labelled episodes, ~100 labelled utterances.
3. Harness L1/L2/L4 on those labels; iterate the question bank until ≥ 90 %.
4. Playbook v1 from tagged exemplars (verbatim lines, cleaned) → live copilot POC on a recorded call (replay mode from the reference) using a DO per session.
5. Score the full backlog; add outcome joins; L3; tune weights.
6. Later: Vectorize exemplar search, autoresearch feature discovery against outcomes, nova-3 live streaming.

#8. Open questions for Stevan

1. Call-coach server: file format(s), presence of audio, timestamps, speaker labels, and whether calls can be joined to CT accounts (email / account id / RM + date) for outcomes.
2. The actual must-say list per scenario (safeguarding wording, fee disclosure, rate indicative-until-booked, settlement timelines, FCA-required statements). The rubric's must-say Nouls and the dashboard checklist are placeholders until this exists.
3. Compliance rules that define "overpromising" / non-compliant phrasing, so the response_compliant question can carry them as structured criteria.
4. Lawful basis and retention for recordings; whether pseudonymised transcripts may go to a third-party model on Workers AI (Cloudflare marks typesafe/jev "Third-party").
5. Who the admins are and how many hours/week they can label; that sets whether option 3b-B (LLM-drafted questions, human-gated) or 3b-C (autoresearch) is realistic in the first quarter.
6. Whether the Workers AI Jev path really enforces 32k rather than TypeSafe's 64k/32k split; this decides whether phrasing Choices for all moves can ride on every request or only the shortlisted ones.
7. Whether D1 FTS5 (available, including fts5vocab) is good enough as the exemplar search (no Vectorize) for the first version.
8. The actual Workers AI rate limit for typesafe/jev: the limits page has no row for it and the Text Generation default is 300 req/min vs 1,200 req/min on the direct API; measure in build step 1, since it sets backlog wall-clock (~7 h vs ~2 h).