Reference dissection: jev-sales-copilot (moritzkremb)
Source tree: /home/stevan/dev/jev/reference/jev-sales-copilot (all paths below are relative to it unless absolute).
Every source, data, test, eval and doc file was read in full (uv.lock skipped). Line references are file:line and were verified with grep -n against the checked-out tree on 2026-09-24.
Where a claim comes from a reported run (not something I executed), it is labelled reported. Where I am inferring, it is labelled assumption.
0. Reading map
| File | Lines | Role |
|---|---|---|
copilot/constants.py |
375 | Model pin, the 37 questions, weights, priors, every threshold, LLM settings, the two verification questions. The whole "policy" of the app. |
copilot/engine.py |
484 | CallSession: state window, feature extraction, speaker masking, composite + EMA, fact persistence, objection state machine, coaching gate, talk ratio / pace, sensitivity, recompute-without-Jev. |
copilot/jev_client.py |
132 | Async wrapper over typesafe_sdk.AsyncTypeSafeClient.system_one; SDK answer objects to plain dicts; cost accounting. |
copilot/personalize.py |
221 | Optional tailored line: trigger rule, Haiku prompt, output cleaning, Jev verification. |
copilot/server.py |
308 | FastAPI: static UI, /api/*, one /ws per tab; replay task, live utterances, weight changes, background personalize task. |
copilot/replay.py |
68 | Load calls from JSON or rep:/prospect: text. |
copilot/cli.py |
116 | Headless replay to terminal / eval/timelines.json. |
data/playbook.json |
170 | 15 moves: id, title, what, not_for, 3 phrasings. |
data/calls/{good,bad,mixed}.json |
46-50 | Hand-written 38-42 turn calls for a field-service SaaS ("Relay"). |
data/calls/cloudtalk.json |
58 | Real recorded CloudTalk BANT call, 48 turns, t/t_end aligned to YouTube uovWCGCl2-s. |
static/index.html |
552 | Single-page UI (Chart.js), WebSocket client, Web Speech API, YouTube IFrame sync. |
tests/*.py |
999 | Unit (mock Jev), server (mock), personalize (mock LLM), integration (real API), e2e (real API). |
eval/integration_report.json |
820 | Last integration run: 56/56 checks, per-case observed answers. |
eval/timelines.json |
7878 | Last e2e run: per-utterance probability, signals, sensitivity, latency, tokens for the three calls. |
README.md, DEMO.md |
218, 107 | Architecture write-up; 3-5 min demo script with utterance numbers. |
Runtime: Python >= 3.13, typesafe-sdk>=0.6.0, FastAPI, uvicorn, httpx, pyyaml>=6.0.3, websockets>=17.1 (pyproject.toml:6-13). No build step. run.sh resolves keys env > .env > ~/.clawdia-secrets/.env and never prints them (run.sh:11-32); binds 127.0.0.1 only (run.sh:43).
The one-paragraph version of how it works: each utterance triggers exactly one Jev request carrying a compact JSON state (12-turn window + a few call facts) and 37 typed questions; Jev never generates text; all numbers (closing probability, talk ratio, fact persistence, objection state) are arithmetic in engine.py over Jev's typed answers; the coaching card is a Jev Choice over a hand-written 15-move playbook and code shows the pre-written text; an optional Haiku call rewrites one line off the critical path and Jev is then asked whether that line invents a fact. Pinned to jev-1.13.0 (constants.py:16).
1. Per-utterance state sent to Jev
Built by CallSession.build_state() (engine.py:185-200), called once per ingested utterance (engine.py:203-213).
Exact shape:
{
"call_facts": {
"duration_min": 5.7,
"talk_ratio_rep": 0.54,
"stage_history": ["opening", "discovery", "pitch_demo"],
"known_facts": ["budget_discussed", "pain_identified"],
"objection": {"open": true, "type": "price"}
},
"recent_transcript": [
{"t": "05:12", "speaker": "rep", "text": "..."},
"... up to 12 entries, the newest is the latest utterance ..."
],
"latest_utterance": {"t": "05:45", "speaker": "prospect", "text": "..."}
}
Field by field:
| Field | Source | Notes |
|---|---|---|
call_facts.duration_min |
round(latest.t / 60, 1) (engine.py:190) |
t is seconds since call start; for live/typed input with no t, engine estimates prev.t + max(2.0, words/2.5) (engine.py:205). |
call_facts.talk_ratio_rep |
talk_ratio(self.utterances) over the whole call, rounded 2dp (engine.py:187,191) |
Rep words / total words; 0.5 when no words (engine.py:30-40). |
call_facts.stage_history |
_compress(stage_history)[-6:] (engine.py:192, engine.py:478-484) |
Consecutive duplicates collapsed, last 6 distinct runs. Stage comes from Jev's previous stage answers. |
call_facts.known_facts |
sorted(self.facts) (engine.py:193) |
Names only, no values or timestamps. Can include competitor_mentioned (see section 4). |
call_facts.objection |
{"open": true, "type": type or "unknown"} or {"open": false} (engine.py:194-196) |
|
recent_transcript |
self.utterances[-12:] (engine.py:186, RECENT_WINDOW = 12 at constants.py:24) |
Includes the latest utterance as its last element. Each entry is Utterance.to_state() = {"t": "MM:SS", "speaker", "text"} (engine.py:122-123, format_ts at engine.py:105-107). |
latest_utterance |
Same to_state() of the newest utterance (engine.py:199) |
Duplicated deliberately so questions can point at latest_utterance.text by name. |
Speaker handling: speakers are the literal strings "rep" and "prospect" (engine.py:116-119). The server normalises anything starting with p to prospect, else rep (server.py:288). Replay lowercases the JSON's speaker (replay.py:50). There is no diarization anywhere: replay/video calls carry speaker labels in the JSON, live mic uses a manual toggle (section 8).
Size: reported 7,189-8,121 input tokens per request across the e2e run (eval/timelines.json, computed over all rows), ~7.4-7.5k in the integration cases (eval/integration_report.json:29,72,...). README puts it at "~7,500 input tokens per request, ~$0.0003". Almost all of that is the question bank (the 15 speculative phrasing Choices alone add ~1,900 tokens per README "Limitations"), not the state.
The unit test pins the shape: exactly 12 transcript entries once the call is longer than that, and call_facts keys are exactly {duration_min, talk_ratio_rep, stage_history, known_facts, objection} (tests/test_engine_unit.py:283-292).
What Jev does not see: the full call, any fact values (only fact names), any timing between turns beyond the MM:SS stamps, the closing probability, the previous coaching move, or any product/company knowledge. All domain knowledge lives in the question texts and the playbook option descriptions.
2. The question bank, verbatim
QUESTIONS is a dict at constants.py:117-258 plus a loop that appends 15 phrasing::<move> Choices (constants.py:261-268). Total = 16 Nouls + 3 Scores + 3 Choices + 15 phrasing Choices = 37 questions, all sent in every request (speculative fan-out). Ids are never sent to the model; each instructions is complete and names the state path it reads (enforced by tests/test_engine_unit.py:332-343).
Helper _noul(question, true, false) builds {"type": "noul", "instructions": {"question": ...}, "criteria": {"true": ..., "false": ...}}; criteria are omitted when neither is given (constants.py:110-114). _PROSPECT = "spoken by the prospect (\latest_utterance.speaker` is prospect)" (constants.py:107`).
2a. Prospect-turn Nouls (masked to 0 in code when speaker is rep; PROSPECT_ONLY_SIGNALS at constants.py:271-280)
buying_signal (constants.py:119-123)
- question: Does
latest_utterance.text, spoken by the prospect (latest_utterance.speakeris prospect), express interest in moving forward with the product? - true: Asks how to get started, about onboarding, implementation, contract terms, or rollout; says the product would solve their problem; asks to see it or try it; positive reaction such as 'that's exactly what we need'.
- false: Neutral information, a question about how a feature works, a concern, or small talk.
commitment (constants.py:124-128)
- question: Does
latest_utterance.text, spoken by the prospect (...), explicitly commit to buying, signing, or starting a paid engagement? - true: 'Let's do it', 'send over the contract', 'we're in', 'sign us up', agreeing to a purchase or paid pilot.
- false: Interest without commitment, agreeing only to a demo or follow-up call, or any concern.
next_step_agreed (constants.py:129-133)
- question: Does
latest_utterance.text, spoken by the prospect (...), agree to a concrete next step such as a demo, trial, follow-up meeting, proposal review, or introducing a colleague? - true: 'Thursday works', 'yes, send the proposal', 'let's get my boss on the next call', 'sure, set up the demo'.
- false: Vague 'send me some info', 'we'll think about it', a question, or a concern.
prospect_objecting (constants.py:134-138)
- question: Does
latest_utterance.text, spoken by the prospect (...), raise a concern, hesitation, or pushback about buying the product? - true: Says it is too expensive, not the right time, needs someone else's approval, doubts they need it or that it will work, prefers a competitor, or asks a skeptical 'why would we...' question.
- false: Neutral questions about how something works, sharing information about their situation, agreement, or small talk.
prospect_accepts (constants.py:139-143)
- question: Does
latest_utterance.text, spoken by the prospect (...), accept, agree with, or acknowledge as reasonable what the rep just said? - true: 'That makes sense', 'ok, fair enough', 'that would work for us', 'good point', 'I see'.
- false: Disagreement, a new concern, a neutral question, or unrelated information.
prospect_disengaging (constants.py:144-148)
- question: Does
latest_utterance.text, spoken by the prospect (...), signal that they want to end or shorten the call, or brush the rep off? - true: 'Just send me some info', 'I need to run', 'we'll think about it and get back to you', 'not really a priority right now', one-word dismissive answers.
- false: Engaged questions, detailed answers about their situation, or agreeing to next steps.
buyer_confused (constants.py:149-153)
- question: Does
latest_utterance.text, spoken by the prospect (...), show that they are confused or did not follow what the rep said? - true: Asks the rep to repeat or clarify, asks what a term means, says 'I'm not sure I follow', or asks a question the rep's previous turn already addressed.
- false: A clear question about something new, a statement, or agreement.
prospect_asked_price (constants.py:154-156) - no criteria; note the question text does not name the speaker, and because prospect_asked_price is not in SIGNAL_WEIGHTS the engine never masks or reads it (see 2g). A port that promotes it to a weighted signal must add the speaker condition to the question text or rely on the PROSPECT_ONLY_SIGNALS mask.
- question: Does
latest_utterance.textask what the product costs, its price, plans, or pricing model?
2b. Rep-turn Nouls (masked to 0 when speaker is prospect; REP_ONLY_SIGNALS at constants.py:281)
rep_discovery_question (constants.py:158-162)
- question: Is
latest_utterance.text, spoken by the rep (latest_utterance.speakeris rep), an open-ended question inviting the prospect to describe their situation, current process, problems, goals, or how they make decisions? - true: 'Walk me through how scheduling works today', 'what's the most painful part of that?', 'what happens when a job runs late?', 'who else is involved in a decision like this?'
- false: A yes/no question, a statement, describing product features, quoting a price, or proposing a meeting.
rep_pitching (constants.py:163-167)
- question: Is
latest_utterance.text, spoken by the rep (...), describing product features, capabilities, or benefits without asking the prospect a question? - true: Lists what the product does, how it works, its integrations, or benefits, and ends without a question to the prospect.
- false: Asks the prospect a question, talks about the prospect's situation, quotes a price, or proposes a next step.
2c. Either-speaker Nouls (not masked)
next_step_proposed (constants.py:168-172)
- question: Does
latest_utterance.textpropose a concrete next step such as a demo, a trial or pilot, a follow-up meeting with a day or time, sending a proposal, or bringing in another stakeholder? - true: 'Can we set up a demo Thursday?', 'I'll send the proposal tonight', 'let's get your ops lead on the next call', 'we could start with a two-week pilot'.
- false: Describing features, asking about the prospect's situation, or a vague 'let's stay in touch'.
competitor_mentioned (constants.py:194-198)
- question: Does
latest_utterance.textmention another vendor, another software product, or an alternative the prospect uses or is evaluating instead of the rep's product? - true: Names a competing product or company, 'we're also looking at two other vendors', 'we currently use a tool from another company'.
- false: Mentions only the rep's product, spreadsheets, or manual processes without naming an alternative product or vendor.
2d. Transcript-window Nouls (durable facts; read recent_transcript, so they can fire on a rep turn)
pain_identified (constants.py:174-178)
- question: In
recent_transcript, has the prospect described a specific problem, frustration, inefficiency, or cost they are currently experiencing in their business? - true: Prospect mentions things like double-booked technicians, hours of manual scheduling, customers complaining, missed jobs, wasted drive time, or paying for a tool that does not work.
- false: Only the rep talks about problems in general, or the prospect says things are fine.
budget_discussed (constants.py:179-183)
- question: In
recent_transcript, has the prospect said anything about their budget, what they currently pay, an acceptable price range, or whether money is available for this? - true: Prospect states a budget, a range, what the current tool or manual work costs them, or that funds are or are not approved.
- false: Only the rep quoted prices, or budget has not come up.
decision_maker_identified (constants.py:184-188)
- question: In
recent_transcript, has the prospect stated who makes the purchase decision (themselves, a named person, or a role) or how the decision gets approved? - true: 'I can sign off on this', 'the owner has the final say', 'my boss and I decide together', 'it needs to go through our GM'.
- false: Nothing said about who decides or approves.
timeline_known (constants.py:189-193)
- question: In
recent_transcript, has the prospect stated when they want to decide, buy, or have a solution in place? - true: 'Before the summer season', 'this quarter', 'we need something by March', 'not until next year'.
- false: No timing for a decision or rollout has been mentioned by the prospect.
2e. Scores (3 levels each; criteria is a list of {summary, signals[]})
rapport (constants.py:200-211)
- question: How would you describe the tone between rep and prospect in
recent_transcript? - focus: Judge the human tone, not whether the deal is going well.
- levels: 0 "Tense or cold" [Curt answers; Irritation, sarcasm, or impatience; Interrupting or talking past each other] / 1 "Neutral and businesslike" [Polite and factual; No personal remarks or humor; Efficient question-and-answer] / 2 "Warm and friendly" [Humor or personal remarks; Mutual acknowledgement ('great question', 'I appreciate that'); Relaxed, collaborative language]
engagement (constants.py:212-223)
- question: How engaged is the prospect, judging by the prospect's turns in
recent_transcript? - focus: Look only at the prospect's turns.
- levels: 0 "Disengaged" [One-word or one-line answers; Deflecting ('just send info'); Trying to end the call] / 1 "Passive" [Answers what is asked; Volunteers nothing extra; No questions back] / 2 "Active" [Asks questions about the product or process; Gives detail about their situation; Volunteers information or ideas]
urgency (constants.py:224-234)
- question: How urgent is the prospect's need to solve their problem, judging by the prospect's turns in
recent_transcript? - levels: 0 "No urgency" [Exploring or curious; 'Someday', 'eventually'; No deadline or pressure mentioned] / 1 "Moderate" [The problem hurts and is mentioned as ongoing; Wants to fix it but no deadline] / 2 "Pressing" [A deadline, season, mandate, or contract renewal is named; Acute pain: losing customers or staff now]
2f. Choices
stage (constants.py:236-242; options = STAGES at constants.py:46-79, each {what, not_for})
- question: Which stage of a sales call best describes the last few turns of
recent_transcript, weightinglatest_utterancemost? opening- what: Greetings, small talk, agenda setting, or thanking for the time. Nothing about the prospect's business problem yet. not_for: Turns that describe the prospect's process or problems.discovery- what: The rep asks about, or the prospect describes, their current process, problems, team, or goals. not_for: Turns about budget, decision makers, or timelines (that is qualification), or about the product itself.qualification- what: Talk about budget, who decides, approval process, or the timeline for deciding or implementing. not_for: Describing the problem itself, or the product's features.pitch_demo- what: The rep explains or shows what the product does, its features, or how it would work for the prospect. not_for: Turns where the prospect is pushing back or asking about price.objection_handling- what: The prospect has voiced a concern, doubt, or pushback (about price, timing, need, trust, or a competitor) and the rep is responding to it. not_for: Neutral questions about how the product works.pricing- what: Discussion of the price, plans, tiers, discounts, or contract terms of the product being sold. not_for: The prospect's own budget in general (qualification).closing- what: The rep asks for the commitment, or the prospect says yes to buying, signing, or starting. not_for: Scheduling a demo or follow-up (that is next_steps).next_steps- what: Agreeing on what happens after the call: a demo, trial, follow-up meeting, sending a proposal, or introducing other people. not_for: Asking for the purchase itself.
objection_type (constants.py:243-249; options = OBJECTION_TYPES at constants.py:93-101, each {what} only)
- question: If the prospect's most recent turns in
recent_transcriptexpress a concern or pushback about buying, which kind of concern is it? Choosenoneif no concern has been voiced. price: The cost, price, fees, or value for money is too high or hard to justify.timing: Not the right time: too busy, mid-project, revisit next quarter or after the season.authority: The prospect cannot decide alone; someone else (boss, owner, committee, IT) must approve.need: The prospect doubts they have the problem, or thinks the current way is fine.trust: Doubts about the vendor, the product working for them, implementation risk, or disruption.competitor: Prefers, uses, or is evaluating another vendor or tool instead.none: The prospect has not voiced any concern or pushback about buying.
next_move (constants.py:250-257; options = every playbook move's {what, not_for})
- question: Which coaching move should the rep make next, given
recent_transcript,latest_utterance, andcall_facts(call_facts.known_factslists what is already established,call_facts.objectiondescribes any open concern)? - focus: Pick the move whose
whatmatches the current moment; respect each move'snot_for.
phrasing::<move_id> x15 (constants.py:261-268)
- question: If the rep's next move is '
', which of these lines fits best as the rep's next sentence given recent_transcript? - options:
p0,p1,p2= the move's three phrasings verbatim.
2g. Which questions are actually consumed
| Consumer | Questions |
|---|---|
Weighted into probability (SIGNAL_WEIGHTS, constants.py:293-317) |
commitment, buying_signal, prospect_accepts, next_step_agreed, pain_identified, budget_discussed, decision_maker_identified, timeline_known, engagement, rapport, urgency, prospect_disengaging, buyer_confused, competitor_mentioned, rep_pitching, rep_discovery_question |
Objection state machine (engine.py:273-297) |
prospect_objecting, objection_type, prospect_accepts, buying_signal, commitment, next_step_agreed, prospect_disengaging |
Stage (engine.py:323-329) |
stage |
Coaching (engine.py:299-320) |
next_move, phrasing::* |
Fact persistence (engine.py:257-271) |
the 5 fact-kind signals + competitor_mentioned |
| Asked but only read by the integration test | prospect_asked_price, next_step_proposed (neither is in SIGNAL_WEIGHTS, so extract_features and the UI signals map never touch them; they exist as labels for tests/test_integration_jev.py:63-178) |
Masking is implemented in extract_features (engine.py:234-237; the adjacent 238-239 is the separate persisted-fact clamp to 1.0) and _update_facts (engine.py:264-265), and only for signals that are in self.weights, so prospect_asked_price being listed in PROSPECT_ONLY_SIGNALS has no runtime effect. Unit test: tests/test_engine_unit.py:156-166 and :146-153.
3. Closing-probability maths
All in engine.py; the formula is documented in the comment block at constants.py:284-292.
3a. Composite (composite(), engine.py:61-81)
inst = clamp( STAGE_PRIOR[stage] + sum_i contribution_i , P_MIN, P_MAX )
contribution_i =
w_i * x_i for kind in {noul, fact, code}, x in [0,1]
w_i * (x_i - 0.5) * 2 for kind == score, x = score/(levels-1) (middle level = 0)
Missing features are skipped; an unknown stage falls back to INITIAL_PROBABILITY (0.30) as the prior (engine.py:70,73-79). normalize_score is score/(levels-1), clamped (engine.py:55-58); for a 3-level score, level 1 contributes 0, level 0 contributes -w, level 2 contributes +w.
STAGE_PRIOR (constants.py:82-91): opening 0.25, discovery 0.30, qualification 0.35, pitch_demo 0.35, objection_handling 0.30, pricing 0.38, closing 0.45, next_steps 0.42.
SIGNAL_WEIGHTS (constants.py:293-317), 18 entries:
| signal | kind | weight | label |
|---|---|---|---|
| commitment | noul | +0.15 | Prospect committed |
| buying_signal | noul | +0.10 | Buying signal |
| prospect_accepts | noul | +0.03 | Prospect agrees with rep |
| next_step_agreed | fact | +0.09 | Next step agreed |
| pain_identified | fact | +0.05 | Pain identified |
| budget_discussed | fact | +0.04 | Budget discussed |
| decision_maker_identified | fact | +0.04 | Decision maker identified |
| timeline_known | fact | +0.04 | Timeline known |
| engagement | score | +0.08 | Buyer engagement |
| rapport | score | +0.04 | Rapport |
| urgency | score | +0.05 | Urgency |
| objection_open | code | -0.14 | Objection open |
| prospect_disengaging | noul | -0.16 | Prospect disengaging |
| buyer_confused | noul | -0.05 | Buyer confused |
| competitor_mentioned | noul | -0.04 | Competitor mentioned |
| rep_pitching | noul | -0.04 | Rep pitching features |
| rep_talking_too_much | code | -0.10 | Rep talking too much |
| rep_discovery_question | noul | +0.03 | Rep asked discovery question |
Sums: positives 0.74 (of which 0.17 are score weights that can also go negative), negatives -0.53. tests/test_engine_unit.py:90-100 asserts positives in [0.5, 1.2], negatives in [-0.8, -0.3], and that every noul/fact/score weight has a matching question. Theoretical range of inst before clamping: 0.25 - 0.53 - 0.17 = -0.45 up to 0.45 + 0.74 = 1.19, so the clamps P_MIN, P_MAX = 0.03, 0.95 (constants.py:320) are reachable; tests/test_engine_unit.py:81-87 tests them.
The kind values mean: noul = raw Jev probability this turn; fact = raw Jev probability until persisted, then hard 1.0 for the rest of the call (engine.py:242-243); code = computed in code (objection_open, rep_talking_too_much); score = centred level.
3b. EMA (ema(), engine.py:51-52; applied at engine.py:335)
p_t = EMA_ALPHA * inst_t + (1 - EMA_ALPHA) * p_{t-1}, EMA_ALPHA = 0.40 (constants.py:319)
p_0 = INITIAL_PROBABILITY = 0.30 (constants.py:321)
The EMA is per utterance, not per second: a fast back-and-forth moves the number faster than a slow one. Half-life is ln(0.5)/ln(0.6) = 1.36 utterances, so a single transient spike (e.g. commitment 1.0) decays to ~36% of its effect two turns later (tests/test_engine_unit.py:240-250 asserts the fade). No clamp is applied to p_t itself; it stays inside [0.03, 0.95] because inst is clamped and p_0 is inside the range.
3c. "What moved the number" (sensitivity(), engine.py:84-93)
Per-signal contribution deltas between the previous step and this one (stage prior counted as a signal labelled "Call stage"); deltas below 0.0005 are dropped; sorted by |delta| desc, then delta desc, then name (deterministic on ties); top SENSITIVITY_TOP_N = 3 (constants.py:336). Called in _apply with the previous step's contributions (engine.py:336,360). Rendered as +/- pts in the UI (index.html:402-405). Note it is attribution of instantaneous contribution changes, not of the EMA'd probability change; the two can disagree in sign on a given turn.
3d. Talk ratio, pace, monologue
talk_ratio(utterances)= rep words / all words over the entire call (engine.py:30-40); word count is whitespace split (engine.py:22-23).rep_talking_too_muchfeature (engine.py:248-255):ratio_part = clamp((ratio - 0.65) / (0.90 - 0.65))but only once the call has >=TALK_RATIO_MIN_UTTERANCES = 6utterances (constants.py:26), else 0;monologue = speaker == rep and words > REP_MONOLOGUE_WORDS (70)(constants.py:27); feature =max(ratio_part, 0.6 if monologue else 0). So one 71+ word rep turn is worth 0.6 * -0.10 = -6 pts instantly even in a balanced call. ThresholdsTALK_RATIO_WARN = 0.65,TALK_RATIO_MAX = 0.90(constants.py:334-335). Test:tests/test_engine_unit.py:227-237.pace_wpm(utterances, speaker, now_t)(engine.py:43-48): words by that speaker in the trailingPACE_WINDOW_S = 60s window, divided bymin(60, max(now - first_t, 10))seconds. Display only (not weighted).talkblock in every snapshot:ratio_rep, rep_words, prospect_words, pace_wpm_rep, pace_wpm_prospect, rep_monologue(engine.py:338-345). UI colours the ratio red above 0.65 (index.html:397).
3e. Recompute without Jev (set_weights + recompute, engine.py:364-382)
Every Step stores raw answers and the derived features (engine.py:136-152). recompute() re-runs composite + EMA over step.features with the current weights and stage priors and rewrites instant, probability, contributions, sensitivity on each step. Because it reuses step.features (not answers), weights and stage priors are retunable offline, but thresholds are not: fact persistence (0.70), objection open/clear (0.60), speaker masking and the talk-ratio curve are already baked into the stored features. Changing FACT_PERSIST_THRESHOLD would require re-deriving features from answers (the data is there; the code path is not). tests/test_engine_unit.py:253-268 checks zero Jev calls on recompute; tests/test_server.py:48-75 checks the WebSocket round-trip.
3f. Reported outcomes (eval/timelines.json, real jev-1.13.0 run)
| call | final p | min | max | facts persisted | coaching gated | latency mean / p95 | input tok | output tok | cost |
|---|---|---|---|---|---|---|---|---|---|
| good (42 utt) | 0.905 | 0.258 | 0.908 | pain, budget, DM, timeline, next step | 5/42 | 345 / 522 ms | 330,109 | 58,646 | $0.0139 |
| bad (38 utt) | 0.075 | 0.075 | 0.304 | competitor, budget | 4/38 | 391 / 532 ms | 298,774 | 53,177 | $0.0125 |
| mixed (42 utt) | 0.805 | 0.155 | 0.882 | pain, budget, DM, timeline, next step | 2/42 | 382 / 525 ms | 330,793 | 58,698 | $0.0139 |
Per request: ~7.2-8.1k input tokens, ~$0.0003 at $0.042/Mtok input, output free (constants.py:17; /home/stevan/dev/jev/docs/vendor/typesafe/models.md:13,18). First request of a process is slow (876 ms on good, eval/timelines.json row 0) because of connection setup.
4. Fact persistence and objection lifecycle
4a. Facts (_update_facts, engine.py:257-271)
- Candidates: every
SIGNAL_WEIGHTSentry withkind == "fact":next_step_agreed,pain_identified,budget_discussed,decision_maker_identified,timeline_known. - Rule: if not already persisted, and the answer is a noul with
noul >= FACT_PERSIST_THRESHOLD (0.70)(constants.py:326), and (if the signal is prospect-only, which onlynext_step_agreedis) the speaker isprospect, thenfacts[name] = utterance.index. - The four window facts read
recent_transcript, so they can lock on a rep turn (e.g.timeline_knownlocked on rep turn #16 in thegoodrun, where the rep paraphrased "there's a real clock on this"). - Facts never un-lock and there is no per-fact value (no "budget = GBP 1,000/month"), only the name and the index it locked at.
- Extra:
competitor_mentioned >= 0.70is also stored infacts(engine.py:269-271) so it appears inknown_factsand gets the lock icon, but since its weightkindisnoulit is not clamped to 1.0 in features; its probability contribution stays transient. - Effects of a persisted fact: feature forced to 1.0 (
engine.py:242-243), UI lock icon (index.html:391), name fed back to Jev viacall_facts.known_factssonext_move'snot_fortexts ("Calls where budget has already been discussed") can take effect. Test:tests/test_engine_unit.py:124-143.
Observed in the integration report: because BASE_CONTEXT already contains a pain description (tests/test_integration_jev.py:28-33), pain_identified reads 0.82-0.99 on nearly every case regardless of the latest utterance (eval/integration_report.json, e.g. :42, :218, :808). Window facts are context-driven, which is the point, but it also means one strong statement 11 turns ago still counts.
4b. Objection state (Objection dataclass engine.py:126-132; _update_objection engine.py:273-297)
State: open, type, confidence, since (utterance index), probability.
On a prospect turn:
- If
prospect_objecting >= OBJECTION_OPEN_THRESHOLD (0.60):- if not already open, open it with
since = this index; (if already open,sinceis not refreshed); probability = prospect_objecting;- if
objection_type.choice != "none"andobjection_type.confidence >= OBJECTION_TYPE_MIN_CONFIDENCE (0.40), set/overwritetypeandconfidence; else if no type yet,type = "unknown"; return(no expiry check this turn).
- if not already open, open it with
- Else, if open and
max(prospect_accepts, buying_signal, commitment, next_step_agreed) >= OBJECTION_CLEAR_THRESHOLD (0.60)andprospect_disengaging < 0.60, clear (freshObjection()), return. The disengaging guard is the "sure, okay, I have to run is not acceptance" rule (engine.py:292-294, tested attests/test_engine_unit.py:169-191).
On any turn that reached the end: if open and index - since > OBJECTION_MAX_AGE_UTTERANCES (8), clear (engine.py:296-297).
Two behaviours worth knowing because they differ from the README's wording ("or 8 utterances without re-raising"):
- Expiry is measured from the first open, not the last re-raise.
sinceis only set when the objection transitions closed->open. A prospect who keeps objecting every other turn still has the card silently cleared on the first non-objecting turn aftersince + 8. Confirmed in the reported timelines:badopens at #9 (price), re-typed at #11 (need) and #13 (competitor), and is cleared on rep turn #18 (18 - 9 = 9 > 8) although nobody accepted anything; it re-opens at #19 astiming.mixedlikewise opens at #1, re-typed at #5 and #7, expires on rep turn #10. - The type is overwritten on every objecting turn that passes the 0.40 confidence gate, so the card shows the latest concern, not the first one, and the
sinceshown ("raised at utterance #N") refers to the first.
objection_type is asked speculatively every turn and will name a type for non-objections: in the integration report a neutral "What does something like this cost?" gets price (0.68) (eval/integration_report.json:775), and "just send me some info... I need to run" gets timing (0.96) (:435). That is why the type is only consulted when prospect_objecting fires, and why prospect_objecting and not objection_type != none is the gate.
Effect on the number: objection_open feature = 1.0 while open, weight -0.14 (engine.py:244), i.e. -14 pts of instantaneous estimate, -5.6 pts on the EMA in the first turn.
5. The 15-move playbook, gating, and not_for
data/playbook.json (_comment at line 3: "Jev PICKS a move (Choice over the ids); code shows the text. what/not_for become the Choice option descriptions Jev sees."). Loaded once at import (constants.py:35-41); served raw at GET /api/playbook (server.py:75-77) and in the hello message.
| # | id (playbook.json line) |
title | what | not_for | phrasings |
|---|---|---|---|---|---|
| 1 | dig_into_the_problem (5) |
Ask a discovery question | Early in the call, or the prospect's situation, current process, and problems are not yet understood. Ask an open question that gets the prospect describing how things work today. | Situations where the pain is already clearly described, or where the prospect just raised a concern that needs a response first. | "Walk me through how that works for your team today, start to finish." / "What's the most painful part of the current process for you?" / "What made you take this call now, of all times?" |
| 2 | quantify_the_pain (16) |
Quantify the pain | The prospect described a problem in words, but its cost in hours, money, lost jobs, or missed revenue has not been pinned down. Ask for numbers so the value is concrete. | Calls where no problem has been mentioned yet, or where the cost has already been stated. | "Roughly how many hours a week does that eat up across the team?" / "If you had to put a dollar figure on that problem per month, what would it be?" / "How many jobs a month would you say slip because of that?" |
| 3 | ask_about_budget (27) |
Ask about budget | The prospect has a real need, but nothing is known about their budget or what they pay today, and no concern is currently open. Find out whether money exists for this. | Calls where budget has already been discussed, where the prospect just objected to price, or before any problem has been identified. | "What are you spending today on scheduling and dispatch, including the manual work?" / "Is there a budget set aside for fixing this, or would we need to build the case together?" / "What range would make this an easy yes for you?" |
| 4 | confirm_decision_process (38) |
Confirm the decision process | The prospect is interested, but it is unclear who makes the purchase decision, who else must approve, or how buying works at their company. | Calls where the decision maker and approval process are already known, or where a price or trust concern is open. | "Besides you, who else would need to weigh in on a decision like this?" / "How did the last software purchase like this get approved on your side?" / "If we agree this is a fit, what happens next internally?" |
| 5 | state_price_with_context (49) |
State the price, anchored to value | The prospect asked what the product costs and has not pushed back on price. Give the number plainly and immediately tie it to the cost of the problem they described; do not dodge or delay. | Prospects who already heard the price and objected to it (that is a price objection), or calls where no one has asked about price. | "For a team your size it's eleven hundred a month, which is roughly a quarter of what you said the slipped jobs cost you." / "Happy to. It's nine hundred a month on the annual plan. Compare that to the thousand a week in wasted hours we just worked out." / "Straight answer: eight fifty a month for nine techs. Want me to show how that nets out against the double bookings?" |
| 6 | handle_price_objection_roi (60) |
Handle the price objection with ROI | The prospect pushed back on price, cost, or value for money. Respond by connecting the price to the cost of the problem they described, not by discounting. | Calls where no price or cost concern was raised, or where the concern is about trust, timing, or authority. | "Fair. Let's compare it to what the problem costs you now: you mentioned roughly ten hours a week of manual scheduling." / "If it saved even two missed jobs a month, would the math work for you?" / "What would it need to save you per month for the price to feel obvious?" |
| 7 | handle_trust_or_risk_concern (71) |
Reassure with proof | The prospect doubts the product will work for a company like theirs, doubts the vendor, or worries about implementation risk or disruption. Offer evidence: a similar customer, references, a pilot, or a guarantee. | Price or cost concerns, or calls where no doubt has been voiced. | "That's a fair worry. Would it help to talk to a company your size that switched last quarter?" / "We can start with a two-week pilot on one crew, so nothing changes for the rest of the team until you've seen it work." / "What would you need to see to feel confident this is low-risk?" |
| 8 | handle_timing_objection (82) |
Handle the timing objection | The prospect says it is not the right time: too busy, mid-season, mid-project, or 'revisit next quarter'. Find out what actually changes by then and connect waiting to the cost they described, without pressuring. | Price, trust, or authority concerns, or prospects who have already named a deadline they want to hit. | "Totally understand. What would be different in September that would make this easier?" / "If the busy season is the pain, would it be worth having the fix in place before it starts rather than after?" / "Would a lighter pilot on one crew be manageable now, so nothing lands on your plate during the peak?" |
| 9 | address_competitor_comparison (93) |
Address the competitor comparison | The prospect mentioned another vendor or their current tool. Ask what they like and dislike about it, then differentiate on what matters to them. | Calls where no other vendor, tool, or alternative was mentioned. | "What do you like about them, and what's made you keep looking?" / "Where is the current tool letting you down most?" / "The main difference for teams like yours is real-time re-dispatch; how do you handle same-day changes with them?" |
| 10 | clarify_simply (104) |
Clarify in plain words | The prospect asked what something means, asked the rep to repeat, or seems lost. Explain plainly in one or two sentences with a concrete example. | Calls where the prospect clearly follows the conversation. | "Let me put that more simply: when a job runs late, the schedule updates itself and the customer gets a text." / "Good question, I skipped a step. In plain terms it means..." / "Can I show you with an example from your own day?" |
| 11 | stop_talking_ask_open_question (115) |
Stop talking, ask an open question | The rep has been talking at length or listing features, and the prospect has gone quiet or is giving short answers. Hand the floor back with an open question. | Moments where the prospect is actively talking or has just asked a direct question. | "I've been talking a lot. What's your reaction so far?" / "How does that compare to what you were hoping for?" / "What questions does that raise for you?" |
| 12 | summarize_and_check_understanding (126) |
Summarize and check understanding | The prospect shared several details, priorities, or concerns. Reflect back what you heard in a sentence or two and confirm you got it right. | Calls where the prospect has said very little so far. | "So if I've got this right: the dispatcher is drowning, techs get sent to the wrong jobs, and customers are calling to complain. Did I miss anything?" / "Let me make sure I understand what matters most to you." / "It sounds like the real issue is same-day changes, not the weekly schedule. Is that fair?" |
| 13 | create_urgency_honestly (137) |
Create urgency honestly | The prospect is interested but has no timeline or says 'someday'. Tie the decision to the real cost of waiting they have already described. | Prospects who already have a deadline, or who just raised a price or trust concern. | "Every month this stays as is costs you the ten hours a week you mentioned. What would it take to decide before the busy season?" / "Is there a reason to wait, or is it more that it hasn't been a priority?" / "If we started this month, you'd be live before summer peak. Does that timing matter to you?" |
| 14 | propose_next_step (148) |
Propose a concrete next step | The conversation is positive, the prospect is engaged, and no concrete next action (demo, trial, follow-up meeting with a date, proposal) has been proposed or agreed yet. | Calls where a next step is already agreed, or where an objection is still open. | "How about I set up a 30-minute demo with your dispatcher on Thursday, using your actual job list?" / "Can I send a one-page proposal tonight and we review it together on Friday?" / "Let's get your ops lead on a call next week; would Tuesday or Wednesday work?" |
| 15 | ask_for_the_close (159) |
Ask for the close | Need, budget, decision maker, and timeline are known, concerns have been handled, and the prospect is showing buying signals. Ask directly for the commitment. | Calls where key qualification facts are missing or a concern is still open. | "It sounds like this solves the problem. Shall I send the agreement over today so you can start Monday?" / "Is there anything stopping us from moving forward?" / "Would you like to go ahead with the two-crew plan we discussed?" |
5a. Gating (_coaching, engine.py:299-320)
- No
next_moveanswer (e.g. Jev error) ->{"gated": true, "reason": "no answer"}. gated = next_move.confidence < NEXT_MOVE_MIN_CONFIDENCE (0.35) or move id unknown(constants.py:331,engine.py:304).best_phrasing= the phrasing chosen byphrasing::<move>if that Choice'sconfidence >= PHRASING_MIN_CONFIDENCE (0.30)(constants.py:332,engine.py:306-310), parsed from the option idp<i>.- Payload:
gated, move_id, title, what, phrasings[3], best_phrasing, confidence, probabilities (top 4). - UI: gated -> italic "listening... (leaning "
", confidence x)"; else title, what, the tailored line if any, then the three lines with the best one prefixed "say:" (index.html:353-373). Test:tests/test_engine_unit.py:206-224.
Why 0.35 is low: per TypeSafe's definition (/home/stevan/dev/jev/docs/vendor/typesafe/confidence.md:143-157 and the explorer's formula (n*peak - 1)/(n - 1)), a 15-option Choice with confidence 0.35 has a peak probability of about 0.39. next_move confidences on correct picks in the report include 0.34, 0.36, 0.39, 0.43, 0.47, 0.48 (eval/integration_report.json:350,264,815,561,225,518), so a stricter gate would hide the card most of the time. The e2e test only requires coaching to be shown on >= 50% of utterances (tests/test_e2e_replay.py:97-100); the reported runs show it on 88-95%.
5b. not_for filtering: there is none in code
The not_for text is only ever used as part of the Choice option description (constants.py:257) plus the instruction "respect each move's not_for" (constants.py:255). The engine does not remove moves whose not_for conflicts with known_facts or the open objection, and does not re-rank Jev's probabilities. The mechanism by which "ask about budget stops being suggested once budget is known" (README) is entirely: code adds budget_discussed to call_facts.known_facts -> Jev reads not_for: "Calls where budget has already been discussed" -> Jev lowers that option. The only code-side decision is the confidence gate. This is a deliberate design choice consistent with TypeSafe's guidance to keep conditionals in code where they are conditionals, but it means a hard business rule ("never suggest X after Y") has no enforcement path today.
6. Personalize (tailored phrasing) flow
Optional; enabled only if ANTHROPIC_API_KEY is set (server.py:36-38, personalize.py:29-34). Off by default; without the key, behaviour is identical to the base app (tests/test_personalize.py:130-140).
6a. Trigger rule (should_personalize, personalize.py:49-65; state in PersonalizeState, personalize.py:43-46)
Return a reason, or None, evaluated on every snapshot (server.py:135-147):
Noneif coaching is gated or has nomove_id.Noneif the move is inPERSONALIZE_SKIP_MOVES = {dig_into_the_problem, stop_talking_ask_open_question, clarify_simply}(constants.py:361) or unknown.Noneifindex - last_index < PERSONALIZE_COOLDOWN_UTTERANCES (2)(constants.py:354)."move changed"ifmove != last_move_id."new fact"iffacts_now - last_factsis non-empty."refresh"ifindex - last_index >= PERSONALIZE_REFRESH_UTTERANCES (8)(constants.py:355).- else
None.
On trigger the server records last_move_id / last_index / last_facts immediately (before the LLM returns), cancels any in-flight personalize task, snapshots the last PERSONALIZE_CONTEXT_UTTERANCES = 6 utterances and the sorted facts, and starts _personalize as a background task (server.py:141-147). Tests: tests/test_personalize.py:22-46.
6b. LLM call (AnthropicPersonalizer.__call__, personalize.py:164-197)
Plain httpx POST to https://api.anthropic.com/v1/messages (constants.py:348), headers x-api-key, anthropic-version: 2023-06-01 (personalize.py:158). Body: model = "claude-haiku-4-5" (constants.py:347), max_tokens = 140 (constants.py:350), temperature = 0.4 (personalize.py:168), timeout LLM_TIMEOUT_S = 3.0 (constants.py:349).
System prompt, verbatim (personalize.py:72-78):
You are whispering one sentence into a sales rep's ear during a live call. Write exactly ONE sentence the rep can say next, in the rep's own voice, natural and spoken. It must carry out the coaching move you are given. Use only details that appear in the transcript: never invent numbers, prices, names, dates, features, or promises. Prefer a question over a statement. Output the sentence only: no quotes, no preamble, no explanation.
User prompt template (build_prompt, personalize.py:85-98):
Coaching move: {move.title}
What it means: {move.what}
Generic example lines for this move (match their tone, but make yours specific to this call):
- {phrasing 0}
- {phrasing 1}
- {phrasing 2}
Facts already established on this call: {comma-joined fact names | 'none yet'}
Open objection: {type | 'unknown'} <- only if objection.open
Recent transcript (rep = you):
{speaker}: {text} x up to 6
Your one sentence (at most {max_words} words):
max_words = PERSONALIZE_MAX_WORDS_BY_MOVE.get(move_id, 32), with 55 for summarize_and_check_understanding, 40 for state_price_with_context and handle_price_objection_roi (constants.py:356-357, personalize.py:81-82).
6c. Code-side validation (clean_line, personalize.py:104-118)
First line only; strip surrounding quote characters ("'“”‘’«»); strip a leading rep: / you: / say: / sentence:; reject if < 8 chars; reject if > max_words + 6 words; append ? if it starts with a question word else . when no terminal punctuation. Output that hit stop_reason == "max_tokens" is discarded as "truncated" (personalize.py:185-186); an unusable line is "unusable output". Tests: tests/test_personalize.py:49-61.
6d. Jev verification (verify_with_jev, personalize.py:205-221; VERIFY_QUESTIONS, constants.py:363-375)
A second Jev request (so 2 Jev requests on utterances that trigger a rewrite) with state {candidate_line, move: {title, what}, recent_transcript: [{speaker, text}], known_facts} and two Nouls, verbatim:
invents_fact
- question: Does
candidate_line, a sentence the rep is about to say, state a number, price, name, deadline, product feature, or commitment that does not appear anywhere inrecent_transcriptorknown_facts? - true: The line quotes a figure, a date, a person, a feature, or a promise that nobody said on the call.
- false: Every specific detail in the line was said on the call, or the line contains no specific details (a plain open question).
on_move
- question: Is
candidate_linea reasonable way for the rep to carry out the coaching move described inmove? - true: The line does what
move.whatdescribes, in the rep's voice, as one natural sentence. - false: The line does something else, addresses the wrong concern, or is not something a rep would say aloud.
Drop rules: invents_fact >= PERSONALIZE_INVENTED_FACT_THRESHOLD (0.5) -> "invents a fact"; on_move < 0.5 -> "off move". If the verification request errors or returns nothing, the line passes ("verification unavailable: do not block on it", personalize.py:214-215). Toggle PERSONALIZE_VERIFY_WITH_JEV = True (constants.py:358).
6e. Stale check and delivery (server.py:149-190)
After verification, if the session's latest step now has a different move_id or is gated, reason "stale". A phrasing message is sent whether or not the line was rejected (text: null + rejected: <reason> + candidate: <raw>), so the UI can show dropped candidates struck through with the reason (index.html:367-368). Counters requests, shown, rejected, errors, cum_cost_usd, last_ms ride along. The UI also drops a displayed tailored line client-side as soon as the move changes (index.html:320).
Reported (README "Tailored phrasing"): on the CloudTalk call, 7 LLM calls over 48 utterances (~$0.006), 2 shown, 5 dropped (1 invents_fact 0.80 for mentioning "automatic note summaries", 2 for length before per-move caps, 2 stale at 50x replay). Haiku pricing constants $1/$5 per Mtok (constants.py:351-352), ~900 in / ~30 out tokens per call.
7. Server, WebSocket protocol, session model
7a. Process model
- One FastAPI app; lifespan creates one shared
JevJudgeand oneAnthropicPersonalizer(server.py:31-43).JevJudgewrapsAsyncTypeSafeClient(model, timeout=8s, retry=RetryPolicy(max_retries=2, backoff 0.2->1.0s, timeout=16s))(jev_client.py:92-102,constants.py:19). Any exception becomesJudgeResult(answers={}, error=...)and the loop continues (jev_client.py:110-115); the engine then gates coaching, keeps the previous stage, and computes probability from zeros (tests/test_engine_unit.py:295-310). - One
Connectionper WebSocket, holding oneCallSession(server.py:100-112). All state is in memory and dies with the socket. No session id, no auth, no persistence, no reconnection to a previous session (the browser's auto-reconnect atindex.html:267starts a fresh, empty session). - Within a connection, messages are handled serially (
server.py:236-243), so a burst ofutterancemessages (video catch-up) queues Jev calls one after another.
7b. REST
| Route | Returns | Line |
|---|---|---|
GET / |
static/index.html |
server.py:52-54 |
GET /api/health |
{ok, model, has_api_key, questions: 37, llm} |
server.py:57-59 |
GET /api/calls |
list of {id, title, description, expected, utterances, duration_s, video} |
server.py:62-64, replay.py:18-33 |
GET /api/calls/{id} |
full call JSON (404 if unknown) | server.py:67-72 |
GET /api/playbook |
the 15 moves | server.py:75-77 |
GET /api/config |
model, weights, stage_prior, ema_alpha, thresholds, all 37 questions | server.py:80-94 |
7c. WebSocket /ws (server.py:215-308)
Client -> server (handle_message, server.py:250-308):
| type | fields | effect |
|---|---|---|
start_replay |
call (id, default good), speed |
stop any replay, session.reset(), reset personalize state, send reset, start replay task |
pause / resume |
clear/set the asyncio.Event; reply paused / resumed |
|
set_speed |
speed |
clamped 0.25-50 (server.py:277; note start_replay does not clamp) |
stop |
cancel replay task; reply stopped |
|
reset |
stop replay, session.reset(), reply reset with call: null |
|
utterance |
speaker, text, optional t |
session.ingest -> reply update; then maybe_personalize |
set_weights |
weights: {signal: w}, optional stage_prior: {stage: p} |
session.set_weights, recompute, reply recomputed |
ping |
pong |
|
| anything else | error |
If the server has no Jev key, every message except ping gets an error (server.py:252-254).
Server -> client:
| type | when | payload |
|---|---|---|
hello |
on connect (server.py:219-233) |
model, has_api_key, llm, personalize_skip_moves, calls[], playbook[], weights[], stage_prior, ema_alpha, thresholds{next_move_min_confidence, objection_open, signal_on} |
reset |
after start_replay / reset |
call (full JSON or null), weights |
update |
after each ingested utterance | the snapshot (below), plus replay: {call, index, total} during replay |
replay_done |
end of replay | call, final_probability |
recomputed |
after set_weights |
the full snapshot for the last ingested step (same shape as update, with contributions, sensitivity, instant and probability recomputed), type rewritten to recomputed. snapshot() defaults step to self.steps[-1] (engine.py:407) and the handler calls it with no argument (server.py:301-303), so the per-utterance block is absent only if no utterance has been ingested. tests/test_server.py:68-70 asserts only on timeline and jev, so this shape is not pinned by a test. |
phrasing |
when a personalize task finishes | `index, move_id, text |
paused / resumed / stopped / pong / error |
Snapshot (CallSession.snapshot, engine.py:406-463): probability, timeline[{index,t,speaker,probability,instant,stage}], facts{name:{since}}, objection{open,type,confidence,since}, stage_history[], weights[], stage_prior, jev{model,requests,errors,total_input_tokens,total_output_tokens,cum_cost_usd,mean_ms,p95_ms,last_ms} and, when a step exists, index, t, speaker, text, instant, stage{choice,confidence}, signals{name:{label,kind,value,on,level,persisted,contribution}}, coaching{...}, talk{...}, sensitivity[], contributions{}, jev_last{model,latency_ms,input_tokens,output_tokens,cost_usd,request_id,error}. Note signals only contains weighted signals; the raw answers are never sent to the browser.
7d. Replay pacing (run_replay, server.py:192-212)
Sleep max(0.15, min(6.0, real_gap) / speed) between utterances (server.py:27-28,198-200), wait on the pause event, ingest, send, maybe personalize. Real gaps above 6 s are capped so silences do not stall the demo.
8. The four UI modes and video sync
Single file static/index.html; Chart.js 4.4.1 from jsDelivr (index.html:7); dark theme; two-column grid collapsing under 1000 px (index.html:107). Tabs at index.html:123-128; each tab toggles a .mode block (index.html:415-421).
Panels (index.html:165-233): closing-probability hero (big number coloured green >= 60%, amber >= 35%, red below; delta in pts) + line chart of EMA'd probability (green fill) and instantaneous estimate (dashed grey), points coloured by speaker, tooltip shows the utterance; Transcript with talk-share bar, pace, "rep monologue" warning; Composite weights table (editable inputs, Apply / Reset); Call stage chips; Next best move card; Open objection card (only when open); Live signals grid (dot lights when on, lock icon when persisted; scores show low/mid/high); What moved the number (top 3); collapsible "Last Jev request" JSON.
| Mode | Input | Speaker | Timing | Lines |
|---|---|---|---|---|
| Replay | server-side task over a chosen data/calls/*.json |
from JSON | server sleeps real gaps (capped 6 s) / speed 0.5-20x slider | index.html:140-147, 422-425 |
| Video | client polls the embedded YouTube player and sends utterance messages itself |
from JSON | fires when t_end (or t) <= player time |
index.html:131-138, 469-546 |
| Live mic | webkitSpeechRecognition, continuous, interimResults, en-US, auto-restart on onend; each final result is one utterance |
manual toggle buttons or key S |
t = seconds since first mic start (liveT) |
index.html:149-155, 444-467 |
| Type | text box + Enter | dropdown; alternates automatically after each send | liveT() |
index.html:157-162, 434-440 |
Video sync in detail (index.html:469-546):
- Load: fetch
/api/calls/<id>, requirevideo.youtube_id, load the IFrame API script on demand (:474-481), sendreset, createYT.Playerwithstartoffset (:512-537). - While
PLAYING, a 200 mssetInterval(:504-510) callssyncTo(player.getCurrentTime()). videoTick(now)(:490-497) sends every not-yet-sent utterance whosefireAt = t_end ?? tis <=now, in order. This mimics an STT finalising a segment when the speaker stops.syncTo(:499-503): ifnow < lastT - 1.5the user scrubbed backwards ->videoReset(now)sendsresetand immediately re-fires everything beforenow(silent catch-up, each a real Jev call). Forward scrubs are just a larger tick.PAUSED/BUFFERINGstop the loop and do one sync;ENDEDflushes everything (:528-534); Restart = reset +seekTo(start)+ play (:543-546).
There is no audio processing anywhere in the app: video mode depends on a pre-made, hand-labelled transcript with t_end per turn (video.youtube_id and the first t/t_end entries at data/calls/cloudtalk.json:7-9; the file's description at :4 and README.md:64-67 record that it was transcribed with faster-whisper small.en with speakers labelled by hand). The README gives the manual recipe (README.md:68-70) (yt-dlp -> ffmpeg 16 kHz mono -> faster-whisper -> group into turns).
9. Eval and test methodology; reported cost and latency
9a. Offline (mock Jev, no network)
tests/conftest.py: neutral_answers() builds a full 37-answer set where every noul is 0.05, scores are level 1.0, stage = discovery (0.9), objection_type = none, next_move = dig_into_the_problem (0.7), every phrasing p0 (0.6) (:34-50); MockJudge pops one override dict per call and records the states it was sent (:53-67). requires_api_key() skips live tests when no key or COPILOT_SKIP_LIVE=1 (:80-84).
tests/test_engine_unit.py(25 tests): arithmetic, clamps, EMA, talk ratio, pace window, sensitivity, percentile; fact persistence and feedback intoknown_facts; prospect-only facts ignore rep turns; speaker masking; objection open / not-cleared-by-disengaging / clear / expire; coaching gate + best phrasing; talk-ratio penalty and monologue; EMA fade; recompute with zero Jev calls; sensitivity per utterance; state window shape; Jev error resilience; replay files well-formed (35-80 utterances, sortedt, both speakers); text transcript parser; every question is literal and complete.tests/test_server.py(3): HTTP endpoints; WShello -> start_replay -> reset -> update x2 -> pause -> set_weights -> recomputedwithjudge.calls == jev.requests(no extra calls); live utterances, reset, unknown-type error.tests/test_personalize.py(11): trigger rule matrix;clean_linecases; prompt contents;phrasinground-trip with a mock LLM (exactly one LLM call per move change,PERSONALIZE_VERIFY_WITH_JEVmonkeypatched off); silent when no key.
9b. Integration (real API) - tests/test_integration_jev.py
19 hand-labelled cases, each = BASE_CONTEXT (4 neutral turns, :28-33) + case-specific context turns + one latest utterance; the window is built in code without Jev calls for context turns, then one system_one request with all 37 questions (:181-224). 56 threshold checks in total using T_ON = 0.60, T_FACT = 0.70, T_OFF = 0.40 (:58-60) and choice_is(...) membership. Pass criterion: overall pass rate >= MIN_PASS_RATE = 0.90 (:25, asserted :240); writes eval/integration_report.json with observed values per case.
Cases (id -> checks): price_objection (objecting >= .6, type price, stage in {objection_handling, pricing}, buying < .4); buying_signal; commitment (commitment, buying, stage in {closing, next_steps}, not objecting); rep_feature_dump (pitching, not discovery, stage pitch_demo, move stop_talking); budget_mention (fact); decision_maker (fact, stage qualification); next_step_proposed_by_rep; next_step_agreed_by_prospect; rep_discovery_question; prospect_disengaging; buyer_confused (+ move clarify_simply); competitor_mentioned (+ objecting, type in {competitor, price}); timeline_known (+ urgency >= 1, pain); pain_identified (+ move in {quantify, dig, summarize}, stage discovery); timing_objection (+ move in {handle_timing, create_urgency}); authority_objection (type authority, DM fact, objecting); prospect_accepts; price_asked_neutral (asked_price, stage pricing, not objecting, move in {state_price, ask_about_budget}); warm_opening (not pitching, no next step) (:63-178).
Reported last run: 56/56 = 100% on jev-1.13.0 (eval/integration_report.json:2-6); per-request latency 274-609 ms with a 1,014 ms first request; 7,391-7,502 input tokens.
9c. End-to-end (real API) - tests/test_e2e_replay.py
Module fixture replays good, bad, mixed through cli.run_call and writes eval/timelines.json (:27-39). Assertions:
- zero errors,
model == jev-1.13.0,requests == utterances(:46-50); - good final >= 0.70; bad final <= 0.40; good > bad + 0.30; good's last-quarter mean > first-quarter mean + 0.30; good persisted {next_step_agreed, pain_identified, decision_maker_identified} (
:53-63); - bad min <= 0.25; no
next_step_agreed;competitor_mentionedpersisted; apriceobjection opened at some point;stop_talking_ask_open_questionshown un-gated at least once (:66-75); - mixed: min of first half <= 0.30; final >= 0.65; final - dip >= 0.35;
next_step_agreedpersisted (:78-86); - latency p95 < 2,000 ms; cost per call < $0.05; cost == input_tokens x $0.042/M (
:89-94); - coaching shown on 50-100% of utterances (
:97-100).
Reported numbers are in section 3f. DEMO.md cross-references utterance numbers, but its good call markers are stale relative to the recorded eval/timelines.json run: DEMO.md:37-40 says #11 pain lock and an authority objection at #25, whereas the recorded run locks pain_identified at #5 (prospect turn), opens the objection at #23 as unknown, types it price (0.47) at #25 and trust (0.56) at #27, and clears it at #29; no authority objection appears anywhere in the recorded good timeline (utterance #25, "Eleven hundred... The owner, Greg, would need to sign off", is ambiguous between price and authority and Jev returned price). The bad #9 price objection and the mixed dip to ~16% at #7 do match. Anyone using "authority at #25" as a canonical demo of the objection taxonomy will not be able to reproduce it.
9d. What the methodology does and does not establish
- It shows that the 37 questions separate 19 cherry-picked, cleanly written utterances at the chosen thresholds, and that three hand-written scripts produce the intended curve shapes. The calls were written by the same author who tuned the weights; README "Limitations" says so explicitly.
- No held-out real calls beyond CloudTalk, no inter-annotator agreement, no calibration check of "closing probability" against outcomes (README: "a coaching signal, not a calibrated forecast").
- Threshold drift on a model upgrade is guarded only by re-running the integration test (pin at
constants.py:16; TypeSafe's own advice at/home/stevan/dev/jev/docs/vendor/typesafe/models.md:40).
10. Transferable vs demo-specific (for CurrencyTransfer)
10a. Reuse as-is (architecture and mechanics)
- One speculative fan-out request per utterance, typed answers only, all arithmetic in code. This is exactly TypeSafe's recommended shape (
/home/stevan/dev/jev/docs/vendor/typesafe/patterns__fan-out.md; jaggedness doc says keep math in code,model-jaggedness__jev-1.13.mdsection "Math and Numbers"). It maps directly onto Cloudflare'senv.AI.run('typesafe/jev', {state, questions})(/home/stevan/dev/jev/docs/vendor/cloudflare/ai__models__typesafe__jev.md:34-66): the state and questions objects are the same JSON. - State shape:
call_facts+ 12-turnrecent_transcript+ duplicatedlatest_utterance, with questions that name the path they read. Keeps state small (jaggedness "Large state full of irrelevant detail"). KeepRECENT_WINDOW, theMM:SSstamps, the compressedstage_history. - Engine skeleton (
engine.py):Stepwith raw answers stored,composite+ EMA, speaker masking table, fact persistence with feedback viaknown_facts, objection state machine, confidence gate, recompute-without-Jev, sensitivity. Port to TypeScript nearly line for line; it is ~480 lines with no I/O. - Question-writing style: literal, one condition per question,
true/falsecriteria with concrete example phrases, scores as 3 named situations withsignals, Choice options as{what, not_for}. This is the actual "secret" of why it works quickly: no prompt engineering loop, just a reviewable list of questions. - Test harness pattern: mock judge with
neutral_answers()+ scripted overrides for offline tests; a labelled-utterance integration test that writes a report and gates on pass rate; e2e curve-shape assertions on canonical calls. Reuse the structure; replace the content. - The verification-by-Jev trick for any generated text (
invents_fact,on_move), if generation is kept at all. - Protocol:
hello / reset / update / recomputed / phrasing / errorand theset_weights -> recomputedloop. Good fit for a Durable Object holding one session per call with a WebSocket.
10b. Redesign for CurrencyTransfer (content, not mechanics)
Everything below is domain content that is hard-coded to selling field-service scheduling software and must be rewritten. Assumptions about the CT domain are marked; they need Stevan's confirmation.
- Two question banks, one per scenario (onboarding call vs customer-success call), selected per session. The reference has one global
QUESTIONS;CallSessionalready acceptsquestions=(engine.py:161), so the plumbing exists. - Stages: the eight sales stages are wrong for both scenarios. Assumption for onboarding:
opening, needs (purpose/currencies/amount/timing), how_it_works, rate_and_fees, safety_and_regulation, settlement_and_funding, kyc_and_documents, next_steps. Assumption for customer success:opening, reason_for_contact, account_status, rate_or_fee_question, transfer_status, issue_resolution, upsell_or_forward, wrap_up. Stage priors should be re-derived from real outcome data, not copied. - Objection / concern taxonomy replaces price/timing/authority/need/trust/competitor. Assumption:
rate(rate not competitive / wants to wait for a better rate),fees(hidden charges, margin),safety(is my money safe, FCA/safeguarding, fraud worry),settlement(when do funds land, cut-off times, weekends),process(KYC friction, documents, how to fund),competitor(bank, Wise, Revolut, another broker),timing(not transferring yet). Keepnone. Keep the rule "type only counts when the separate objecting Noul fires". - Durable facts replace pain/budget/DM/timeline/next step. Assumption:
purpose_known(property, salary, invoice, savings),currency_pair_known,amount_or_range_known,timing_known(when funds are needed / when they will fund),funding_source_known,beneficiary_known,kyc_status_known,rate_expectation_known,competitor_named,next_step_agreed(e.g. book a rate, register, upload documents, fund the account). Because the reference stores only fact names, consider adding a small value-extraction Choice per fact (e.g. purpose as a closed set) so the dashboard can show "GBP->EUR, ~200k, property, needed by mid-Oct" rather than five ticks. Jev's date/number handling is weak, so keep values as closed Choices and do any date maths in code (model-jaggedness__jev-1.13.md, "Date and time comparison"). - The number: "closing probability" is the wrong hero metric. Assumption: onboarding wants "likelihood this caller registers / books a first trade", customer success wants "risk of churn / escalation" or a plain "call health" composite. Same maths, different weights and sign conventions; the EMA, clamps and sensitivity panel carry over unchanged.
- Playbook: 19 of the 45 lines (including all three
state_price_with_contextlines) hard-code field-service details or dollar prices ("jobs", "techs", "dispatcher", "$1,100/month"); the remaining ~26 lines and nearly allwhat/not_fordescriptions are domain-neutral ("Is there anything stopping us from moving forward?", "What questions does that raise for you?") and can be kept. Rewrite the domain-specific lines per scenario with CT's compliance-safe phrasing and add CT-specific moves. Assumption: moves such asexplain_how_the_rate_works,explain_fees_transparently,reassure_on_safeguarding(FCA-authorised, segregated client funds; exact wording must come from compliance),explain_settlement_timeline,walk_through_kyc,handle_rate_shopping,offer_forward_or_rate_alert,confirm_next_step_register,summarize_and_check_understanding,clarify_simply,stop_talking_ask_open_question. - Speaker labels:
rep/prospect->agent/client, and consider a third labelunknownfor undiarised segments so masking degrades gracefully instead of mislabelling. - Regulated wording: the tailored-phrasing LLM step is the one place free text is generated and shown to an agent. For FX (FCA/PSR rules on financial promotions and misleading statements) the safer default is curated lines only with the LLM step off, or gated behind a much stricter check that also asks Jev "does
candidate_linemake a claim about rates, returns, guarantees, safety or timing?" -> drop. Assumption: treat as a compliance decision, not an engineering one. - Calibration data: replace the 3 hand-written calls and 19 labelled utterances with CT call-coach transcripts. The reference's own README says its thresholds are starting points. The labelled-utterance test format (
context turns + latest utterance + threshold checks) is exactly the format an admin curation tool should produce. - Language: Jev's primary training language is English; other languages have lower accuracy (
/home/stevan/dev/jev/docs/vendor/typesafe/concepts__state.md, Note). If CT calls include non-English callers, measure before relying on it.
10c. Missing for production (the reference is a demo)
| Gap | Evidence | What is needed |
|---|---|---|
| Auth / tenancy | /ws accepts anyone; server binds 127.0.0.1 (run.sh:43); one global judge; no user or org concept |
Cloudflare Access or app auth on the Worker; per-org config (question bank version, playbook, weights); rate limits per tenant. |
| Persistence | Everything in CallSession memory per socket (server.py:100-112); reconnect = new empty session (index.html:267) |
Durable Object per call (WebSocket + state), append-only step log (raw answers + features) to D1/R2 for replay, recompute, audit, and later admin labelling. Assumption on CF product choice. |
| Diarization / STT | None; mic mode is browser Web Speech with a manual toggle (index.html:444-467); video mode uses a pre-made transcript |
Real STT with diarization (or telephony that gives separate agent/client legs). Jev only sees text, so the transcription layer is a separate decision. |
| Recording ingestion | Manual yt-dlp/ffmpeg/faster-whisper recipe in README; JSON by hand |
A pipeline from call-coach recordings/transcripts into {t, t_end, speaker, text} turns. |
| Admin curation | Playbook is a static file loaded at import (constants.py:35-41); weights are per-session and lost; questions are code |
Versioned question bank + playbook in a store; admin UI to label utterances (the integration-test case format), mark calls as exemplary/poor, edit what/not_for/lines, tune weights with the recompute loop, and promote a version after the pass-rate gate passes. |
| Hard business rules | not_for is prompt-only (section 5b) |
Code-side allow/deny of moves on facts/objection state where a rule must be guaranteed. |
| Objection expiry semantics | Expiry from first open, type overwritten (section 4b) | Decide: refresh since on re-raise; keep a list of open concerns rather than one. |
| Threshold retuning | recompute reuses features, not answers (section 3e) |
Re-derive features from stored raw answers so thresholds are tunable offline too. |
| Observability | Python logging only (jev_client.py:128-132) |
AI Gateway logging on the Workers side; per-request request_id and model stored with each step (the reference already captures both). |
| Model pinning on Cloudflare | Reference pins jev-1.13.0 via the SDK; the Cloudflare page shows 'typesafe/jev' and a response model: "jev-1.13.0" (ai__models__typesafe__jev.md:38,109; second example at :158,230) |
Open question: whether the CF binding lets you pin a version. If not, the integration test must run on every deploy and on TypeSafe releases. |
| Context limit | Cloudflare page lists a 32k context window (ai__models__typesafe__jev.md:29); TypeSafe lists 64k per request / 32k for state + longest question (models.md:15) |
Budget: ~7.5k tokens today, fine. Two scenario banks with value-extraction questions will grow it; measure. |
| Cost/billing | $0.042 per Mtok input, output free (models.md:13); Unified Billing adds a 5% credit fee (ai-gateway__features__unified-billing.md:19) |
~$0.013-0.014 per 40-turn call in the reference; a 15-minute CT call at ~100 turns and ~8k tokens each is ~$0.03-0.04 plus verification calls. |
| Adversarial content | Jev treats state as data but can be steered by injected text (model-jaggedness__jev-1.13.md, "Adversarial content") |
Callers can say anything; keep questions precise and test with hostile transcripts. |
| Error handling UX | A Jev error yields zeros for that turn and a toast (engine.py:215-224, index.html:329) |
Degrade explicitly (hold last state, mark unknown) rather than showing zeros as signal. |
10d. Why it was fast to build (so we can be equally fast)
- No training, no embeddings, no RAG, no prompt chain: one question file + one playbook file + arithmetic.
constants.py(375 lines) andplaybook.json(170 lines) are the entire behaviour;engine.pyis generic. - Jev's typed outputs remove all parsing/validation code. Every decision is a number with a threshold in one file.
- Evaluation is cheap enough to run on every change (~$0.05 for the full suite, README "Tests and evaluation"), so tuning is iterative.
For CT the equivalent minimum is: two constants-style banks, two playbooks, a rep/prospect -> agent/client rename, and 20-30 labelled utterances per scenario taken from real call-coach transcripts to make the integration test meaningful before touching any UI.