Measured cost and latency
Measured from every Jev request logged between 2026-09-27T07:13:10.000Z and 2026-09-27T07:20:51.019Z (770 attempts: 770 answered, 0 without usage), Worker version 2f03062938a0. Generated 2026-09-27T07:20:51.356Z. These numbers replace the estimates in the architecture document §12.
Requests by kind
| kind | requests | answered | no usage | Σ input tokens | Σ output tokens | mean latency ms | p95 latency ms |
|---|---|---|---|---|---|---|---|
| eval | 172 | 172 | 0 | 2,358,483 | 456,557 | 360 | 472 |
| full | 598 | 598 | 0 | 4,668,010 | 718,195 | 341 | 450 |
| total | 770 | 770 | 0 | 7,026,493 | 1,174,752 |
Latency is the provider round trip of answered requests; p95 is linearly interpolated.
By kind and scenario
| kind | scenario | requests | answered | Σ input tokens | mean latency ms | p95 latency ms |
|---|---|---|---|---|---|---|
| eval | customer_success | 81 | 81 | 1,075,863 | 344 | 425 |
| eval | onboarding | 91 | 91 | 1,282,620 | 373 | 521 |
| full | customer_success | 306 | 306 | 2,207,139 | 330 | 441 |
| full | onboarding | 292 | 292 | 2,460,871 | 353 | 478 |
Replays per scenario
| scenario | replay sessions | decision points | Σ input tokens | tokens per decision point | cost per replayed call |
|---|---|---|---|---|---|
| customer_success | 4 | 306 | 2,207,139 | 7,213 | $0.0232 |
| onboarding | 4 | 292 | 2,460,871 | 8,428 | $0.0258 |
Cost per replayed call is Σ input tokens ÷ replay sessions × $0.042/M (output not priced); a session that stopped early counts as one call.
Credit balance against usage
credit delta: not measured (dashboard-only); effective $/Mtok assumed at list price $0.042/M input, output free
The AI Gateway "Credits Available" balance has no API. Reading it just before and just after a replay and re-running with --balance-before <usd> --balance-after <usd> reconciles the window.
AI Gateway analytics cross-check
aiGatewayRequestsAdaptiveGroups for gateway jev-copilot over the same window, independent of the request log above.
| model | requests | tokens in | tokens out | gateway cost | cost per Mtok in |
|---|---|---|---|---|---|
| typesafe/jev | 765 | 6,990,232 | 1,169,400 | $0.2936 | $0.0420 |
The two are counted independently: the gateway sees every caller routed through it (local scripts as well as the Worker) and records its own view of attempts that failed, so its figures differ from the request log by a few percent.
Characters per token
Σ request characters ÷ Σ input tokens over the 770 answered requests: 25,261,132 ÷ 7,026,493.
measured chars_per_token: 3.6
unknown_usage_requests: 0 (0 characters sent without a usage figure; not in the divisor).
Cold start
A session's first answered request against its own median, as the cold-start proxy.
| scenario | sessions | first request mean ms | session median mean ms | first − median ms |
|---|---|---|---|---|
| customer_success | 4 | 822 | 304 | 517 |
| onboarding | 4 | 940 | 313 | 627 |
M1b: STITCH_V2 (104e)
Eval run m1b-stitch2-20260927 (L4 + L6, policy v4 for both scenarios, fresh answers, Worker a3bbf38), the first on the v2 stitch: the four sample calls were re-imported with sentence-sized parts on 2026-09-27. Compared with the M1a final run final-20260927 on the v1 stitch, at the same policy and model (jev-1.13.0). Cost is Σ input tokens × $0.042/M, as above.
| call | scenario | decision points v1 → v2 | cost per call v1 → v2 | change |
|---|---|---|---|---|
| 3339895706 | onboarding | 96 → 128 | $0.0345 → $0.0460 | +33 % |
| 3485591407 | onboarding | 50 → 70 | $0.0172 → $0.0237 | +38 % |
| 3303259297 | customer_success | 131 → 164 | $0.0396 → $0.0494 | +25 % |
| 3347356034 | customer_success | 22 → 43 | $0.0068 → $0.0131 | +93 % |
| figure | v1 (final-20260927) |
v2 (m1b-stitch2-20260927) |
|---|---|---|
| requests | 299 | 405 |
| max input tokens | 10,511 | 10,411 |
| p95 input tokens | 9,395 | 9,332 |
| p95 provider latency ms | 498 | 440 |
| e2e p95 ms (3347356034 at 1×) | 457 (paced, gaps capped at 6 s) | 5,319 (wall clock, COPILOT-118a) |
Cost grows with the number of decision points, not with the prompt: the per-request input size is unchanged. Every call stays under the L6 limit of $0.05 per call, but 3303259297 is at $0.0494, 1.2 % under it.
The L6 e2e gate (≤ 3,000 ms) failed on this run. This is the first eval on wall-clock pacing: each utterance is sent at its own t_end, so real gaps are left in. The 43 figures have a median of 349 ms. Three consecutive decisions (i 26, 27, 28) took 10,894, 5,401 and 5,319 ms, and the p95 of 43 samples is the third-worst figure. The replay has no reconnect counter for this path, so the cause is inferred rather than measured. The likely cause is the session waking after hibernating in a gap, as COPILOT-118a found; the old 6 s cap on gaps hid that. It is not a stitch effect.
M1b: cadence re-tune (106)
Policy v12 for both scenarios (stitch_version 2), Worker a82fe8e, model jev-1.13.0, fresh answers. The counters that were tuned for one decision per turn now match the sentence-level cadence of the v2 stitch: concern_max_age 8 → 12, card_cooldown_utts 3 → 4, token_budget.window 12 → 16, purpose_max_decisions 8 → 12 and COVERED_DWELL 4 → 6. Kept: confirm_updates 2, stage_confirm 2, ema_alpha 0.4, soft 11,000 and hard 12,000 (decision 15).
The L6 cost gate is now $0.06 per call (L6_LIMITS.cost_per_call_usd, decision 16). The $0.05 limit quoted in the sections above was the gate when they were written.
Gate run m1b-tune-20260928b (L1 + L4 + L6), exit 0: L1 onboarding 97/97, customer success 80/82, L4 PASS, L6 PASS. The first run, m1b-tune-20260928, failed only L4 zero_errors: 2 unanswered decisions on each strong call, from "2018: Invalid User Credentials" upstream errors between 04:23:56 and 04:24:02, the same transient seen in COPILOT-122's gate.
| call | scenario | decision points | cost per call (104e, v4) | cost per call (v12) |
|---|---|---|---|---|
| 3339895706 | onboarding | 128 | $0.0460 | $0.0452 |
| 3485591407 | onboarding | 70 | $0.0237 | $0.0240 |
| 3303259297 | customer_success | 164 | $0.0494 | $0.0495 |
| 3347356034 | customer_success | 43 | $0.0131 | $0.0127 |
| figure | m1b-tune-20260928b |
|---|---|
| requests | 405 |
| max input tokens | 10,266 |
| p95 input tokens | 9,037 |
| p95 provider latency ms | 410 |
| e2e p95 ms (3347356034 at 1×) | 442 |
The wider window costs little: no request in the run needed a budget drop. Of the 810 decisions in both runs, window_to_8 fired on 0 (the baseline was 10.5 %), and every request went out at window 16. The worst case the unit tests build (the longest 16-utterance v1 window that ends on a rep turn, with a rate concern open) is 11,876 tokens for onboarding and 10,675 for customer success, both after window_to_8. Real v2 turns are shorter than those v1 turns.
M1b: pre-judge of the item about to open (108b)
Policy v14 for both scenarios, Worker bed9a1e, model jev-1.13.0, fresh answers. On client turns the request also asks fits:: (up to 3) and the situation::<move> Choice for the item that would become current next, so a new item can open with points instead of "Listening" (decision 3 R6). The pre-judge is a list of its own, and the code-rule step prejudge drops it first when a request is over the soft budget, before fact_values. v14 differs from v13 only in token_budget.drop_order (prejudge added), so the bank and playbook hashes are unchanged.
Gate run m1b-prejudge-20260928b (L1 + L4 + L6, --fresh), exit 0: L1 onboarding 96/97, customer success 79/82, L4 PASS, L6 PASS. The first run, m1b-prejudge-20260928 (exit 0), answered every L1 case from the local cache, so it recorded nothing as evaluated. The four L1 misses in run b have all failed in earlier runs on the same bank: lfl_cs_payment_mentioned_no_amount in 23 of 23 runs, lfl_cs_upcoming_payments_unknown in 9 of 23, rate_ob_neg_rep_first_one_line (0.41 against < 0.4) in 7 of 23, and client_confused_sender_details (0.67 against ≥ 0.7) in 7 of 23.
| call | scenario | decision points | cost per call (v12, 106) | cost per call (v14) | delta |
|---|---|---|---|---|---|
| 3339895706 | onboarding | 128 | $0.0452 | $0.0457 | +$0.0005 |
| 3485591407 | onboarding | 70 | $0.0240 | $0.0246 | +$0.0006 |
| 3303259297 | customer_success | 164 | $0.0495 | $0.0496 | +$0.0001 |
| 3347356034 | customer_success | 43 | $0.0127 | $0.0127 | $0.0000 |
| figure | m1b-tune-20260928b (v12) |
m1b-prejudge-20260928b (v14) |
|---|---|---|
| requests | 405 | 405 |
| max input tokens | 10,266 | 10,340 |
| p95 input tokens | 9,037 | 9,256 |
| p95 provider latency ms | 410 | 424 |
| e2e p95 ms (3347356034 at 1×) | 442 | 799 |
The pre-judge costs less than expected ($0.0005-0.0006 per onboarding call against the $0.003-0.005 estimated). It is asked only while a next item exists and has candidate situations. On the typed templates, a fact item has no row until it surfaces, and a must-say waits until it is due, so often there is none. Of the 405 decisions in run b, 4 dropped prejudge for budget and none dropped anything else.
Live replay of 3339895706 on v14 (session m1b-108b-20260928), with scripts/ws-probe.mjs --count-points, against 108a's v13 session m1b-108a-20260928:
| figure | v13 | v14 |
|---|---|---|
| decisions_with_point | 35 | 33 |
| items_with_point | 5 | 6 |
| withdrawn_within_12s | 2 | 4 |
| items opening with a situation on their first decision | 0 of 11 | 1 of 11 |
In the v14 session, explain_who_holds_the_money opened at i 146 already showing is_my_money_safe. Of the other 10 opens, 4 were inserted concern items, which the plan cannot foresee, and 1 was the call's first item. For the remaining 5 (i 10, 17, 28, 95 and 141), this probe does not show why they opened unseeded: the target might not have been pre-judged on a client turn, or its answers might not have passed the §3.3 rule. Each replay gets fresh answers, so a single call cannot tell a 2-decision difference from variance.
M1b: the whole milestone (118)
The cost report over the M1b window, from the first M1b iteration to the final verification (COPILOT-118): node scripts/cost-report.mjs --since 2026-09-27T14:43:47Z --out /tmp/m1b-cost.md, exit 0, Worker 5767218623d1 (the deployed HEAD for the final pass). Every replay, eval and probe of the milestone is in it, not only the final run. The final run's own cost per call is in eval/reports/m1b-final-20260928.l4l6.json (L6 PASS, $0.0495 per call at most against the $0.06 limit).
Measured from every Jev request logged between 2026-09-27T14:43:47.000Z and 2026-09-28T08:12:57.686Z (15,776 attempts: 15,195 answered, 581 without usage), Worker version 5767218623d1. Generated 2026-09-28T08:12:58.296Z. These numbers replace the estimates in the architecture document §12.
Requests by kind
| kind | requests | answered | no usage | Σ input tokens | Σ output tokens | mean latency ms | p95 latency ms |
|---|---|---|---|---|---|---|---|
| eval | 3,546 | 3,546 | 0 | 49,025,409 | 9,472,487 | 349 | 448 |
| full | 12,168 | 11,587 | 581 | 90,756,150 | 13,870,067 | 336 | 440 |
| moves-only | 62 | 62 | 0 | 167,042 | 15,795 | 469 | 1135 |
| total | 15,776 | 15,195 | 581 | 139,948,601 | 23,358,349 |
Latency is the provider round trip of answered requests; p95 is linearly interpolated.
By kind and scenario
| kind | scenario | requests | answered | Σ input tokens | mean latency ms | p95 latency ms |
|---|---|---|---|---|---|---|
| eval | customer_success | 1,636 | 1,636 | 21,890,644 | 337 | 426 |
| eval | onboarding | 1,910 | 1,910 | 27,134,765 | 359 | 482 |
| full | customer_success | 5,351 | 5,077 | 36,250,856 | 335 | 428 |
| full | onboarding | 6,817 | 6,510 | 54,505,294 | 337 | 448 |
| moves-only | onboarding | 62 | 62 | 167,042 | 469 | 1135 |
Replays per scenario
| scenario | replay sessions | decision points | Σ input tokens | tokens per decision point | cost per replayed call |
|---|---|---|---|---|---|
| customer_success | 49 | 5,132 | 36,250,856 | 7,064 | $0.0311 |
| onboarding | 97 | 6,634 | 54,672,336 | 8,241 | $0.0237 |
Cost per replayed call is Σ input tokens ÷ replay sessions × $0.042/M (output not priced); a session that stopped early counts as one call.
Credit balance against usage
credit delta: not measured (dashboard-only); effective $/Mtok assumed at list price $0.042/M input, output free
The AI Gateway "Credits Available" balance has no API. Reading it just before and just after a replay and re-running with --balance-before <usd> --balance-after <usd> reconciles the window.
AI Gateway analytics cross-check
aiGatewayRequestsAdaptiveGroups for gateway jev-copilot over the same window, independent of the request log above.
| model | requests | tokens in | tokens out | gateway cost | cost per Mtok in |
|---|---|---|---|---|---|
| typesafe/jev | 17,308 | 141,521,506 | 23,471,983 | $5.9439 | $0.0420 |
The two are counted independently: the gateway sees every caller routed through it (local scripts as well as the Worker) and records its own view of attempts that failed, so its figures differ from the request log by a few percent.
Characters per token
Σ request characters ÷ Σ input tokens over the 15,195 answered requests: 501,361,724 ÷ 139,948,601.
measured chars_per_token: 3.58
unknown_usage_requests: 581 (16,120,325 characters sent without a usage figure; not in the divisor).
Cold start
A session's first answered request against its own median, as the cold-start proxy.
| scenario | sessions | first request mean ms | session median mean ms | first − median ms |
|---|---|---|---|---|
| customer_success | 49 | 806 | 295 | 511 |
| onboarding | 97 | 875 | 372 | 504 |