Jev call copilot CurrencyTransfer research and architecture built 2026-09-24

Speech-to-text options on Cloudflare for the CurrencyTransfer call copilot

Research date: 2026-09-24. Scope: STT for (A) a proof of concept that uploads a recorded call (mp3/m4a/wav, 5-40 min, two speakers) and replays it utterance-by-utterance so the dashboard behaves as if live, and (B) live call audio later. Every non-obvious claim carries a source. Things I could not verify are marked ASSUMPTION or UNVERIFIED.

Local reference: /home/stevan/dev/jev/reference/jev-sales-copilot (the open-source copilot). The Jev guide is at /home/stevan/dev/jev/docs/jev-guide.md; the local vendor docs at /home/stevan/dev/jev/docs/vendor/cloudflare/ contain nothing about STT, so almost everything below comes from live Cloudflare/Deepgram/MDN pages and from the model definition JSON in the cloudflare-docs repo.


1. What the reference copilot actually does for speech (and why it is not enough)

So the reference sidesteps STT entirely. For us, speaker attribution (rep vs client) is the single hard requirement: the speaker-masked Jev questions ("did the client just raise a rate objection?") are impossible without it. That rules out every option that returns plain text with timestamps and nothing else.


2. Requirements checklist

# Requirement Why
R1 Speaker diarization (rep vs client), or per-speaker channels speaker-masked questions, "who said it" in the timeline
R2 Utterance segmentation with start/end timestamps replay pacing; feeding Jev one utterance at a time
R3 Long-file handling (5-40 min, 5-60 MB) without hand-rolled chunking PoC upload path and bulk ingestion of the historical corpus
R4 UK English accuracy incl. FX vocabulary (GBP, EUR, SWIFT, IBAN, spot, forward, settlement) domain terms drive the objection classifiers
R5 Cost per hour of audio corpus of "many" historical calls; ongoing live use
R6 Live latency (< ~1 s to a final utterance) mode B
R7 Runs from a Worker with no extra vendor account, or at least via AI Gateway "on Cloudflare" constraint

3. Catalogue of options

3.1 @cf/openai/whisper-large-v3-turbo (Cloudflare-hosted)

Source: model JSON src/content/workers-ai-models/whisper-large-v3-turbo.json in github.com/cloudflare/cloudflare-docs (fetched raw 2026-09-24) and https://developers.cloudflare.com/workers-ai/models/whisper-large-v3-turbo/

3.2 @cf/openai/whisper and @cf/openai/whisper-tiny-en

Source: model JSON for both. Input is raw binary (format: binary) or {audio: [uint8...]}; output text, word_count, words[]{word,start,end}, vtt. No segments object, no speakers, none of the anti-hallucination knobs. whisper costs $0.000453/min (41.14 neurons); whisper-tiny-en is beta with no price listed. Same size problem. Strictly dominated by v3-turbo for our purposes.

3.3 @cf/deepgram/nova-3 (Cloudflare-hosted partner model) — HTTP and WebSocket

Source: model JSON nova-3.json; https://developers.cloudflare.com/workers-ai/models/nova-3/ (raw markdown fetched); Deepgram docs as cited.

3.4 @cf/deepgram/flux (WebSocket only)

Source: flux.json; https://developers.deepgram.com/docs/flux/feature-overview. Input only linear16 PCM; parameters are eot_threshold, eager_eot_threshold, eot_timeout_ms, keyterm; events Update / StartOfTurn / EagerEndOfTurn / TurnResumed / EndOfTurn with transcript, words[]{word, confidence} (no per-word timestamps in the Workers AI schema), audio_window_start/end. No diarize parameter; Deepgram's Flux feature table lists no diarization or multichannel. $0.0077/min (700 neurons) = $0.462/h. Built for voice agents that must know when to speak; its turn-end events are attractive for "utterance finalised" but it cannot tell rep from client. Not suitable unless we already have per-speaker streams (then two Flux sockets would work, at 2x cost, with no advantage over nova-3).

3.5 Third-party ASR models in the unified catalogue (developers.cloudflare.com/ai/models/)

These are called with the same env.AI.run('<provider>/<model>', …) / POST /accounts/{id}/ai/run shape but are billed through AI Gateway credits or BYOK, not neurons. Evidence: today's typesafe/jev call returned HTTP 402 "Insufficient balance; add money to your gateway or use BYOK" (scratchpad jev-run-A.out). Pricing on every catalogue page is "View pricing in the Cloudflare dashboard"; I list provider list prices as a proxy.

Model Diarization Timestamps Input Provider list price Notes
assemblyai/universal-3.5-pro (https://developers.cloudflare.com/ai/models/assemblyai/universal-3.5-pro/) speaker_labels, speakers_expected; returns utterances[] word-level ms audio_url (public URL or data URI), audio_start_from/audio_end_at $0.21/h + $0.02/h diarization add-on (https://www.assemblyai.com/pricing) Strong batch contender; no streaming via CF that I could find. universal-3-pro also listed, "Zero data retention".
xai/grok-stt (https://developers.cloudflare.com/ai/models/xai/grok-stt/) diarize: true per word word-level HTTPS URL or base64 data URI; 25 MB direct-upload limit, none for URL fetch $0.10/h REST, $0.20/h streaming (https://docs.x.ai/docs/models) Cheapest diarizing option; keyterm array (100 terms), multichannel 2-8 ch; 25 languages; ZDR. Streaming through Cloudflare is documented: a websocket boolean input "establishes a bidirectional WebSocket connection for real-time audio transcription", mutually exclusive with file/url (untested). Newer, less track record.
openai/gpt-4o-transcribe (https://developers.cloudflare.com/ai/models/openai/gpt-4o-transcribe/) no not exposed data URI or HTTPS URL; formats flac/mp3/mp4/mpeg/mpga/m4a/ogg/wav/webm $0.006/min = $0.36/h (https://developers.openai.com/api/docs/pricing) OpenAI's gpt-4o-transcribe-diarize ($0.36/h) is not in the Cloudflare catalogue as of today.
ElevenLabs Scribe v2: up to 32 speakers, word timestamps, 3 GB / 10 h files, realtime ~150 ms (https://elevenlabs.io/docs/capabilities/speech-to-text) yes — $0.22/h batch, $0.39/h realtime (https://elevenlabs.io/pricing/api) Not in the Cloudflare ASR catalogue; AI Gateway's ElevenLabs provider only documents text-to-speech (https://developers.cloudflare.com/ai-gateway/usage/providers/elevenlabs/). Would need direct API through the gateway's generic passthrough.
Deepgram direct via AI Gateway as nova-3 yes replace https://api.deepgram.com/ with https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/deepgram/; WebSocket supported (https://developers.cloudflare.com/ai-gateway/usage/providers/deepgram/) $0.258/h batch; streaming $0.288/h at the current promo, $0.462/h regular Same model as 3.3, own Deepgram account, ~17% cheaper on batch and regular-price streaming (~48% on streaming while the promo lasts), keeps logging/caching in the gateway.

3.6 Browser Web Speech API (what the reference uses live)

Source: https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API/Using_the_Web_Speech_API and https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition/start

3.7 Cloudflare Realtime family (live transport, not STT itself)


4. Comparison table

Costs are per hour of audio. "CF-native" = billed in neurons through the Workers AI binding with no extra vendor account.

Option Diarization (R1) Utterances + timestamps (R2) Long files (R3) UK English / vocab (R4) Cost/h (R5) Live (R6) CF-native (R7)
@cf/openai/whisper-large-v3-turbo No segments ~5 s + words No (~1 MB/base64 chunks, undocumented) language=en, initial_prompt; hallucination knobs $0.031 No (batch only) Yes
@cf/openai/whisper / -tiny-en No words only No none $0.027 / n/a No Yes
@cf/deepgram/nova-3 HTTP Yes (diarize) + multichannel utterances[] with speaker, words Yes (binary body, ≤100 MB Worker limit; 2 GB at Deepgram) en-GB, keyterm, mode=finance, numerals $0.312 — Yes
@cf/deepgram/nova-3 WebSocket Yes (speaker, no confidence) is_final, speech_final, utterance_end_ms streaming same $0.552 Yes (latency unmeasured) Yes (binding or AI Gateway)
@cf/deepgram/flux WS No turn events, no word times streaming keyterm $0.462 Yes Yes
assemblyai/universal-3.5-pro Yes utterances, ms words URL/data URI language_code ~$0.23 (list) no CF streaming Gateway credits/BYOK
xai/grok-stt Yes words ≤25 MB direct, URL unlimited keyterm ×100 ~$0.10 (list) Yes (documented websocket: true param; untested) Gateway credits/BYOK
openai/gpt-4o-transcribe No no URL/data URI prompt $0.36 (list) No Gateway credits/BYOK
Deepgram direct via AI Gateway Yes yes 2 GB same as nova-3 $0.258 / $0.288 promo ($0.462 regular) Yes BYOK
ElevenLabs Scribe v2 Yes (32 spk) yes 3 GB English $0.22 / $0.39 Yes Not in catalogue
Browser Web Speech API No no timestamps n/a lang=en-GB, phrases $0 Yes (Chrome) n/a; audio goes to Google

5. Recommendation

5.1 PoC (mode A: upload + timed replay)

Use @cf/deepgram/nova-3 over HTTP, one request per file, with: diarize=true&utterances=true&punctuate=true&smart_format=true&language=en-GB&numerals=true&mode=finance&keyterm=<FX terms>&mip_opt_out=true (mip_opt_out may change Deepgram's pricing per their note; check https://dpgr.am/deepgram-mip.)

Rationale: it is the only Cloudflare-hosted model that returns speaker-labelled utterances with timestamps, it accepts the whole file as a binary body so there is no chunking pipeline to build for the PoC, and $0.31/h means a 30-minute call costs ~15 cents (about 14,000 neurons, i.e. over the free daily 10k for the second call of the day). Map Deepgram utterances directly onto the reference's replay format: {t: u.start, t_end: u.end, speaker: map[u.speaker], text: u.transcript} and reuse its fireAt = t_end scheduler. Regroup consecutive same-speaker utterances into turns (Deepgram's utt_split defaults to ~0.8 s, so a rep monologue arrives as several utterances).

Speaker 0/1 -> rep/client mapping: no STT will tell you which is which. Cheap heuristics: the rep speaks first on outbound onboarding calls; the rep says "CurrencyTransfer"; or ask Jev one Choice question on the first two turns ("which speaker is the company representative?") and let the admin override. Mark as a design decision, not an STT problem.

Do these tests before writing any product code (each is one request against POST /accounts/{id}/ai/run/@cf/deepgram/nova-3 or the Workers binding): 0. Establish the request shape, which is UNVERIFIED (§3.3): try (a) a raw binary body with Content-Type: audio/mpeg via --data-binary, options as query parameters, and (b) JSON {audio: {body, contentType}, diarize: true, utterances: true} via the binding; record which works and use it for tests 1-3.

  1. Send scratchpad/video/audio.mp3 (81 s, two speakers) with diarize=true&utterances=true and confirm the JSON contains results.utterances[] and words[].speaker. If Cloudflare strips them (its published schema omits them), fall back to xai/grok-stt or assemblyai/universal-3.5-pro via gateway credits.
  2. Send a real 30-40 min CT recording (20-60 MB) to find the practical body-size ceiling on the Cloudflare proxy; if it fails, use the Batch API (queueRequest: true, but that caps payloads at 10 MB) or chunk at silence boundaries.
  3. Inspect whether CT recordings are stereo with one party per channel; if so run multichannel=true instead of diarize=true and the attribution problem disappears.

I could not run these myself: the earlier whisper test's CLOUDFLARE_API_TOKEN was in the parent session's environment, and ~/.config/jev/cloudflare.env (the location scripts/jev.mjs reads) does not exist on this box.

Keep @cf/openai/whisper-large-v3-turbo as a secondary, text-only path for bulk-indexing the historical corpus where you only need searchable text (10x cheaper), and for cross-checking nova-3's wording on FX terms. Set vad_filter=true, condition_on_previous_text=false, hallucination_silence_threshold=2, and initial_prompt with the FX glossary. Do not use it for anything that needs to know who spoke.

5.2 Live (mode B)

Use @cf/deepgram/nova-3 over WebSocket from a Durable Object (one DO per live call holding the socket, the KeepAlive timer and the running transcript), with encoding=linear16&sample_rate=16000&interim_results=true&endpointing=300&utterance_end_ms=1000&diarize=true&language=en-GB&keyterm=…. Fire a Jev evaluation on each is_final/speech_final result exactly where the reference fires on t_end. Budget $0.55/h on Cloudflare, or direct-Deepgram-through-AI-Gateway with BYOK at $0.29/h at the current promo ($0.46/h at regular price). If nova-3 streaming through the binding falls short, xai/grok-stt documents a websocket: true streaming mode (untested) and is the cheapest diarizing fallback.

Prefer separate audio streams per party over diarization whenever the transport allows it: rep mic on one socket (or channel), remote party on the other, so attribution is structural rather than statistical. That is what RealtimeKit does internally and what the WebSocket media adapter enables for WebRTC. Which transport you get depends entirely on where the calls live, which is the main open question.

Flux is not needed: its end-of-turn machinery solves the "when should the agent speak" problem we do not have, and it lacks diarization.

5.3 What to decide next

  1. Where do live calls happen (dialer / softphone / Zoom / Teams / mobile)? That picks the transport: dialer media stream (Twilio-style) -> @cloudflare/voice Twilio/Plivo adapter (select WorkersAINova3STT explicitly; WorkersAIFluxSTT is the default and has no diarization); browser softphone -> getUserMedia + getDisplayMedia({audio:true}) or WebRTC remote track -> DO -> nova-3; CT-hosted room -> RealtimeKit with built-in per-participant transcription.
  2. Recording format on the call-coach server (mono vs dual-channel, mp3 vs m4a, typical length) — decides multichannel vs diarize and whether whisper chunking is ever worth building.
  3. Data handling: Deepgram's model-improvement program (mip_opt_out), the Zero data retention flag on third-party catalogue models, and whether client call audio may transit Google (Web Speech) at all.
  4. Budget sanity: 1,000 historical hours through nova-3 HTTP ≈ $312 on Cloudflare vs ≈ $258 direct; through whisper-turbo ≈ $31 but without speakers.

6. Sources

Cloudflare

Deepgram

Others

Live verification (2026-09-24, added after the research pass)

Run by Claude with the CT account token (~/.config/jev/cloudflare.env):

3,877 words · Internal working documents. Do not share outside CurrencyTransfer.