Build a $5,000 AI receptionist in a weekend
Claude Code × Fish Audio
Hey, it's Adam. You heard "Sarah" in the reel — the receptionist who answers, books, and de-escalates angry callers, and doesn't exist. This is the exact build: Claude Code wrote the entire app from one prompt, a free local model runs her brain, and Fish Audio is her voice. The whole thing took a weekend, and the master prompt is below, word for word. Fish Audio partners with me on this content — and the workflow is still the real one I'd use without them. You commented AGENT, so let's build.
A phone call is just a pipeline ☎️
Strip the magic away and a receptionist does four things: hears you, thinks, speaks, and stays in character. That's the build:
- 🎤 Ears — your browser's built-in speech recognition. Free, no API key, already on your machine.
- 🧠 Brain — a small local model (llama3.2 via Ollama) running on your laptop. Also free, also keyless. It answers only from a facts file you control.
- 🗣 Voice — Fish Audio's TTS API. The only API key in the entire project. This is where believable lives: the reply comes back fast enough to kill the awkward AI pause, and it takes emotion cues — [warmly], [calmly] — so Sarah sounds like she's actually listening.
- 🛠 The builder — Claude Code. It doesn't run the receptionist; it writes her. Logic, routing, UI, all of it, from the one prompt below.
One key, one weekend, one pipeline. That's the whole product.
What you need ✅
- Claude Code — comes with a Claude paid plan (Pro is ~US$20/mo).
- Node 20+ and Ollama — both free. Run
ollama pull llama3.2once and leave Ollama running. - A Fish Audio account + API key — the S2.1 Pro model is free for developers on the
s2.1-pro-freetier right now, but that window runs through August 31, 2026 (their stated date, with notice before changes). After that, check their pricing page before you build a business on it. No SLA on the free tier — fine for demos, not a guarantee for production. - Chrome, a mic, and a weekend. Truthfully: the first working call takes an hour or two. The weekend is for polish.
One prompt. The entire app. 📋
Open Claude Code in an empty folder and paste this. It's long on purpose — every line below is a decision you won't have to debug later.
Build me a local, browser-based voice agent: an AI phone receptionist named "Sarah" for a fictional medical clinic ("Riverside Family Clinic"). It should feel like a real customer-service phone call. Single-language project (Node.js), plain HTML/CSS/JS frontend — no frameworks, no build step.
=== ARCHITECTURE (the pipeline) ===
Browser mic → Speech-to-Text → LLM (brain) → Text-to-Speech → playback
- Ears (STT): the browser's built-in Web Speech API (webkitSpeechRecognition). Free, no key.
- Brain (LLM): local Ollama via its HTTP API at http://localhost:11434, model "llama3.2". Keyless. Use POST /api/chat with stream:false.
- Mouth (TTS): Fish Audio API (https://api.fish.audio/v1/tts). The ONLY thing needing a key. Read it from .env; NEVER hardcode it.
=== SERVER (Node + Express, port 3200) ===
Start script: "start": "node --env-file-if-exists=.env server.js"
Serve /public statically, plus three endpoints:
- GET /api/config -> { serverTts: <true if FISH_AUDIO_API_KEY is set> }
- POST /api/chat -> takes { messages:[...], loud:<bool> }, calls Ollama, returns { reply, speech, mood }
- POST /api/tts -> takes { text }, calls Fish Audio, streams back audio/mpeg.
If no key is configured, return HTTP 204 so the frontend falls back to the browser's built-in voice.
Fish Audio call specifics (IMPORTANT):
- POST https://api.fish.audio/v1/tts
- Headers: Authorization: Bearer <FISH_AUDIO_API_KEY>, Content-Type: application/json, and a header named "model" that selects the TTS model.
- Body: { text, reference_id: <FISH_AUDIO_VOICE_ID>, format: "mp3" }
- Env vars with defaults: FISH_AUDIO_API_KEY (blank = browser fallback), FISH_AUDIO_VOICE_ID default "933563129e564b19a115bedd57b7406a" (the "Sarah" voice), FISH_AUDIO_MODEL default "s2.1-pro-free", OLLAMA_MODEL default "llama3.2", PORT default 3200.
=== THE BRAIN: mood-aware JSON replies ===
The clinic's entire knowledge lives in one system prompt (hours, address, services, booking, insurance, refills, test results — all made up). Sarah is a warm, young receptionist. Call Ollama with format:"json" and instruct it to reply ONLY with: { "mood": "<neutral|happy|confused|frustrated|angry>", "reply": "<short 1-2 sentence spoken reply>" }.
- The reply includes one or two EMOTION MARKERS in square brackets for the voice engine, e.g. "[warmly] Hi there! [cheerfully] How can I help?" (Fish Audio's S2.1 supports [bracket] emotion cues.)
- TONE MATCHING: if the caller is angry/frustrated, Sarah DROPS the cheerfulness — apologize sincerely, slow down, de-escalate: [calmly], [apologetically], [empathetically]. If neutral/happy, be warm.
- Server post-processing: `speech` = the reply WITH brackets (sent to Fish Audio). `reply` = the same text with all markers stripped (shown on screen).
- Only answer from the clinic facts; for medical advice or records, say you can't help and offer the front desk; for emergencies, tell them to hang up and call 911.
=== ANGER DETECTION (two signals) ===
1. Words: the LLM classifies mood from the transcript.
2. Voice: measure mic volume via the Web Audio API. If the caller's peak volume exceeds a threshold, send loud:true, and the server appends "(the caller's voice is raised — they may be upset)" to the user message.
Show the detected mood as a colored "Caller: <mood>" chip (red + pulsing when angry).
=== THE "REAL PHONE CALL" FRONTEND (public/app.js) ===
A hands-free conversation loop, NOT tap-per-question:
- One big central call button. Tap = Sarah greets first, then continuous listening (continuous:true, interimResults:true). Tap again to hang up. Auto-return to listening after each reply; keep a short rolling history.
- BARGE-IN (interrupt her): the caller can talk over Sarah and she stops instantly. Make it work over laptop SPEAKERS, not just headphones: capture a SECOND mic stream via getUserMedia with echoCancellation, noiseSuppression and autoGainControl — used only as a volume meter. While Sarah speaks, IGNORE the recognizer (its stream has no echo cancellation and would transcribe Sarah herself); detect barge-in from the echo-cancelled meter (sustained spike = caller talking) → stop her audio, flush the recognizer.
- iOS AUDIO UNLOCK (critical, or iPhone stays silent): on the Start-call tap, synchronously unlock ONE reusable Audio element (play a tiny silent clip), then reuse that element for every reply via .src. Also resume the AudioContext.
- If Fish Audio isn't configured (/api/tts returns 204) or a TTS call fails, gracefully fall back to the browser's SpeechSynthesis with the clean text.
=== UI (make it look like a real product) ===
Dark, elegant, mobile-first, safe-area padding, big tap targets. Clinic SVG logo + "Riverside Family Clinic" wordmark + "● Virtual Receptionist · Sarah" tagline with a pulsing online dot. The big (~168px) call button has animated states: idle (breathing mic glyph, radar rings), listening (red, live equalizer driven by real mic volume), thinking (dimmed), speaking (green/teal equalizer). Chat transcript bubbles (caller right, Sarah left). Footer safety pill: "For medical emergencies, call 911." Respect prefers-reduced-motion.
=== FILES ===
server.js, clinic-faq.js (facts + system prompt), tts.js (swappable), package.json, .env.example, .gitignore, README.md, public/index.html, public/style.css, public/app.js.
README setup: Node 20+; Ollama running with llama3.2 pulled; npm install; npm start; open http://localhost:3200 in Chrome; allow the mic. Works with NO keys using the browser voice; add FISH_AUDIO_API_KEY for the real Sarah voice. Note: the mic only works on localhost or HTTPS — to test on a phone, expose it with an HTTPS tunnel (e.g. cloudflared). Plain http://<LAN-ip> won't grant mic access on mobile.
Verify the whole pipeline works before finishing. Build it now.Why this prompt works — steal the shape, not just the text: it names the architecture before any code, so nothing gets invented. It hands over the exact API contract — endpoint, headers, body — so the integration can't be hallucinated. It specs the two places voice demos always die (talking over the agent, and iPhones staying silent) instead of hoping. And it demands a fallback path plus a final verify pass. Write every Claude Code prompt like a one-page PRD and weekend builds stop feeling like gambling.
Test: when Claude Code finishes, run npm start, open localhost:3200, tap the button and talk. Sarah will answer in the browser's robot voice. That's correct — the pipeline works before the key exists. The robot voice is your before.
Give her the voice 🎙
Get your Fish Audio API key — free tier available now
- 1Sign up at fish.audio and create an API key.
- 2
cp .env.example .env, paste the key intoFISH_AUDIO_API_KEY. - 3Keep the default voice ID for Sarah, pick any voice from their library, or clone one — cloning is included on the free tier.
- 4Restart the app.
Test: ask her the same question you asked in robot-voice mode. Same words, different species. That before/after is also exactly how you should film it.
The emotion markers are why she doesn't read like a TTS engine: the model writes [warmly] or [apologetically] into her lines based on the caller's mood, and the voice actually performs it.
Stress-test the call 🧪
Three tests before you show anyone:
- Interrupt her. Talk over the middle of her sentence, on laptop speakers. She should stop instantly. This is the single most "she's real" moment in a demo.
- Get angry on purpose. Raise your voice, complain about a bill. Watch the mood chip go red — and her tone drop from cheerful to [calmly], because both signals fired: your words and your volume.
- Ask what she shouldn't answer. Medical advice, test results. She should decline and offer the front desk. If she freelances, tighten the system prompt in clinic-faq.js.
Phone test: run an HTTPS tunnel (cloudflared) and open it on your iPhone — the mic won't work over plain local IP, and audio only unlocks on that first tap. Both are already handled in the build.
Make her someone's receptionist 🏪
Everything Sarah knows lives in one file: clinic-faq.js. Swap the fictional clinic's facts for a real business — hours, services, booking rules, FAQs — and she's their receptionist in ten minutes. Keep the guardrails exactly as they are: answers only from the facts file, no advice she isn't qualified to give, emergencies escalate to a human. The guardrails aren't the boring part. They're the sellable part.
🔑 What keeps this believable (and safe)
- 1The agent answers from the facts file. Everything else gets a polite "let me take your details."
- 2One API key, in
.env, never in code — and never in a screen recording. - 3Demo on speakers, not headphones. Barge-in working over speakers is the proof.
- 4Film the demo output-first: her voice before your face. The reel already proved this order works.
- 5Emergencies always route to a human. That line is non-negotiable in anything medical.
What this is — and isn't ⚠️
- It's browser-based. Sarah answers in a web app, not on an actual phone number. Putting her on a real line means adding a telephony layer (Twilio or similar) — that's the natural next build, and the upsell.
- The demo clinic is fictional. Riverside Family Clinic exists so you can demo without a client. When you pitch, you swap in the prospect's real FAQs — that personalised demo is the close.
- About the $5,000: that's the market rate agencies charge for custom voice-agent builds of this scope — it's what this sells for, not a promise of what you'll earn. Your price depends on your market, your client, and what you include. Ranges, not promises.
- The free tier has a date. Fish Audio's
s2.1-pro-freeAPI access runs through August 31, 2026, no hard character cap but fair-use rules, and no uptime guarantees. A paid client build should be quoted on paid-tier pricing — never quote a client on a promo window. - A local llama3.2 is a small brain. It occasionally answers slowly or oddly on weak hardware. Swap
OLLAMA_MODELfor a bigger local model, or point the same endpoint shape at a hosted model when a client's paying.
From demo to product 💼
Missed calls are lost revenue for small businesses — that's the pitch, and it needs no invented statistics. The package that works: a setup fee for the build and voice, plus a small monthly retainer for hosting, updates, and FAQ changes. Start with one local business you already know, load their real FAQs into the facts file, and let them call Sarah about their own shop. Nobody buys a slide deck after they've talked to their own receptionist.