105 lines
11 KiB
Markdown
105 lines
11 KiB
Markdown
# AGENTS.md
|
|
|
|
This is a LiveKit Agents project: a **realtime voice AI mock interviewer** — the AI plays the interviewer and conducts a two-stage interview (self-introduction, then past experience). LiveKit Agents is a Python SDK for building voice AI agents, used here with LiveKit Cloud. See @README.md for more about the rest of the LiveKit ecosystem.
|
|
|
|
The following is a guide for working with this project.
|
|
|
|
## Goal
|
|
|
|
Build a demo that:
|
|
1. Implements the **self-introduction** and **past-experience** stages using a LiveKit multi-agent design, with stable performance and clear interaction logic.
|
|
2. Transitions between the two stages smoothly and naturally — well-defined switching logic, no interruptions, conflicts, or repeated prompts.
|
|
3. Provides a **time-based fallback** so the interview always advances even when the normal (LLM-driven) switch does not fire.
|
|
|
|
Reference framework: https://github.com/livekit/agents
|
|
|
|
## Environment
|
|
|
|
- WSL2 (Ubuntu); Python 3.11.
|
|
- Package manager: **`uv`** (see Project structure). Dependencies are declared in `pyproject.toml`.
|
|
- Secrets live in `.env.local` (never commit). The starter uses **LiveKit Inference** for all models, so the only required keys are `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET`. Provider keys (`DEEPGRAM_API_KEY`, `OPENAI_API_KEY`, `CARTESIA_API_KEY`) are needed **only** if you switch a model from `inference.*` to a direct plugin.
|
|
|
|
## Project structure
|
|
|
|
This Python project uses the `uv` package manager. You should always use `uv` to install dependencies, run the agent, and run tests.
|
|
|
|
All app-level code is in the `src/` directory. In general, simple agents can be constructed with a single `agent.py` file. Additional files can be added, but you must retain `agent.py` as the entrypoint (see the associated Dockerfile for how this is deployed).
|
|
|
|
Be sure to maintain code formatting. You can use the ruff formatter/linter as needed: `uv run ruff format` and `uv run ruff check`.
|
|
|
|
## Architecture (target)
|
|
|
|
The starter's `src/agent.py` is the baseline. Build on its **real** structure:
|
|
|
|
- Models use **LiveKit Inference** (routed through LiveKit Cloud — no separate provider keys): `inference.LLM(model="openai/gpt-5.2-chat-latest")`, `inference.STT(model="deepgram/nova-3", language="multi")`, `inference.TTS(model="cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc")`. (gpt-4.1-mini is a cheaper/faster LLM swap if latency or cost bites.)
|
|
- `silero.VAD` is **prewarmed** in `prewarm(proc)` and stored at `proc.userdata["vad"]` (process-level), then read as `ctx.proc.userdata["vad"]`. Also: `turn_detection=MultilingualModel()`, `preemptive_generation=True`, and noise cancellation via `room_options=room_io.RoomOptions(audio_input=room_io.AudioInputOptions(noise_cancellation=ai_coustics.audio_enhancement(...)))`.
|
|
- One `AgentSession[InterviewData]` carries the shared STT/TTS/VAD/turn-detection/LLM defaults plus `userdata=InterviewData()` (session-level state).
|
|
- Two stage agents — `SelfIntroAgent` -> `PastExperienceAgent` — are `Agent` subclasses, each with its own `instructions`, `on_enter`, and a completion `@function_tool`. They inherit the session's models (override the LLM per-agent only if a stage needs a different one, as LiveKit's `multi_agent` example does). Note: the starter sets the LLM on the `Agent`; for two agents, lift it onto the session so both stages share it.
|
|
- Normal transition = the completion tool returns `(NextAgent(chat_ctx=self.chat_ctx.copy()), "<line>")`.
|
|
- Time-based fallback = a per-stage wall-clock watchdog that forces `session.update_agent(...)`.
|
|
- Run pattern: `AgentServer()` + `server.setup_fnc = prewarm` + `@server.rtc_session(agent_name="interview-agent")` + `cli.run_app(server)`; `await ctx.connect()` comes **after** `session.start(...)`.
|
|
- New imports the refactor needs (not in the starter by default): `function_tool`, `RunContext`, `ChatContext` from `livekit.agents`; `TurnHandlingOptions`, `UserTurnLimitOptions` for the fallback; `dataclass`, `field`.
|
|
|
|
Verify exact API names via the LiveKit docs (below) before coding. This follows LiveKit's handoffs-and-tasks model.
|
|
|
|
## Handoffs and tasks ("workflows")
|
|
|
|
Voice AI agents are highly sensitive to excessive latency. For this reason, it's important to design complex agents in a structured manner that minimizes the amount of irrelevant context and unnecessary tools included in requests to the LLM. LiveKit Agents supports handoffs (one agent hands control to another) and tasks (tightly-scoped prompts to achieve a specific outcome) to support building reliable workflows. You should make use of these features, instead of writing long instruction prompts that cover multiple phases of a conversation. Refer to the [documentation](https://docs.livekit.io/agents/build/workflows/) for more information.
|
|
|
|
## Operating rules (IMPORTANT)
|
|
|
|
- **Verify before coding.** Confirm every LiveKit class / method / keyword argument against the LiveKit docs (CLI or MCP, see below) before writing it. This framework moves fast — do not rely on memory for API names.
|
|
- **Stay in scope.** Implement only what the current prompt asks. Do not scaffold ahead.
|
|
- **Do NOT implement the time-based fallback (goal 3).** The developer hand-writes that part. Touch it only if explicitly asked, and then only to verify API names — never to author the logic.
|
|
- **Explain your work.** After any substantive change, walk through how it behaves (especially the handoff and any timing) so the developer can verify it.
|
|
|
|
## Known gotchas (verified against LiveKit issues / docs)
|
|
|
|
- **Two different "userdata".** `proc.userdata` (a dict, process-level) holds the prewarmed VAD; `session.userdata` (our `InterviewData` dataclass, session-level) holds interview state. They are not the same object — don't put `InterviewData` on the process, or look for the VAD on the session.
|
|
- VAD is required, or `user_state` events never fire (issue #2364). The starter already prewarms it.
|
|
- New agents start with a **fresh chat history**; pass `chat_ctx` on both tool-handoff and `update_agent`, or the candidate is forced to repeat themselves.
|
|
- A handoff tool must return **only** `(next_agent, line)` — no other reply-producing tool in the same turn, or the handoff races and repeats (issue #5150).
|
|
- Forced transitions use `session.say(...)` (deterministic), never `generate_reply` (it wanders), and the fallback timer must be a standalone wall-clock task — **not** tied to `agent_state_changed`, which would make it cancel itself (issue #3148).
|
|
- The starter sets `preemptive_generation=True` as a flat session kwarg. If you add `turn_handling=TurnHandlingOptions(...)` for the fallback's user-turn limit, verify (don't guess) whether `preemptive_generation` must move inside `TurnHandlingOptions` — some flat turn kwargs were deprecated in favor of `turn_handling`.
|
|
- The starter ships `ai_coustics` noise cancellation. If startup errors on missing ai-coustics credentials in local dev, configure its key or temporarily comment out the `noise_cancellation=...` line.
|
|
|
|
## Testing
|
|
|
|
When possible, add tests for agent behavior. Read the [documentation](https://docs.livekit.io/agents/start/testing/), and refer to existing tests in the `tests/` directory. Run tests with `uv run pytest`.
|
|
|
|
Important: When modifying core agent behavior such as instructions, tool descriptions, and tasks/workflows/handoffs, never just guess what will work. Always use test-driven development (TDD) and begin by writing tests for the desired behavior. For instance, if you're planning to add a new tool, write one or more tests for the tool's behavior, then iterate on the tool until the tests pass correctly. This will ensure you are able to produce a working, reliable agent for the user.
|
|
|
|
For this project, the **highest-priority test is the self-intro -> past-experience handoff**: assert that `intro_complete` fires once, transfers control to `PastExperienceAgent`, and preserves context. Use LiveKit's test framework (`result.expect`, `judge()`).
|
|
|
|
## LiveKit Documentation
|
|
|
|
LiveKit Agents is a fast-evolving project, and the documentation is updated frequently. You should always refer to the latest documentation when working with this project. For your convenience, LiveKit offers both a CLI and an MCP server that can be used to browse and search its documentation. If the developer has not yet installed the CLI, you should recommend that they install it.
|
|
|
|
### LiveKit CLI
|
|
|
|
The [LiveKit CLI](https://docs.livekit.io/intro/basics/cli/) `lk docs` subcommand gives full access to LiveKit documentation from the terminal. Requires CLI version 2.15.0+. Check with `lk --version`.
|
|
|
|
Install or update the CLI:
|
|
|
|
- **macOS:** `brew install livekit-cli` (update: `brew update && brew upgrade livekit-cli`)
|
|
- **Linux:** `curl -sSL https://get.livekit.io/cli | bash`
|
|
- **Windows:** `winget install LiveKit.LiveKitCLI` (update: `winget upgrade LiveKit.LiveKitCLI`)
|
|
|
|
Run `lk docs --help` to see all available commands. Key commands: `lk docs overview`, `lk docs search`, `lk docs get-page`, `lk docs code-search`, `lk docs changelog`, `lk docs submit-feedback`. Run `lk docs <command> --help` before using a command for the first time, as the help text includes advice on usage and interpreting results. Prefer browsing (`overview`, `get-page`) over search, and prefer `search` over `code-search`, as docs pages are more up-to-date and provide better context than raw code.
|
|
|
|
### LiveKit Docs MCP Server
|
|
|
|
For deeper IDE integration, LiveKit also offers an [MCP server](https://docs.livekit.io/reference/developer-tools/docs-mcp/) with the same documentation access. It uses Streamable HTTP transport at `https://docs.livekit.io/mcp`. Refer to your IDE's documentation for how to add it.
|
|
|
|
### Submit docs feedback to LiveKit
|
|
|
|
If you use the LiveKit CLI or MCP Server to search or browse documentation, also submit constructive feedback to LiveKit with `lk docs submit-feedback` or the MCP `submit_docs_feedback` tool. Add this to your to-do list when planning work, to ensure you file feedback after completing your other tasks. Useful feedback includes incomplete documentation, broken examples, missing steps or any other documentation gap or issue.
|
|
|
|
## LiveKit CLI
|
|
|
|
Beyond documentation access, the LiveKit CLI (`lk`) supports other tasks such as managing SIP trunks for telephony-based agents. Run `lk --help` to explore available commands.
|
|
|
|
## Phasing — not yet in scope
|
|
|
|
The current phase is the **foundational framework** (goals 1-3 above). Do not build the following until the foundation is complete and the developer explicitly asks; if a request seems to call for one of these, confirm scope first: video avatar (the starter stubs `anam`), post-call scoring/report, recording (egress), resume/JD RAG, and any additional interview stages (behavioral, technical). When stages grow beyond two, reconsider the supervisor / task-group pattern instead of pairwise handoffs.
|