The architecture behind AI Tutor Call

AI Tutor Call is a tutor you can call on the phone. You tap call, the tutor picks up, and for the next 10 minutes you can talk about whatever interests you. The tutor corrects your mistakes, helps you in your native language when you get stuck, remembers your interests, and helps you gain confidence. AI Tutor Call launched this week for English and Spanish learners.

The first prototype of AI Tutor Call was just a speech-to-speech model that handled everything with one prompt, built in a single day in December 2025. The version running today is quite different: the speech model only handles speaking. Listening, orchestration and side effects all happen in code, before the model starts talking.

This post covers why the one-prompt version broke, what replaced it, and what it bought us, including a turn-taking model we trained on 44,586 real calls without any hand-labeling.

How we started

Learning a language by phone is a well-known format in Korea: call a tutor at a set time and practice speaking. This was a common feature request from our customers, so we set out to build it with an AI tutor.

Speech-to-speech models seemed promising for our use case. You send audio in and get audio out, and the model takes care of everything in between: listening, thinking, judging, calling tools, and then speaking back in a voice that sounds like a person.

Our first prototype of AI Tutor Call was just that: a realtime speech-to-speech model connected to a LiveKit room, with a prompt to handle everything. It worked on the first try: the voice was natural and the tutor switched between English and Korean seamlessly. Everyone who tried the demo thought that the feature was almost done.

Why one model can’t do everything

The problems started when we tried to turn that demo into a real feature. We tried to get the tutor to do more things: save a learner’s mistakes so they can practice tomorrow, persist facts about the learner in memory, end the session at a set time. All of these were tool calls, and we just could not get the model to do tool calls reliably. Sometimes they did not fire at all, or they fired at random times, or they fired with hallucinated arguments.

As the prompt grew, it also became less and less reliable. We added instructions for corrections, tone and safety. Each instruction worked when we added it. Then a small wording change to one would break several others.

Our first fix was to split the call into phases, and split the prompt with it. Each phase got its own prompt, and the model moved between them with a tool call. For about two weeks things looked good, and then the same problems started coming back. The rules that applied to the whole call had to be repeated in every phase, bloating the prompt once more. Each switch gave the model a new prompt but kept the whole conversation so far, so details from the setup phase leaked into the lesson and threw it off. And remember, the switch itself was a tool call, so something we already knew to be unreliable was now holding the whole thing together. Within a month we realized phase switching was not the answer.

We learned a valuable lesson from this experiment: splitting the prompt is not the same thing as splitting the responsibilities. One model getting six small prompts is still one model that handles everything. The model is good at speaking, but we were asking it to be a database client, an orchestrator and a policy engine all at the same time, and we were doing it all through prose.

The split: listen, judge, speak

We built a new pipeline around one idea: the model is good at speaking, so let’s have it only do the speaking. Everything that the tutor needs to do is handled separately.

A turn has three owners now:

1. Listening: We run a speech recognition model to transcribe the learner’s speech and determine when their turn is done.
2. Judging: Several small LLMs read the transcript and make decisions: was there a mistake in the learner’s speech? Should we correct the mistake? Did the learner say anything against our policy?
3. Speaking: The speech-to-speech model receives the verdict rendered as a short labeled block in its instructions, and has nothing left to decide.

Now listening, judging and speaking no longer interfere with each other, and each piece can be tuned independently. The speaking model has no tools at all: saving a correction, writing a memory, and ending the call all run in code.

Judging

We instructed the tutor to correct mistakes in the learner’s speech. Initially, the tutor’s false positive rate was about 25%, frustrating learners by correcting their perfectly fine sentences.

Once we moved the judging of a learner’s speech into a separate assessment prompt, we were able to easily add gates to the judge. We don’t correct when the transcription confidence is low, or if the user may be speaking in their native language. After a few rounds of work, the false positive rate dropped from 25% to 2%.

None of this is in the prompt. The model never sees "do not correct this mistake"; the correction block is simply not there. Now that these rules are outside of the prompt, each can be tested on its own, put behind a feature flag, rolled out and rolled back without touching anything else.

Speaking

Most LLM applications set the system prompt once and let the chat history grow. We do it the other way around. Before every reply, the prompt is rendered again from the state of the lesson at that moment: the static part, which says who the tutor is and how it talks, plus a few short labeled blocks that are only there when they apply, like a correction that the LLM judge decided should be given, or a scripted goodbye at the end of the call.

Listening: our own turn detection model

Every turn starts with this decision: is the learner done talking? If we’re too early, then the tutor cuts the learner off. Too late, and the experience feels laggy and slow. The core issue is that turn detection models are built for native-speaker conversational patterns, and our learners are not native speakers. They stop in the middle of a sentence while they look for a word, and a detector trained on native speech declares they are done.

The realtime model's own turn detection interrupted learners constantly. Turned down far enough to stop interrupting, it left learners waiting up to seven seconds for a reply.

So we decided to train our own turn detector. As we dug into the data, we hit an unexpected break: our production data was already labeled for training! Every call leaves a recording and an event log. The log says when we declared an endpoint, when the learner spoke over it, and when the tutor was able to speak without being interrupted. The learner's behavior is the label. The cases where we said the learner was done, and they did not start again, are positive examples. When we said the learner was done and they kept talking, that was a false endpoint (a negative example). That second label is the one no public dataset has, because you can only collect it by running a real turn detector on real learners and recording where it was wrong. Half of those false endpoints read as finished sentences on paper, like "I'm good with both | No fixed preference". Only the way it was said tells you the learner was not done.

We took every call since the event log started, 44,586 calls from July to September, which gave us about 500,000 examples from 13,000 learners, and fine-tuned a small model with 8M parameters that listens to the last eight seconds of audio and returns one probability on whether the learner is done. After this went live, replies the tutor had to abandon because the learner was still talking dropped by 69%.

What the split architecture bought us

With the split architecture, we are able to make isolated changes to individual components. Recently we decided to try a different realtime model. It talks quite differently, so it needed its own version of the conversation prompt, but corrections, drills, turn taking and safety were not touched at all. We were able to run a test that only compared how the tutor talks. In the original architecture this test would have involved a full rewrite. With the new architecture it was a few days of work.

New behavior is easy to add too, either as a new flow or as a new gate in front of an existing judge. Each new change can be introduced behind its own flag and with its own scenarios in the evaluation suite. When we build new judges, we can run them in the background on real production data, so we can see how they would act before letting them change a single reply.

In the original architecture, the speech-to-speech model was a black box. Now, when something does go wrong, it’s easy to figure out where. The transcript contains what the tutor heard, and the event log tells us which verdict was computed, which rules downgraded it, which action was being taken and which prompt version was used.

If this is the kind of problem you want to work on, we are hiring! speak.com/careers

‍