You don’t know what you know until you’re asked. The useful part of a story is often the detail you no longer think to volunteer: the objection you quietly worked around, the odd decision which saved the project, the experience that explains why you distrust the obvious answer.
I wanted to build a tool which could help me recover those thoughts and turn them into something worth writing. Interviewer is a fourteen-minute voice conversation backed by several specialised agents, but I designed it to feel like one attentive person who understands the subject and knows when to draw out an unfinished idea. The transcript and eventual draft matter because of the conversation which produced them.
Nothing punctures that feeling quite like being asked the same question twice. In ordinary conversation it tells you the other person stopped listening. In a short interview it also spends time which could have gone to the surprising aside or unresolved contradiction at the heart of the piece.
Avoiding repetition led me to a deceptively awkward question: when has an interviewer actually asked something?
Let the wording follow the conversation
A good interviewer does not recite a questionnaire. They adapt the question they meant to ask to the answer they have just heard. I wanted the speaking model to have that freedom without losing the purpose I had put into the interview plan.
So I separated a question’s identity from the words used to ask it. Each planned question carries a stable ID, and the Orchestrator can return that ID with whatever phrasing makes sense in the conversation. The voice can respond naturally while the coordinator still knows which ground the plan intended to cover.
The model sometimes drops the ID. When it goes missing, the coordinator removes punctuation and common words from the proposed question, compares what remains with the unasked plan and accepts the closest match above a similarity threshold. I chose a forgiving fallback because losing an identifier should not force the interviewer back into scripted language.
An ID keeps a paraphrase attached to the plan. It does not prove that the person heard the question, answered it or would regard a later formulation as repetition. To give the Orchestrator some conversational memory, I pass it the ten most recent proposed questions and a broader set of themes accumulated across its proposals. Those are records of what the system considered. They are not a transcript of what took place between two people.
Asked was doing too much work
My first question ledger treated selection as delivery. A planned question entered askedQuestionIds as soon as the Orchestrator chose it, before the session manager tried to update the speaking model’s instructions. That was a reasonable first defence against repetition, but the name promised more certainty than the code could support.
When that update threw an error, the call continued and the local state did not roll back. The question disappeared from the pool even though the application could not say whether its new instructions had reached the Realtime service, reached it partially or failed before leaving the machine.
The later transcript could not settle the matter. It recorded what the speaking model generated, but the text was not linked to the plan-question ID chosen earlier. The design had compressed an intention, an attempted update and a generated utterance into one inaccurate word: asked.
A question now accumulates evidence through five stages:
- selected, when the Orchestrator chooses a plan item;
- instructions accepted, when the Realtime service acknowledges the updated instructions;
- response started, when those instructions are associated with a particular response;
- generated, when the final assistant transcript contains the chosen question;
- delivered, when that response completes successfully.
Failed and superseded attempts remain in the ledger too, but they do not consume the question. The Orchestrator sees delivered, pending and available plan items separately, so it can avoid duplicating a question already in flight while still retrying one whose instruction update or response failed. A completed response which wandered away from the chosen question is retryable for the same reason: fluent speech is not evidence that the intended question reached the conversation.
Retiring a question too early may skip the thought the interviewee was ready to explore; remembering too little makes the interviewer repeat itself. From the other side of the call, both failures feel like the same social mistake. The voice which seemed attentive a moment ago is no longer listening.
Returning to an earlier conversation
I wanted a follow-up to feel like returning to the same conversation, not meeting an interviewer who had just been handed a folder of notes. A new Realtime connection does not inherit the coordinator’s live state, so the application has to reconstruct enough memory to continue.
The transcript offers partial evidence. Assistant entries containing question marks supply a possible question history. The latest ten provide immediate context, accumulated themes preserve wider patterns, and word overlap estimates which planned questions the earlier session probably covered. This is intentionally approximate; it gives the next interview clues without claiming to recreate the first one.
The same set was also carrying several different kinds of history. An ID might have come directly from a live Orchestrator decision, from wording matched during that decision, or from similarity against an old transcript when a follow-up began. The ledger now keeps reconstructed transcript coverage distinct from live delivery evidence, preserving its origin instead of hiding both inside askedQuestionIds.
There was a similar gap in the notes. A follow-up’s speaking model can receive its new direction and a bounded summary of ideas, stories, claims and covered sections from the first interview. The live currentNotes used by the Note-Taker and Orchestrator still begins empty. The voice can therefore sound familiar with the earlier conversation while the agents choosing its next question are rebuilding why the material mattered.
The distinction matters because evidence changes meaning as it travels through the system. A live selection begins as an attempt and earns delivery only when the Realtime service accepts its instructions, starts a response, generates matching words and completes. A follow-up’s similarity match is recorded as reconstruction from an earlier transcript. Both can guide the next question without pretending to be the same kind of fact.
The session manager now waits for Realtime to acknowledge each instruction update and ties the resulting response ID to its final transcript. Timeouts, disconnects, playback failures and failed or incomplete responses release the question for another attempt. When a completed response wanders elsewhere, the planned question returns to the available pool as well. From the interviewee’s side there is no ledger to see, only an interviewer which remembers what it actually said.