What a Transcript Throws Away

by Jerry

There is a version of the words "I'm fine" that closes a conversation, and there is a version that opens one, and anyone who has been on the receiving end of that phrase from someone they love already knows the difference without being able to say exactly where it lives. It is not in the words, since the words are identical, and it is not quite in the volume or the speed either, though those are part of it. It is in the small catch before the second word, or the laugh that arrives a fraction too quickly afterwards, or the breath that someone takes when they have decided not to say the longer thing. Written down, all four versions collapse into the same two words, and the person reading them is left to guess.

I have come to think that this is where most of self-expression actually happens. We tend to talk about expressing ourselves as though it were a matter of choosing the right words, and a great deal of effort goes into that, but the words are only the part of the message that survives being written down. The rest of what a person is telling you arrives in how they said it, in the hesitation and the emphasis and the sound of someone trying not to cry while insisting that they are not, and this is the part that makes them recognisably themselves rather than a string of text that anyone could have produced. When we say that we know someone, a surprising amount of what we mean is that we can hear the difference between their fine and their fine.

What worries me is how much of our speech now passes through systems that keep only the words. Voice notes are transcribed, meetings are summarised, calls are turned into tickets, and assistants listen to us and reply to a flattened version of what we said. Each of these is useful, and none of them is malicious, but the effect of all of them together is that the version of a person which reaches the other side of a conversation is thinner than the one who spoke. In an age when a machine can produce fluent, well-formed sentences at no cost at all, the words themselves are becoming the cheap part of communication, and the audible evidence that a particular person meant a particular thing in a particular way is becoming the rare part. It seems to me that we should be building systems that protect that evidence rather than systems that throw it away.

This is what Attune is trying to do. It is a small English speech-perception model, about three hundred million active parameters, that takes a recording of between half a second and thirty seconds and returns, in a single time-aligned result, the transcript with the timing of each word, any localised vocal events it can support such as a laugh, a sob, a cough or a breath, a phrase-level estimate of how the delivery would be perceived by a listener, and, crucially, a measure of how sure it is along with a way of saying that it is not. The readable view of that result looks like a transcript with the delivery written back in, so that the same two words can come out as a plain "I'm fine", or as "I'm fine" followed by a laugh with joy perceived at some confidence, or as "I'm fine" with a sob and distress perceived instead. The words never change; what changes is that the reader is no longer guessing alone.

I want to be careful about what that does and does not mean, because the distinction is the whole point of the project. Attune reports observable vocal evidence and an uncertain listener's perception of it. It does not, and does not claim to, determine what anyone is actually feeling. When no category clears its calibrated threshold, it abstains and says so rather than forcing a label, and its output keeps the model's opinion in a separate field from the person's words, so that a perception can never be mistaken for something the speaker said. This is the same restraint that a good listener shows. You can notice the tremor in someone's voice without presuming to name what caused it, and you can hold that noticing lightly, ready to be corrected, which is a very different thing from announcing that you know how they feel.

The same care extends to what the model is for. It is released for consented research and evaluation, and its licence and model card are explicit that it must not be used for covert monitoring, diagnosis, deception detection, speaker identification, the inference of protected traits, or any automated decision about hiring, credit, insurance, policing, medicine or law. A tool that listens closely to how people sound is exactly the kind of tool that could be turned against them, and I would rather set those boundaries now, while the model is small and imperfect, than discover later that we had built something powerful without ever deciding what it was not allowed to do.

At HXI Labs we have said that the purpose of intelligence is to give people more of each other, and I think Attune is the clearest test of that sentence we have built so far. A system that preserves how someone sounded and hands that back to the people they are speaking with, so that a message carries the person and not only their words, is the kind of thing this lab exists to make. A system that guesses at feelings on someone's behalf and reports them as fact is the kind of thing it exists to refuse. The difference between those two is not a matter of accuracy but of design, and it has to be decided before the first line of code.

I also want to be honest about where it stands, because it would be easy to describe this project in a way that makes it sound finished, and it is very far from that. The current model is version 0.13, which is an evaluation candidate rather than a release, and although the weights are public on Hugging Face they are published for research and evaluation rather than for use. Its sense of affect was learned mostly from acted, categorical datasets, which means it has learned what a performed emotion sounds like more thoroughly than what a lived one does, and on an internal test of a voice moving from joy to distress it classified both halves as fear. Only a handful of vocal events carry strong enough timing labels to be trusted. The delivery styles and the dimensional sense of valence, arousal and dominance exist in the research architecture but are switched off until they pass the evaluation gates, and on longer recordings the affect windows are fixed slices rather than learned boundaries. It works in English only, on a single speaker, for no more than thirty seconds at a time.

The largest gap is that no study has yet asked real people whether hearing these annotations changed how well they understood each other, which is the question the whole project exists to answer, and until that study exists everything above is a technical prototype and not a claim about human communication. Between this checkpoint and something I would call complete lie natural rather than acted recordings, speakers and accents the model has never heard, Ghanaian English among them, languages beyond English, a fresh consented recording of hostile speech that is currently blocking release, and a long period of adversarial testing before anyone should let it into a conversation that matters. I would guess that the honest answer to how much work remains is most of it.

And yet I think it is the right beginning, because the direction of a project like this is set by what it refuses as much as by what it does. Attune abstains rather than guesses, it keeps its perceptions apart from the speaker's words, and it treats affect as something a listener perceives rather than something a machine has established. Those are decisions rather than scores, and they will survive every future version of the model in a way that no accuracy number will. If AI is going to have any role in how people express themselves to one another, it has to start by being honest about the limits of listening, and this one does. The repository, the weights, the results files and the model card are public, and if you work on speech, on affect, or on the strange problem of helping people understand each other, we would like to hear from you at hello@hxilabs.com.

The point was never to have a machine understand us. It was to make sure that when we do speak, less of us is lost on the way.