You know a sound is off. What should your mouth do differently on the next attempt? I wanted to build feedback that answered that question: advice on how a learner might move their tongue, jaw or lips to change the sound. Xhadow grew from that goal into a practice application connecting voice cloning, acoustic analysis, comparison playback and coaching. I handled the product, engineering and design myself.
Building a reference in the learner’s voice
Scores and transcripts are useful for reviewing an attempt, but deciding what to do with the mouth requires more information. I wanted to inspect which sound was recognised in place of the expected one, which part lasted too long, and how a vowel changed during its pronunciation. Comparing recordings from different speakers also introduces differences in timbre, pitch range and speaking speed. I began by giving learners a reference they could hear in their own voice.
The application clones a user’s recording through Cartesia, localises the voice for the target language, and synthesises the practice sentence. That reference can then be played and analysed alongside the learner’s recording. The product hypothesis is that a familiar voice could reduce some speaker differences and make articulation and timing easier to focus on. It does not assume that a cloned voice preserves the learner’s vocal tract or removes every difference between speakers.
Connecting acoustic evidence to a physical suggestion
I combined several kinds of analysis to gather useful coaching inputs. Azure pronunciation assessment provides word and phoneme positions, scores, and candidate sounds recognised in place of the expected phoneme. In the browser, I wrote the FFT processing and pitch and formant estimation code to analyse short segments of audio. Formants describe acoustic resonances; I used their trajectories as clues to how a vowel develops over time.
A computed value is not automatically a reliable observation. A spectrum can contain peaks during silence or a fricative as well as during a vowel. I therefore combined signal energy, waveform sign changes and pitch estimates into a voicing-confidence measure, then used that information in phoneme boundary refinement and the analysis sent to the coach. The input includes recognised substitutions, duration differences and corresponding formant trajectories from the reference and the attempt.
The LLM’s job is to explain those inputs in the learner’s language. I designed its prompt to compare the acoustic data internally and suggest changes to the tongue, jaw and lips in ordinary language. Separating measurement and estimation from explanation makes it possible to inspect what a suggestion was based on. A microphone does not directly observe movement inside the mouth, however. The visual articulation guide and written advice are inferences from acoustic clues, not a precise reconstruction of physical positions.
Comparing the same moment in the same word
Once the acoustic features were available, timing became the next problem. A learner may rush one word and sustain a vowel in another. Matching the overall playback speed can therefore place different sounds side by side. An early implementation used dynamic time warping, or DTW, across the complete recordings. It finds corresponding points by acoustic similarity, but its computation grows with the recordings and its matches are not inherently tied to word boundaries.
I then changed the alignment to use word positions that were already available. Timestamps from the synthesised reference and word offsets from the pronunciation assessment became anchors, with detailed alignment performed inside each word. Long segments are reduced at the same rate on both sides before the path is mapped back to the original timeline. Dividing the comparison into word-sized problems also gave word selection and phoneme-duration analysis a shared map between the recordings.
Playing that alignment required another step. The first implementation cut and overlapped audio fragments at the mapped positions; the way those fragments joined also had to be considered. I rewrote the playback code using WSOLA, which searches for a similar waveform continuation before overlapping the next fragment. The implementation aims to stretch or compress the learner’s recording to the reference timing while preserving pitch. It made the alignment something the learner could listen to as well as inspect on a graph.
Turning analysis into another attempt
The interface lets learners select a word or phoneme, listen to that segment, and compare their recording with the reference. Scores, likely substitutions, timing differences and coaching appear in the same flow. I also revised the interpolation of formant trajectories so the coach receives values at corresponding positions. The design focus was connecting the sound being played to the part the advice refers to.
To carry the feedback into further practice, I added a history of phoneme assessments. The application aggregates recurring sounds, identifies weaker phonemes, and generates a personalised course containing them. Longer generation and analysis jobs store their status so a learner can leave the page and return to the result. The implemented flow runs from listening and recording through comparison and coaching to another practice session.
What the implementation establishes
The result is a research prototype in which acoustic analysis and generated coaching can be inspected together. The repository includes experimental scripts using synthetic signals to examine formant estimation, voicing, time alignment and phoneme boundary refinement. These scripts explore computational behaviour; they are not a user study demonstrating pronunciation gains or the accuracy of physical coaching.
The current formant estimator approximates resonances by picking spectral peaks, and word matching relies on text equality. Higher-pitched voices, repeated words and omitted speech therefore need further evaluation. The next work is to compare the acoustic estimates with expert assessments of articulation and test whether the suggested movements help learners. Xhadow provides an implementation in which the path from an acoustic difference to a coaching suggestion can be traced and improved; the learning benefit remains to be established.
Source code · Word-level alignment · Playback implementation change