Xhadow: acoustically grounded pronunciation coaching against your own voice

Pronunciation products commonly score a learner against a native-speaker recording. But the learner and the reference differ not only in articulation; they also differ in timbre, register, vocal-tract geometry and speaking rate. If a low score cannot be separated into speaker difference versus pronunciation error, it can report “how much” without explaining what the tongue or jaw should do differently.

Xhadow changes the reference. It clones the learner’s own voice, localises that voice into the target language, and compares the learner’s recording with that self-reference. The design aims to hold some speaker-specific characteristics in common and turn the remaining phoneme, timing and acoustic differences into articulatory instructions. It has not experimentally established that every speaker confound is removed; that remains a product hypothesis.

The contribution is not a new speech model or a new DTW algorithm. It is a systems composition with an explicit evidence boundary across voice cloning, phoneme assessment, browser signal processing, word-anchored alignment and generation.

1. Design the reference before generating advice

Reference creation has two stages. A 5–10-second sample clones the speaker’s timbre, then localisation transfers that voice to a target language and dialect. Word timestamps returned with the synthesised reference are not karaoke decoration; they become anchors for the later alignment.

The learner’s recording goes through Azure Speech at phoneme granularity. Alongside an expected-phoneme score, it returns likely substitutions, omissions and insertions, prosody and word offsets. If /l/ is heard where /ɹ/ was expected, that direction of substitution is more actionable than the low score by itself.

Stage	Input	Deterministic evidence retained
Self-voice reference	Cloned voice localised into the target language	PCM and per-word start/end times
Phoneme assessment	Learner recording and target text	Scores, substitution candidates, miscues and word offsets
Browser DSP	PCM from both utterances	Spectrum, F0, F1/F2/F3 candidates, voicing and timbre features
Time alignment	Word spans and spectra from both sides	Bidirectional piecewise-linear time map
Coaching translation	Aligned acoustic evidence	Actions for the tongue, jaw and lips

2. A handwritten DSP front end in the browser

FFT, spectral formant-peak extraction, autocorrelation pitch, zero-crossing rate, RMS and voicing are implemented in TypeScript without an external DSP library. Frames use a 2,048-point Hann window and a 512-sample hop; an iterative radix-2 Cooley–Tukey FFT runs in the browser.

The formant method is band-constrained spectral peak picking, not LPC. F1, F2 and F3 candidates are selected under ordering constraints from 200–1,000 Hz, 800–2,500 Hz and 2,000–4,000 Hz. Because silence and fricatives can still yield plausible peaks, a separate voicing gate combines pitch confidence, zero-crossing rate and RMS.

These transforms structure the acoustic difference without sending the source recording into a generative model. The LLM receives a computed table, not audio.

3. Decompose global alignment into word-bounded problems

A ten-second utterance represented by roughly 1,000 frames makes global DTW evaluate around a million cells of high-dimensional distance. Cost is only one problem: a cough, reflection or fricative burst can bend the global path and propagate that error through every later segment.

Xhadow uses word boundaries already available from recognition and synthesis:

1. Match words after normalising punctuation and case.
2. Map reference and attempt word times into frame ranges.
3. If either side exceeds 100 frames, decimate both by the same factor.
4. Run standard DTW inside each word pair.
5. Sample about ten path points, project them back into real time, and add word-edge anchors.
6. Join all anchors into a bidirectional reference↔attempt time map.

This is not a novel DTW algorithm. It is an application composition that uses linguistic anchors to bound the search and the propagation of failure. An alignment error inside one word does not have to pull every later word off course, and each local problem has a fixed size cap.

The same map is reused three ways. WSOLA resynthesises the learner onto the reference timeline while preserving pitch; a synchronised-playback servo corrects drift; and per-phoneme F1/F2/F3 and voicing trajectories are compared on a fixed 30-point grid.

4. Limit the LLM to translation, not measurement

No raw audio is passed to the generative model. For each phoneme it receives only:

• the expected phoneme and likely heard substitution;
• duration deviation against the self-reference;
• aligned F1/F2/F3 trajectories and voicing difference; and
• known L1 interference patterns for the learner’s first language.

The output contract prohibits reporting formant notation directly and asks for an action a person can perform. An internal F2 difference becomes an instruction to move the tongue forward or back; an F1 difference becomes a change in jaw opening. The model may still phrase advice poorly, but the input boundary prevents it from inventing an acoustic difference that was never measured upstream.

5. What was validated—and what was not

Bundled scripts run each layer on synthetic inputs and print console results. All three were rerun during an August 2026 code audit.

Target	Synthetic condition	Audit-run result
Formant peak recovery	Injected peaks at 720 / 1,250 / 2,550 Hz	Errors of 9.4 / 1.1 / 9.1 Hz
Voicing gate	Features corresponding to /s/, /z/ and /a/	0.150 / 0.510 / 0.910
Phoneme boundary refinement	Synthetic transition placed at 0.60 s	Even 0.50 s boundary moved to 0.60 s
Word-segmented DTW	Random 1,000×1,100 frames, three word anchors	Global 481.44 ms → segmented 25.58 ms, 40 anchors
Time-map inversion	Forward lookup followed by reverse lookup	0.000000 s round-trip error

The DTW harness injects random noise and the scripts contain no assertions. The timings are therefore one local audit run, not a fixed benchmark. This is also why the preserved 422.5→23.4 ms reference run has different absolute values. The repeated direction is a substantial reduction after decomposition into bounded word problems; hardware-level performance and a statistical timing distribution were not measured.

More importantly, these harnesses are not evidence that learners improved their pronunciation. There is no real-speech formant benchmark, pre/post learning study, retention result or teacher-rated outcome.

6. Current limits

• Peak-picked formants can select individual harmonics rather than the spectral envelope for high-F0 speakers; LPC is the stronger baseline.
• Pitch confidence is not a normalised autocorrelation coefficient and behaves more like an energy gate in the current implementation.
• DTW uses cosine distance over peak-normalised dB spectra. MFCCs with delta features are a better candidate local cost.
• The Sakoe–Chiba window accepted by the function is not passed on the product path; the effective bound is the 100-frame per-word cap.
• DSP and DTW run synchronously on the main thread and should move to an AudioWorklet or Web Worker.
• Voice cloning is biometric-adjacent. Any path beyond cloning one’s own voice needs explicit consent and misuse controls first.

7. Contribution boundary and next evaluation

Xhadow does not yet establish that “AI teaches pronunciation well.” It demonstrates an engineering method that designs the comparison and evidence boundary before generation, then composes disparate speech technologies into a traceable coaching pipeline.

The next evaluation should compare peak picking with LPC on real speakers, measure alignment error with and without word anchors, and run a blind teacher-rated study comparing score-only feedback with articulatory instructions. Until then, performance claims remain scoped to layer-level synthetic operation.

Measurement record

Area	Primary record
Full design and limits	Xhadow repository README.md
Browser DSP	lib/audio-analyzer.ts, scripts/test-dsp.ts
Word-anchored DTW	lib/dtw.ts, scripts/test-hybrid-dtw.ts
WSOLA	lib/granular.ts
Phoneme boundaries	scripts/test-phoneme-boundary.ts
Development history	19 Nov 2025 initial app → 6 Dec 2025 phoneme-timing integration

Xhadow source repository