Xhadow

A pronunciation coach that clones the learner’s own voice, has it speak the target language, and compares that reference against the learner’s recording acoustically to correct articulation. FFT, formants, pitch and voicing are all computed in the browser; the two recordings are aligned on the time axis by a word-anchored DTW; the aligned time series is then handed to an LLM to be translated into instructions for the tongue, jaw and lips. I did the product, the engineering and the design myself.

The problem

Most pronunciation apps return a score. You read a sentence, you get 73, and that is the end of it.

That score has no reference point. The comparison is a native speaker with a different vocal tract, a different register and a different speaking rate, and much of the acoustic difference between that recording and the learner’s is not a pronunciation error but a difference in anatomy. If the cause of a low score cannot be separated into articulation error versus speaker characteristics, there is no way to answer what to change or how.

Xhadow removes that confound. It clones the user’s own voice, has it speak the target language with a native accent, and compares the user against themselves. Timbre, register and vocal tract characteristics are fixed on both sides, so what remains in the difference signal is articulation and timing.

Making that comparison hold up is a signal processing problem, and most of this codebase is that problem.

The design principle: measurement is deterministic, generation only translates

The LLM never sees audio. Per-phoneme scores, the substitution that was actually heard, duration deviations and the aligned F1/F2/F3 trajectories are all computed deterministically before the prompt is assembled. What the model receives is a table of those computed values.

The boundary exists to bound the failure mode. Feed raw audio to a multimodal model and accuracy collapses at the phoneme level, while the output still arrives as confident prose. Coaching generated without acoustic evidence cannot even be checked for being wrong.

One job is left to the generative step: translating an already-measured difference into an articulatory instruction in the learner’s language. The model can phrase advice badly, but it cannot invent a difference that was never measured. So when the coaching looks wrong, it is always determined whether to look in the prompt or in the measurement.

The pipeline

1. The reference: a target utterance in my own voice

Voice cloning runs in two Cartesia calls, because cloning and accent transfer are separate operations.

Step	Call	What it does
Clone	POST /voices/clone	Instant cloning from a single 5–10 s clip recorded in a language you speak fluently
Localise	POST /voices/localize	Transfers that timbre to the target language, dialect and speaker gender (42 languages)

Only the localised voice_id is stored; the intermediate base voice is deleted immediately. Synthesis streams raw PCM from sonic-3 with add_timestamps: true, which returns word-level timestamps. Those timestamps are not decoration for karaoke highlighting; they are the anchor set that makes the alignment in step 4 computable.

2. Phoneme-level assessment

Recording captures 16 kHz, 16-bit, mono PCM directly in the browser. MediaRecorder returns WebM/Opus at the device’s native sample rate and Azure’s pronunciation assessment requires exactly the format above, so the capture path uses AudioContext({ sampleRate: 16000 }) to delegate resampling to the browser and writes the 44-byte RIFF header by hand.

Assessment is configured at maximum resolution: a hundred-mark scale, phoneme granularity, enableMiscue for omission and insertion detection, prosody assessment, and nbestPhonemeCount: 5.

That last one is the important one. Beyond scoring the expected phoneme, Azure returns the top five phonemes it actually heard. If a Korean speaker says rice and /l/ was heard where /ɹ/ belongs, the app knows not that the score is low but which substitution occurred. The diagnosis and the correction come from that substitution, not from the score.

3. The DSP front end, written without dependencies

There is no fft.js, no meyda, no dsp.js. Every transform below is hand-written and runs in the browser.

• Framing — fftSize 2048, hopSize 512 (a 46.4 ms window and 11.6 ms hop at 44.1 kHz)
• Window — symmetric Hann
• FFT — in-place iterative radix-2 Cooley–Tukey, bit-reversal permutation followed by butterfly stages, about 60 lines
• F0 — time-domain autocorrelation over lags sr/500 to sr/50
• Formants — spectral peak picking, then band constraints to select F1/F2/F3
• Timbre — spectral centroid, flatness, 85% rolloff

Formant selection takes the top ten local maxima between 200 and 4000 Hz and applies ordering constraints: F1 is the largest magnitude in 200–1000 Hz, F2 is above F1 + 200 Hz within 800–2500 Hz, F3 is above F2 + 200 Hz within 2000–4000 Hz.

Peak picking returns plausible values in unvoiced and silent regions too. So a separate gate decides whether a formant value can be trusted, as a weighted combination of three cheap features.

confidence = pitchConf × 0.5 + max(0, 1 − zcr/0.15) × 0.3 + min(1, rms/0.02) × 0.2

Each term encodes one fact: fricatives have a high zero-crossing rate, and silence has no energy. The value is not for display. Phoneme boundary refinement uses the 0.35 crossing as the vowel-to-consonant transition point: boundaries start from an even split, then each interior boundary moves within ±40% of the mean phoneme duration to sit on the frame whose voicing is closest to 0.35.

4. Hybrid segmented DTW

The learner’s utterance and the reference differ in length, and the mismatch is not uniform. The first clause was rushed and the last word was drawn out. A single global rate ratio cannot express that; what is needed is a monotonic non-linear warping function between the two timelines, which is what dynamic time warping computes.

Running DTW over the two full spectrograms was rejected for two reasons.

1. Cost. A ten-second utterance is about 1000 frames and the DP table is O(N·M). Each of the million cells is a 1024-dimensional cosine distance, and in the bundled stress harness it blocks the main thread for 422 ms. It grows quadratically with length.
2. Drift. DTW optimises a global path. One cough, one room reflection, one fricative burst that skews the alignment, and every frame after it inherits the error. Nothing pins the path to a linguistic reality.

The fix uses a fact already available on both sides: where each word starts and ends. The reference has Cartesia word timestamps and the utterance has Azure word offsets, and those anchors are far more reliable than spectral distance. So one large alignment problem is decomposed into a chain of small, independently bounded ones.

For each target word:

1. Find the matching spoken word by text with case and punctuation stripped.
2. Convert the Azure centisecond spans to seconds and map both time ranges to frame indices.
3. If either side exceeds 100 frames, decimate both by the same factor so the local time axis stays consistent.
4. Run standard DP over a flat row-major Float32Array: cosine distance, three-neighbour steps, unit transition weights.
5. Backtrack the accumulated cost with a diagonal-first tie-break.
6. Sample about ten points from the path, reproject them linearly into the word’s real time span, and append the word’s start and end anchors.

The result is a sorted, piecewise-linear, bidirectional time map. On the same harness it finishes in 23 ms against 422 ms for the global path, and alignment error is structurally confined to the word that caused it. Lookups use binary search and linear interpolation in both directions, reference to attempt and attempt to reference, with a round-trip inversion error of 0.000000 s.

This is the most important decision in the project. Because alignment is pinned to linguistic units, every comparison downstream happens between the same sounds.

5. What the alignment produces

One time map yields four results.

WSOLA time-warped playback. The learner’s recording is resynthesised onto the reference timeline so the two can be compared frame-for-frame. It places 40 ms Hann grains at 50% overlap, looks up the ideal input position in the time map for each output grain, then cross-correlates over ±15 ms around it to pick the point that continues the previous grain best. That similarity search prevents the pitch-period discontinuities plain resampling produces, so pitch is preserved and only time is warped.

A predictive playback-rate servo. During synchronised playback the audio is not resampled; the browser’s own pitch-preserving playbackRate is driven by a numerical derivative of the alignment function. It looks 0.1 s ahead to get the required rate and clamps to 0.5–2.0. Drift beyond 50 ms triggers a hard resync, and hysteresis keeps the rate from audibly hunting.

Honest per-phoneme timing. Azure gives the duration of the spoken phoneme; the time map gives how much of the reference timeline that span occupies. The difference is the real deviation, flagged past 80 ms or 50%.

Trajectory comparison. For any phoneme, the F1/F2/F3 and voicing tracks from the attempt and the reference are aligned through the time map and resampled onto a fixed 30-point grid, which makes two vowels of different lengths directly comparable point by point. A single mean value erases the movement in a diphthong such as /aɪ/ or /eɪ/, and without that movement there is no way to say where the tongue should travel from and to.

6. The acoustic evidence pack and the output contract

What the LLM receives is the structured table assembled by everything above. Per phoneme: the score, the substitution actually heard, the duration deviation against the reference, the attempt and target formant trajectories sampled every third point of the 30, and the voicing delta.

Alongside it goes contrastive-phonology prior knowledge for the learner’s first language. Six languages — Korean, Japanese, Chinese, Spanish, German and French — carry an explicit list of the interference patterns speakers of that language are known to produce. A Korean speaker’s /r/–/l/ confusion comes from Korean having a single liquid that varies by position; a Japanese speaker’s identical error comes from a flap that sits between the two. Same error, different cause, and therefore a different correction.

The prompt carries two prohibitions: never write F1, F2, F3 or “formant” in the output, and convert every observed difference into something the user can actually do. The translation rules fix the correspondences from articulatory phonetics: F1 maps to jaw opening, F2 to tongue frontness, F3 to lip rounding. “F2 is 190 Hz below target” stays internal, and the output becomes “your tongue is too far back, so /æ/ sounds closer to /e/”.

The token budget is controlled before the prompt is assembled. A phoneme that scores 100 and sits within 30% of the reference duration is dropped, and the remaining trajectories are sent at one point in three. Input size then scales with the number of real problems rather than with sentence length.

The model is pinned to temperature: 0.2 and response_format: { type: "json_object" }, and must return a fixed contract: an array of per-word issues, each with a diagnosis, a correction and an importance.

After the diagnosis

A diagnosis on its own is not a product.

Articulatory visualisation. F1/F2/F3 are mapped to tongue height, tongue frontness, lip rounding and jaw opening, driving a sagittal mouth animation and a set of 22 viseme images. Learners who cannot act on written corrections get the same information through a second channel.

Adaptive practice. Per-phoneme occurrence counts and mean scores are aggregated over the last 100 attempts; the five lowest-scoring phonemes with at least three occurrences are selected, and a practice script dense in those sounds is generated. This works without a separate analytics pipeline because every attempt stores its full normalised Azure result in a Json column.

A context-aware tutor. A streaming chat that already knows the phoneme scores from the last attempt, so “why do I keep getting /r/ wrong?” is answered from that user’s measurements rather than in general terms.

Course Studio. Courses are generated from a URL, YouTube captions, live news search, uploaded PDF and Word files, or free text, and can be published to a public catalogue, forked and rated.

Engineering decisions

Noise suppression is left off. echoCancellation, noiseSuppression and autoGainControl are not forced on in getUserMedia; they stay at the browser default. Those DSP chains change formant structure and dynamic range, which is exactly the signal being measured.

Four rendering technologies are used deliberately. The spectrogram writes about 90,000 pixels per redraw, so it uses pixel-level putImageData. The vowel space chart has roughly 30 elements needing CSS transitions and hit testing, so it is inline SVG. The voice orb is a fragment shader, so it is WebGL, seeded from a hash of the voice ID to give each clone a deterministic visual identity. A chart library is used in exactly one place, the score trend chart.

Long work goes through a job table. Generating a course from a URL means an LLM call, a TTS call and several DB writes, well past 30 seconds and outside the request budget. Work is recorded as a row carrying progress and a human-readable status line, and the client polls every two seconds. Job IDs live in localStorage, so a refresh mid-generation reconnects to the running job.

Server Actions are a public API surface. Every export of a "use server" module can be called from any browser with a crafted request. Functions that must run server-to-server without a session, particularly the ones background jobs call, live in lib/ and leave authorisation to the caller. The gate for route handlers returns the authenticated user and a rate-limit scope in the same call, because authentication answers who and a metered upstream also needs how often.

Recordings are private, course images are public. A voice recording must not be readable from a leaked URL, so the bucket is private and every read goes through an authorising proxy that redirects to a five-minute signed URL rather than streaming bytes through a serverless function. Object keys are namespaced as <userId>/…, so one RLS policy on the first path segment covers the whole tree.

The rate limiter lives in Postgres. There is no Redis and serverless instances cannot share an in-memory counter, so the database is the only shared state. One row per (key, window) estimates a rolling count by weighting the previous window by its remaining overlap — the same cost as a fixed window, without letting a caller spend two quotas back to back across the boundary.

Validation

A bundled synthetic harness checks that each layer behaves as intended. The figures below are from a re-run while writing this page.

Target	Method	Result
Formant extraction	Synthetic spectrum with peaks at 720 / 1250 / 2550 Hz	Errors of 9.4 Hz (F1), 1.1 Hz (F2), 9.1 Hz (F3)
Voicing gate	Pitch, ZCR and RMS matching /s/, /z/ and /a/	0.150 / 0.510 / 0.910 — separated at the 0.3 threshold
Hybrid DTW	1000×1100 frames (about 10 s), three word anchors	23.4 ms against 422.5 ms for global DTW, 40 anchors
Time map inversion	Forward lookup, then reverse lookup on the returned value	Round-trip error of 0.000000 s
Boundary refinement	Synthetic track with an /a/→/s/ transition at 0.60 s	Even split at 0.50 s moved to 0.60 s

Limits

A document that lists only strengths is no use to anyone reading the code, so these are on the record.

Signal processing

• The Sakoe–Chiba band is inert in production. The DTW function accepts a windowSize but no application code passes one, so the real path is full O(N·M) DP inside each word segment. The band is used only in the test harness. What actually bounds the cost is the 100-frame decimation cap.
• The DTW feature vector is weak. Frames are per-frame peak-normalised dB magnitude spectra, so every vector is dominated by large negative numbers and cosine distance compresses toward zero. It works because the word anchors do most of the alignment; MFCCs with delta features would be a strictly better local cost.
• Formants come from peak picking, not LPC. At a 46 ms window and 21.5 Hz resolution, the peaks found for high-F0 speakers are individual harmonics rather than peaks of the formant envelope. LPC with Levinson–Durbin and root finding is the textbook solution.
• Pitch confidence is not normalised. The autocorrelation confidence should be divided by autocorr[0], but that value is never recorded, so it acts as an energy gate rather than a normalised correlation coefficient.
• All DSP runs synchronously on the main thread. A Web Worker or AudioWorklet is the next step, and the ScriptProcessorNode used for capture is already deprecated.

Engineering

• There are no automated tests with assertions. The harness scripts only print to the console, and the DTW harness injects random noise, so timings differ between runs. The figures in the table above are reference values rather than benchmarks, and what carries meaning is the ratio, not the absolute number.
• The job queue is in-process fire-and-forget. It writes the row and then calls the processing function without awaiting it, inside the same serverless invocation. It survives a client refresh but not an instance shutdown. A real queue is the correct shape.
• The rate limiter is keyed on user ID rather than IP. It caps what one account can spend but not how many accounts someone creates.
• There is no runtime schema validation on LLM output. Responses are parsed and cast. That boundary is where schema validation belongs.
• Voice cloning is biometric-adjacent and the app currently has no consent step. Opening a path to cloning anyone other than yourself requires adding one first.