The problem I wanted to address with Rxly.ai was the work of turning a consultation into a usable record. While listening to a patient, a physician also needs to identify key information, remember questions to follow up, and prepare documentation for use after the visit. I designed and built an assistant that prepares draft records and items to review as the conversation unfolds. Working on clinical requirements with physicians practising in the US, I focused on a workspace they could use during the consultation.
Turning a transcript into clinical context
A transcript still leaves substantial work to do. Patient-reported symptoms need to be distinguished from the clinician’s observations, and information scattered through a conversation needs to be organised into history, assessment and plan. I connected transcription, live insights, differential-diagnosis candidates and draft records within a consultation session. Generated content remains working material for the physician to review and edit.
In the audio workflow, I separated identifying different speakers from determining their roles. A speaker number supplied by speech recognition does not tell the application which person is the clinician. The implementation groups speech by speaker, uses conversational content to assign roles, and retains an unknown state before identification. It also handles interim transcription separately from finalised utterances. A medical speech model is selected for supported languages; the implementation does not assume equivalent medical-vocabulary performance across every language.
Keeping the physician’s work as the conversation grows
Live analysis presents a different problem from summarising a completed recording. Sending the whole conversation on every update grows the request, while sending only the new speech can lose earlier context. I built a flow that normally sends added utterances with the previous summary, then periodically returns to the full transcript. An inline comment from the physician also triggers a full-context pass. Inputs that arrive during an analysis can queue a subsequent run.
Updates also need to preserve the physician’s work. If a new AI response resets a completed checklist, the assistant creates more work. I send existing item identifiers into the next request and reconcile the response with the current items, retaining checked states and physician notes. Manually added items are preserved separately. The implementation therefore has to handle both the quality of new suggestions and their interaction with changes a person has already made.
Preventing a late response from entering another consultation
Switching consultations became an important boundary for asynchronous work. A physician can open another session while an AI response is still arriving, and a scheduled autosave may remain pending. I revised these paths to check that the session receiving a result is the one that initiated the request. Autosaves remember the session at scheduling time, check it again before writing, and clear pending timers when the session changes. What looks like a simple navigation action must also preserve which consultation a record belongs to.
Giving the physician a path back to the sources
I connected differential-diagnosis candidates and research answers to material a physician could inspect. PubMed and Europe PMC provide literature searches; OpenFDA and DailyMed provide drug-related material; ClinicalTrials.gov provides clinical-trial information. ICD-11 classification lookup is handled alongside those five medical knowledge sources. The structure keeps the type and origin of retrieved material visible to the model and the interface.
The connectors run in parallel with individual failure handling, allowing successful searches to remain useful when another service is slow or unavailable. The prompt receives the sources that actually returned results, together with titles and links for citation. The model can still generate candidates when retrieval returns no material, so it would be inaccurate to describe every answer as verified by a search. The implementation provides a route for reviewing sources; connecting them does not itself establish diagnostic accuracy.
Connecting draft documents to review before export
Record generation uses the transcript together with physician notes and the existing draft. This keeps what the patient said, what the clinician added, and what has already been written available to the next generation step. I expanded the structured SOAP record into editable document templates with their own fields. Templates containing diagnosis fields can require a confirmed-diagnosis selection, distinguishing a differential candidate from the diagnosis to be entered in a document.
I also separated preparing external data from sending it. The application prepares a FHIR R4 bundle, presents it for review, and then provides a path to send it to Medplum. This creates an inspection point before model-generated structured data becomes an external record. It establishes an export format and connector implementation; it does not mean interoperability with a working hospital EMR has been validated.
What the prototype demonstrates
I added a simulation that replays sample consultations so the transcription and analysis flow can be examined repeatedly. It can be used to inspect application behaviour such as pausing and resuming, updating analysis and switching sessions. For clinical-data handling, I implemented HIPAA-informed technical controls including AES-256-GCM field encryption, authentication, audit logging and request limits. Neither simulation nor these controls substitutes for clinical validation or a compliance audit.
I developed the project alone over one month from concept to hackathon submission, and was the only Korean finalist in Anthropic’s Built with Opus 4.6 Claude Code hackathon — 230 finalists from roughly 13,000 applicants. The submitted prototype connects live transcription, analysis during the visit, draft records, source retrieval and export. I have not separately measured reductions in documentation time or improvements in diagnostic outcomes. Further evaluation needs to examine transcription, speaker attribution, record accuracy and the physician’s review workload together in a clinical setting.
Submission demo · Source code · Live-analysis flow · Review before export
