November 13, 2026
Comments Off on How to Build a Consent-Aware Healthcare Voice AI Agent with LiveKit and vCon

A healthcare voice AI agent’s intake call usually leaves its transcript, recording, consent, and clinical data in separate services, each with its own format and IDs. We built a telehealth intake agent on LiveKit that writes all of it into one signed, portable vCon, a conversation record format now moving through IETF standardization. 

This post walks through the architecture and the code for this voice AI audit trail: capturing consent before the first dialog turn, linking each AI analysis back to the turn it came from, and producing a redacted copy that stays tied to the full record.

Why a healthcare voice AI agent needs one conversation record

A telehealth intake call lasts a couple of minutes. The record of it has to last for years.

When a healthcare voice AI agent runs the intake, that record ends up spread across a pipeline of artifacts, each in a different service’s format:

  • a live transcript from the real-time speech-to-text engine,
  • an audio recording sitting in object storage,
  • a second, clinical-grade transcript from a medical transcription service,
  • a list of extracted medications and conditions from a medical NLP service,
  • a physician summary from a large language model,
  • and a consent flag, usually a boolean in some table, that is supposed to tie all of this together.

When an auditor asks what the patient consented to and whether the recording matches the transcript you acted on, when the clinic wants to hand the visit to a partner, or when you swap one vendor for another, you are reconciling six formats connected only by IDs your own code keeps in sync.

A vCon keeps all six in one record.

The vCon format and our AI patient intake demo

A vCon (virtualized conversation) is a JSON container for a whole conversation: its participants, its content, and everything derived from it. It can be compared to a vCard, with a vCard being a standard way to exchange contact details and a vCon a standard way to exchange a conversation. A vCon can be signed and encrypted, and it has built-in places for consent and redaction. Jeff Pulver of the vCon Foundation made the broader case for vCon as the data layer for voice AI on WebRTC Live Episode 118: Building the Data Layer for Voice AI with vCon.

Our proof of concept, vcon-livekit-demo, tests how much of the integration work vCon removes. It demonstrates the mechanisms vCon provides for consent, integrity, and redaction. You can clone it and browse a committed sample record with no AWS account and no live calls, or run a call end to end against your own LiveKit and AWS setup.

It is not a HIPAA-certified deployment, so use it with test data only and never with real protected health information (PHI). 

The Architecture: a LiveKit voice agent plus an AWS enrichment pipeline

The build splits into two domains with opposite priorities. The vCon is the link between them.

The real-time leg is the live call. It optimizes for conversational feel: Deepgram for speech-to-text, ElevenLabs for text-to-speech, and a large language model (LLM) in between, all orchestrated by a LiveKit voice agent. It produces a provisional transcript as the patient speaks.

The enrichment leg is everything after the words are spoken. It optimizes for clinical accuracy and compliance: Amazon Transcribe Medical for an authoritative transcript, Amazon Comprehend Medical for clinical entities, and Amazon Bedrock for the physician summary. It produces the record that lives forever.

vCon enrichment pipeline in AWS Step Functions: Transcribe Medical runs first, then Comprehend Medical and Bedrock run in parallel before the results are merged and the vCon is sealed.
The live call builds the unsigned vCon in the LiveKit agent. After the call, the Step Functions pipeline adds the analysis and signs the record, and the doctor and analyst views both read from it.

At the center of both legs is one small vCon document that both legs read and write. It has a fixed shape: version and metadata on top, then four arrays. Parties index 0 is always the patient and index 1 is always the agent; everything else references parties by that index rather than embedding an identity.

// The unsigned vCon document that VConBuilder (src/vcon_agent/vcon.py) assembles,
// turn by turn, during a call. parties[0] = patient, parties[1] = agent.
// dialog / analysis / attachments reference parties by index so provenance
// can hang off the index later.
{
  "vcon": "0.0.1",
  "uuid": "<uuid4>",
  "subject": "Patient intake call",
  "created_at": "<ISO 8601 UTC>",
  "parties": [ {patient}, {agent} ],
  "dialog": [ ... ],      // transcript turns + a reference to the recording
  "analysis": [],         // filled in later: transcript, entities, summary
  "attachments": [ {consent}, ... ]
}

That analysis array starting empty is the whole idea in miniature. The live agent produces the skeleton; the enrichment leg writes back into the same object. Next, how the skeleton gets built while the call is still happening.

The vCon is not assembled after the call from a session report. It grows turn by turn, in its unsigned, in-progress form, while the patient is still talking. This matters for consent: consent is captured as structure before any clinical content exists, not patched in afterward.

The builder that assembles this document is framework-independent: it has no LiveKit, Deepgram, or AWS dependency, so the assembly logic is fully testable offline. It also enforces that consent ordering. Consent is recorded as an attachments entry that references the patient party, and the builder refuses to append a dialog turn until consent is on the record.

# src/vcon_agent/vcon.py
def add_consent(self, granted: bool, script: str = DEFAULT_CONSENT_SCRIPT):
    # Consent is an attachment tied to the patient (party index 0), captured
    # BEFORE any dialog turn. add_dialog_turn() enforces this ordering.
    entry = {
        "type": "consent",
        "party": PATIENT_INDEX,          # 0 = patient
        "granted": bool(granted),
        "script": script,               # the agent's exact wording
        "timestamp": _utc_now_iso(),
    }
    self._doc["attachments"].append(entry)
    self._consent_recorded = True
    return entry

def add_dialog_turn(self, party_index: int, text: str, start_offset=None, ...):
    # Guard: no clinical content may be recorded before consent exists.
    if not self._consent_recorded:
        raise RuntimeError(
            "consent must be recorded via add_consent() before appending dialog"
        )
    # ... append a {type: text, party, body, start, ...} turn to dialog[]

This one-way guard is why consent lives inside the conversation record instead of in a separate table that can drift out of sync with the audio. Consent that holds up later needs clear notice before capture and a durable record of what the caller agreed to, and here that record is a timestamped attachment in the same signed object as the words that followed it.

Wiring this to LiveKit is a thin layer. In the agent entrypoint, the agent subscribes to LiveKit’s conversation_item_added event, which fires once per committed turn. The handler pulls the role and text off the event and hands them to a translation function:

# src/vcon_agent/agent.py
@session.on("conversation_item_added")
def _on_conversation_item_added(event: ConversationItemAddedEvent) -> None:
    item = event.item
    role = getattr(item, "role", None)             # 'user' or 'assistant'
    text = getattr(item, "text_content", None)
    # Seconds from call start, so the clinical transcript can align later.
    start_offset = _offset_seconds_from(session_start_iso, _utc_now_iso())
    handle_item(builder, role, text, start_offset=start_offset)

The handler keeps the LiveKit-specific parts (the event, the clock) and delegates the mapping to handle_item:

# src/vcon_agent/agent.py
def handle_item(builder, role, text, start_offset=None):
    # Map a LiveKit ChatMessage role to a fixed party index and append a turn.
    # 'user' -> patient (0), 'assistant' -> agent (1). Prompt-scaffolding roles
    # (system/developer) and empty text are skipped.
    if role is None:
        return False
    normalized = role.strip().lower()
    if normalized not in ("user", "assistant"):
        return False
    if not text or not text.strip():
        return False
    builder.add_dialog_turn(role_to_index(normalized), text, start_offset=start_offset)
    return True

On session end the finished document is flushed once to S3 under a unique per-run key, and the agent triggers enrichment by calling Step Functions directly. The call recording is captured separately through LiveKit Egress and referenced in the vCon by its S3 URL. By the time the pipeline starts, the vCon already holds the parties, the consent, the live transcript, and a pointer to the audio.

Next, the enrichment leg fills in that empty analysis array, and links each result back to the turn it came from.

Linking each AI analysis to the transcript turn it came from

After the call, an AWS Step Functions state machine enriches the skeleton. It polls S3 for the recording, runs Transcribe Medical as an async batch job (which is the reason for Step Functions: a medical transcription job on real audio can blow past a single function’s timeout, so the state machine owns the waiting), then runs Comprehend Medical and Bedrock in parallel before merging and sealing.

vCon enrichment pipeline in AWS Step Functions: Transcribe Medical runs first, then Comprehend Medical and Bedrock run in parallel before the results are merged and the vCon is sealed.
Step Functions handles the wait for the Transcribe Medical batch job. Comprehend Medical and Bedrock then run in parallel, and each writes an analysis entry that points back to the dialog turns it came from.

Each derived fact is written as an analysis entry that references the dialog turn indices it came from. That reference is the provenance link. An entities entry points at the single turn the medication was spoken in; a summary points at every turn it covers. In a custom format you would invent and maintain this linkage by hand. Here it is just a dialog field of indices on every analysis object:

# src/vcon_agent/vcon.py
def add_entities_analysis(self, entities, dialog_index, vendor="aws-comprehend-medical"):
    # Clinical entities (meds, conditions) scoped to ONE dialog turn.
    # The dialog field is the provenance: it names the turn these came from.
    entry = {
        "type": "entities",
        "vendor": vendor,
        "dialog": [int(dialog_index)],   # turn-index provenance
        "body": [dict(e) for e in entities],   # [{category, text, score}, ...]
    }
    self._doc["analysis"].append(entry)
    return entry

This is also where the dual-transcript design pays off. The live Deepgram transcript stays in dialog; the clinical Transcribe Medical transcript goes into analysis as a separate, authoritative entry that references the same turns. The two serve different needs: a latency-tuned transcript for the live conversation and a clinically accurate one for the record. The clinical copy is marked authoritative so downstream consumers know which to trust. The summary is built from the authoritative transcript: the live transcript can mishear a medication name, and summarizing from the clinically accurate source keeps that error out of the physician’s note.

# src/vcon_agent/vcon.py
def add_transcript_analysis(self, transcript_text, dialog_indices,
                            vendor="aws-transcribe-medical", authoritative=True,
                            turns=None):
    # The clinical transcript sits ALONGSIDE the live transcript, not replacing
    # it. It references the dialog turns it covers and is marked authoritative.
    entry = {
        "type": "transcript",
        "vendor": vendor,
        "authoritative": bool(authoritative),
        "dialog": [int(i) for i in dialog_indices],   # provenance
        "body": transcript_text,
    }
    if turns:                       # speaker-attributed, timestamped turns
        entry["turns"] = [dict(t) for t in turns]
    self._doc["analysis"].append(entry)
    return entry

Holding two transcripts over the same dialog, each attributed to a different producer with its own provenance, is something vCon models natively. A custom record would have to reconcile a Deepgram result and an AWS result by hand. Here is a trimmed enriched document showing the entities and summary referencing the turns they derive from:

{
  "dialog": [
    { "type": "text", "party": 1, "start": 0.0,
      "body": "What brings you in today?" },
    { "type": "text", "party": 0, "start": 4.0,
      "body": "I've had a bad headache since yesterday." },
    { "type": "text", "party": 1, "start": 7.5,
      "body": "Have you taken anything for it?" },
    { "type": "text", "party": 0, "start": 11.5,
      "body": "I took some ibuprofen this morning but it didn't help." },
    { "type": "recording", "start": 0.0, "mimetype": "audio/ogg",
      "url": "s3://.../recording.ogg" }
  ],
  "analysis": [
    { "type": "entities", "vendor": "aws-comprehend-medical",
      "dialog": [3],
      "body": [
        { "category": "MEDICATION", "text": "ibuprofen", "score": 0.99 }
      ] },
    { "type": "summary", "vendor": "aws-bedrock", "dialog": [0, 1, 2, 3],
      "body": "Headache since yesterday; ibuprofen gave little relief." }
  ]
}

Because the entities entry carries "dialog": [3], the viewer can take the medication “ibuprofen,” find turn 3, read its numeric start, and seek the audio element to the exact moment it was said.

Before it reaches a viewer, though, the record gets signed and sealed.

Signing, encrypting, and redacting the vCon

The merge step hands off to a seal step that turns the enriched document into the record you actually store and share. It does three things, all with the standard’s own constructs:

  • Sign (JSON Web Signature, JWS): the canonical JSON of the vCon is signed so any later tampering is detectable.
  • Encrypt (JSON Web Encryption, JWE): the full record, with its PHI, is stored encrypted.
  • Redact: a de-identified sibling is produced and linked back to the full version through the vCon redacted pointer.

The signing and encryption are standard JOSE over the document:

# infrastructure/functions/seal/sealing.py
def sign_vcon(vcon_doc, private_key):
    # Sign the canonical JSON so the record is tamper-evident. The protected
    # header carries the alg and a vCon content type so a verifier knows the
    # payload is a signed vCon.
    token = jws.JWS(canonical_json(vcon_doc).encode("utf-8"))
    token.add_signature(private_key, alg="PS256", protected=...)
    return token.serialize()

Redaction is where the record shows its structure. Instead of masking a known allowlist of fields and hoping no new PHI-bearing field slips through, the redactor is deny-by-default: it walks every string in the document and runs it through the masking pass unless the key is a known structural field (the spec version, a party role, a timestamp, an entity category). A name is hard-replaced outright because a name identifies a person on its own, and the recording URL is cleared so the audio location does not ride along. Modeling “full record” and “redacted sibling” as one linked concept is what keeps the two from diverging into separate, unreconciled exports.

The backend does all of this crypto server-side and hands readable JSON to the browser, so the viewer never touches a signing key. The demo video below walks through the flow end to end.

Watch for two moments. First the call itself: the patient talks to the agent and gives consent. Then the doctor view of the finished record, assembled from the single vCon: the call recording, the physician summary, the extracted medications and conditions, and both transcripts, the live one and the authoritative clinical one, side by side. Clicking a medication seeks the audio to the moment it was spoken, which is the provenance link from the enrichment section made visible.

The video stops at the doctor view, but the same record drives a second one: an analyst view that loads the redacted sibling. It is recognizably the same vCon, linked through the redacted pointer, with PHI masked. One record, two role-scoped views, no divergent copies to keep in sync.

Hardening the demo for production

A few pieces would harden the demo for production:

  • Signing key: the demo uses a self-signed keypair; production moves it to a KMS-managed, HSM-backed key with rotation.
  • Access control: the role toggle between doctor and analyst is a URL parameter; production gates each view behind real authentication and authorization with audit logging.
  • Redaction: the offline pass catches regex-matchable personally identifiable information (PII), and removing clinical-term PHI relies on a live Comprehend Medical detection pass; production replaces both with vetted de-identification reviewed against the applicable standard.

vCon vs. a custom record format

The context from a voice AI call usually stays inside the vendor that handled it: the transcript in the speech-to-text service, the recording in object storage, the analysis in whatever service produced it. A signed vCon moves as one file. You can load it into any vCon-aware tool or hand it to a partner clinic, and it works without a custom adapter. Every benefit in this build comes from that choice: one conversation container instead of a separate format per service.

DimensionCustom format per servicevCon
Record shapeTranscript blob + recording + metrics + consent row, linked by conventionOne object: parties / dialog / analysis / attachments
Consent and redactionHand-rolled columns and flagsFirst-class redacted chain plus consent attachment
IntegrityBuild and prove your own schemeStandard JWS signing, tamper-evident
ProvenanceImplicit via your own IDsanalysis entries reference exact dialog turns
Adding a serviceWrite a new adapterRead and write the same vCon
Handoff or migrationBespoke export each timeAlready portable

A custom format is built around the services you use today. A vCon is built for any service that reads the conversation later, including an auditor three years from now.

Adding vCon to your own voice AI agent

The pattern we used works for any voice agent: listen for each committed conversation turn, record the call audio, and map both into the vCon schema in your own code. In this build, those come from LiveKit’s conversation_item_added event and LiveKit Egress. Other voice agent frameworks expose equivalent hooks. 

If you are building voice AI and need the call record to hold up after the call ends, look at vCon before you design your own format.

WebRTC.ventures builds voice AI products on LiveKit, including vCon pipelines like the one in this post. Contact us if you want help designing yours.

WebRTC.ventures is a LiveKit development partner.
WebRTC.ventures is a LiveKit development partner.

Further Reading:

Recent Blog Posts