Self-Hosted Voice AI Pipeline on EKS: GPU Scheduling with Karpenter.

Self-hosting a voice AI pipeline gives you control that managed APIs can’t offer: you choose the models, you decide where they run relative to each other, and you keep audio data within your own infrastructure. Getting there means managing GPU workloads, model placement, and service networking yourself. Kubernetes provides the underlying mechanisms for GPU scheduling, autoscaling, and service discovery. Amazon EKS packages them into a managed platform, using Karpenter for GPU provisioning, so you get this behavior without configuring it from scratch.

Here’s how to deploy a full self-hosted voice AI pipeline (VAD → STT → LLM → TTS) on EKS using open-weight models, LiveKit for real-time media transport, Karpenter for GPU provisioning, and pod affinity to keep services close.

The full implementation is on the WebRTC.ventures GitHub: colocating-voiceai-models-eks

Prerequisites

• AWS account with permissions for EKS, EC2 (GPU instances), IAM, and VPC
• Service quota: 8 vCPUs for G-instance On-Demand. The default is often 0. Two g5.xlarge nodes require 8 vCPUs. Request an increase via the Service Quotas console under Amazon EC2 → “Running On-Demand G and VT instances.” This can take 1-3 days to approve.
• Hugging Face token with access to meta-llama/Llama-3.1-8B
• CLI tools: AWS CLI v2, Terraform ≥ 1.5, kubectl, Helm 3, Node.js 20+ (for the browser client)

The self-hosted voice pipeline and its infrastructure requirements

A voice agent pipeline is sequential. Each stage must complete before the next begins:

Each component has different resource needs: the LLM benefits from GPU acceleration, STT and TTS can run on either CPU or GPU depending on model choice, and VAD is lightweight enough to run in-process. On Kubernetes, this means deciding how to distribute these workloads across nodes — which pods need GPUs, which can share, and how close they should be to each other.

The practical challenge is ensuring your GPU workloads land in the same place, and that non-GPU workloads (like the orchestrator) can reach them efficiently. That’s what this architecture is designed around.

Voice AI pipeline architecture on EKS

We deploy all pipeline components in the same availability zone. Two GPU nodes for the AI models and one CPU node for the orchestrator and WebRTC gateway (for this post, we keep these in the same node for simplicity, but in production you’d likely want to scale these separately). Scheduling is handled by Karpenter NodePool constraints rather than complex affinity rules.

Amazon EKS gives us the building blocks:

• Karpenter NodePools: provision GPU nodes in a specific AZ with specific instance types
• Pod affinity: schedule related pods on the same node using topology keys
• EKS Auto Mode: removes node group management overhead while supporting GPU workloads

The architecture looks like this:

Three nodes in total:

• CPU Node: LiveKit (WebRTC media server) and the orchestrator share a node via hostname-level pod affinity. Both use hostNetwork, so they communicate over the node’s local interface.
• GPU Node 1: vLLM serving Llama 3.1 8B AWQ on a dedicated A10G.
• GPU Node 2: Speaches serving STT (Faster-Whisper) and TTS (Kokoro) on a dedicated A10G.

The orchestrator coordinates the pipeline flow: it receives audio from LiveKit, sends it to Speaches for transcription, passes the text to the LLM, then sends the LLM’s response back to Speaches for speech synthesis. All service-to-service calls travel over cluster networking (~1ms per hop).

Let’s look at how this is implemented in Kubernetes.

Implementation on EKS

The implementation has three layers: infrastructure (Terraform), scheduling (Karpenter + Helm), and the voice pipeline itself (LiveKit Agents orchestrator + open-weight models).

Infrastructure: EKS Auto Mode with Karpenter

Terraform provisions an EKS Auto Mode cluster with all resources in a single availability zone.

# EKS Auto Mode cluster — Karpenter is built-in
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "voiceai-eks"
  cluster_version = "1.36"

  # This enables EKS Auto Mode — Karpenter runs as a managed component
  cluster_compute_config = {
    enabled    = true
    node_pools = ["general-purpose", "system"]
  }

  cluster_endpoint_public_access  = true
  cluster_endpoint_private_access = true
}

EKS Auto Mode includes Karpenter out of the box. We define two NodePools: one for GPU workloads (LLM and Speaches) and one for CPU workloads (LiveKit and the orchestrator).

They’re managed separately: the CPU NodePool is a standalone manifest applied before deploying LiveKit, while the GPU NodePool is bundled inside the voice-pipeline Helm chart.

# k8s/nodepools.yaml — applied with kubectl before the Helm charts
# CPU NodePool for LiveKit + Orchestrator
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: voice-agents
spec:
  template:
    metadata:
      labels:
        workload-type: voice-agents
    spec:
      # EKS Auto Mode requires nodeClassRef pointing to the default NodeClass
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
        # Same AZ as GPU nodes — keeps inter-service latency low
        - key: "topology.kubernetes.io/zone"
          operator: In
          values: ["us-east-1a"]
      # Taint prevents GPU or system pods from landing here
      taints:
        - key: workload-type
          value: voice-agents
          effect: NoSchedule
  # Enough CPU for LiveKit + Orchestrator on one node
  limits:
    cpu: "8"
  disruption:
    consolidateAfter: Never
    consolidationPolicy: WhenEmpty
    # Never evict — voice calls are stateful
    budgets:
      - nodes: "0"

# helm/voice-pipeline/templates/karpenter-nodepool.yaml — managed by Helm
# GPU NodePool for LLM and Speaches
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-voiceai
spec:
  template:
    metadata:
      labels:
        workload-type: gpu-voiceai
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      requirements:
        # Each g5.xlarge has 1 A10G GPU — one model per node
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values: ["g5.xlarge"]
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
        # Same AZ as CPU node
        - key: "topology.kubernetes.io/zone"
          operator: In
          values: ["us-east-1a"]
      # Keeps non-GPU workloads off these nodes
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
  # 2 GPUs = 2 nodes (one for LLM, one for Speaches)
  limits:
    nvidia.com/gpu: "2"
  disruption:
    consolidateAfter: Never
    consolidationPolicy: WhenEmpty
    budgets:
      - nodes: "0"

Key choices here:

• Single AZ for both pools. Forces all nodes (CPU and GPU) into us-east-1a. The zone constraint on the NodePool handles colocation between GPU pods without requiring pod affinity rules.
• Disruption budget of 0 nodes. Karpenter will never voluntarily evict a pod from either pool.
• Separate taints. GPU nodes get nvidia.com/gpu (only GPU workloads schedule there), CPU nodes get workload-type: voice-agents (only LiveKit and the orchestrator schedule there).

When a pod with nvidia.com/gpu resource requests shows up and a suitable node doesn’t yet exist, Karpenter provisions a g5.xlarge within 2-3 minutes. Since each g5.xlarge has one A10G GPU and each AI pod requests one GPU, Karpenter automatically provisions separate nodes for the LLM and Speaches.

GPU Scheduling: how pods land on the right nodes

The scheduling strategy is straightforward:

• LLM and Speaches have GPU tolerations and request nvidia.com/gpu: 1. They land on the GPU NodePool. Since each g5.xlarge only has one GPU, Karpenter provisions a separate node for each. The NodePool’s zone constraint keeps them colocated.
• Orchestrator has a nodeSelector for workload-type: voice-agents and a hostname-level pod affinity to LiveKit. Both land on the same CPU node.

The orchestrator’s affinity rule is the only explicit pod affinity in the system:

# Orchestrator deployment — colocate with LiveKit on the same CPU node
affinity:
  podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            # Target the LiveKit server pod
            - key: app.kubernetes.io/name
              operator: In
              values:
                - livekit-server
        # Same node, not just same zone
        topologyKey: kubernetes.io/hostname

This ensures the orchestrator shares a node with LiveKit. Both use hostNetwork: true, so they communicate over the node’s local network interface without crossing the cluster network.

The LLM and Speaches pods have no pod affinity. The zone constraint on the GPU NodePool keeps them colocated.

The voice pipeline: open-weight models across three nodes

The pipeline runs across three nodes with carefully chosen models:

Component	Model	Runs on	Why
LLM	Llama 3.1 8B Instruct AWQ (4-bit)	GPU Node 1 (g5.xlarge, A10G)	~200ms TTFT, fits in 24GB with KV cache
STT + TTS	Faster-Whisper large-v3-turbo (float16) + Kokoro 82M	GPU Node 2 (g5.xlarge, A10G)	GPU-accelerated for fast STT/TTS
VAD	Silero VAD	CPU Node (in-process)	~30ms, runs inside the orchestrator
Orchestrator	LiveKit Agents	CPU Node (shared with LiveKit)	Coordinates pipeline, manages streaming

Each GPU node gets a dedicated model workload. Speaches (STT+TTS) gets the full GPU on its own node for inference acceleration.

Speaches bundles both STT and TTS into a single OpenAI API-compatible container, exposing /v1/audio/transcriptions and /v1/audio/speech. This reduces pod count and simplifies service discovery.

The orchestrator: LiveKit Agents wiring it together

The orchestrator is a Python agent built on LiveKit Agents that connects the pipeline stages. It handles VAD detection (Silero, in-process), routes audio to STT, passes transcripts to the LLM, streams LLM output to TTS, and sends synthesized audio back to the user. All with interruption (barge-in) support.

Since every service in the pipeline exposes an OpenAI-compatible API, the orchestrator configuration is straightforward. Each service is just a ClusterIP address:

# Service endpoints — injected via ConfigMap environment variables
STT_BASE_URL = os.environ.get("STT_BASE_URL")
LLM_BASE_URL = os.environ.get("LLM_BASE_URL")
TTS_BASE_URL = os.environ.get("TTS_BASE_URL")

session = AgentSession(
    # Speaches STT — OpenAI-compatible /v1/audio/transcriptions
    stt=openai.STT(
        model=STT_MODEL,
        base_url=STT_BASE_URL,
        api_key="sk-placeholder",
    ),
    # vLLM — OpenAI-compatible /v1/chat/completions
    llm=openai.LLM(
        model=LLM_MODEL,
        base_url=LLM_BASE_URL,
        api_key="sk-placeholder",
    ),
    # Speaches TTS — OpenAI-compatible /v1/audio/speech
    tts=openai.TTS(
        model=TTS_MODEL,
        base_url=TTS_BASE_URL,
        api_key="sk-placeholder",
    ),
    # Silero VAD — runs in-process, preloaded at worker startup
    vad=silero.VAD.load(),
)

The orchestrator also publishes total pipeline latency to the browser client via LiveKit’s data channel after each conversational turn. For production, you’d want a proper observability stack — Amazon Managed Prometheus for metrics collection, Grafana for dashboards, and NVIDIA DCGM Exporter for GPU utilization. Per-stage latency histograms (STT, LLM TTFT, TTS TTFB) and tail latency (P95/P99) are the metrics that matter most for voice AI. Averages hide the worst user experiences.

LiveKit: WebRTC media transport

LiveKit is the WebRTC media server that relays audio between the browser and the orchestrator. It runs with hostNetwork: true to expose WebRTC UDP ports (50000–60000) directly on the node’s public IP.

The browser connects to LiveKit over WebSocket (TCP 7880) for signaling and UDP for media. The orchestrator runs on the same node (also with hostNetwork), so traffic between them stays on the node’s local interface. LiveKit routes audio to the orchestrator, which joins the room as a participant.

The critical path, orchestrator ↔ STT ↔ LLM ↔ TTS, goes over same-AZ ClusterIP networking (~1ms per hop). The LiveKit ↔ orchestrator hop is free (localhost).

Demo

Here’s the pipeline handling a restaurant reservation call end-to-end. The voice agent plays the role of a reservation assistant at “The Golden Fork,” an upscale Italian restaurant.

In this interaction:

1. The caller requests a table for two on Saturday around 7:30 PM.
2. The agent confirms availability and collects the name — Martinez.
3. The caller mentions they’re celebrating an anniversary; the agent offers to add a special touch to the table.
4. The booking is confirmed: two people, Saturday at 7:30 PM, under Martinez.
5. The caller declines additional offers (drinks, dessert pre-orders, confirmation call), and the conversation wraps up naturally.

Turn-to-turn latency throughout the call sits between 500–800ms — the time from when the caller finishes speaking to when the agent’s voice begins. That’s the full pipeline round-trip: VAD detects silence, audio goes to STT, the transcript hits the LLM, the response streams to TTS, and audio starts playing back.

Operational considerations

Running this pipeline reliably for real callers surfaces a different set of constraints around networking, cold starts, and model tradeoffs. They are worth knowing before you scale past a single test node.

• LiveKit with hostNetwork. This is LiveKit’s standard Kubernetes deployment model. It requires direct network access for WebRTC media ports. The tradeoff is one LiveKit pod per node. For multi-replica deployments, STUNner is a Kubernetes-native WebRTC gateway that removes this constraint.
• Model loading time dominates cold start. The LLM takes 3–5 minutes to load weights on a fresh node. For always-on production, keep at least one warm node rather than relying on scale-from-zero.
• AWQ quantization. 4-bit quantization cuts weight size by 4x with negligible quality loss for short, conversational responses. Smaller weights also leave more GPU memory for KV cache, supporting more concurrent conversations per GPU.

Next steps

This post covered the patterns for running a self-hosted voice AI pipeline on EKS: Karpenter for GPU scheduling and provisioning, pod affinity for orchestrator placement, and open-weight models stitched together with LiveKit Agents. The full implementation is on GitHub: colocating-voiceai-models-eks.

At WebRTC.ventures, we build and deploy real-time voice and video applications, from prototype to production. If you’re working on a voice AI pipeline and need help with infrastructure, model integration, or scaling on Kubernetes, reach out to our team today. Let’s make it live!