Real-Time Speech Translation ML Engineer

LILT (Production); Washington (District of Columbia); On-site; USD 140,000 - 200,000; Full time

Job summary

LILT is building a new live translation product. We are seeking an ML Engineer to build the real-time speech translation backend that powers it.

You will own the backend end-to-end, from live audio input to translated output, on our production model serving platform with Ray Serve on GPU Kubernetes clusters and our in-house adaptive MT models. This hands-on backend role requires deep Python, ML stack knowledge, and collaboration with senior architects and language researchers to deliver a

Qualifications

• BS or MS in Computer Science or related field, or equivalent practical experience.
• 3+ years building production backend or ML serving systems in Python, including strong async programming (asyncio) skills.
• Hands-on experience with real-time streaming transport: WebSocket or gRPC bidirectional streaming, session state, backpressure, and connection lifecycle handling.
• Experience serving ML models in production on GPUs (Ray Serve, Triton, vLLM, or similar), with Docker and Kubernetes.
• Experience integrating speech or NLP models into production systems, ideally streaming ASR (partial hypotheses, endpointing, VAD).
• A latency-engineering mindset: you have profiled, instrumented, and optimized a real-time or low-latency system and can reason in per-stage budgets.
• Effective use of AI coding agents (Claude Code, Codex, or similar) on top of fundamentals learned the hard way: you let agents do the typing, but you can debug, review, and reason about every line without them, and you know when not to trust them.
• US citizenship and residence in the United States (contract requirement).

Responsibilities

• Real-time pipeline architecture: Build and manage services for high-throughput, real-time audio and text streaming.
• ML model integration: Integrate and serve streaming speech recognition and machine translation models, collaborating with research teams to ensure models operate within required latency budgets.
• Quality and confidence workflows: Develop logic for model-based confidence scoring, routing segments for human intervention as needed, and broadcasting real-time updates and corrections to end-users.
• Infrastructure and scale: Architect and scale production ML infrastructure on GPU-accelerated Kubernetes clusters. Implement batching, load balancing, and autoscaling strategies to maintain performance and cost-efficiency.
• Latency engineering: Establish comprehensive instrumentation for real-time performance. Identify bottlenecks, optimize system throughput, and drive down end-to-end latency metrics to meet production standards.
• Interface and API definition: Define technical contracts and interfaces for audio ingestion and downstream service integrations. Partner with frontend and platform engineering teams to maintain clean, robust integration points.
• Collaboration and technical leadership: Drive cross-team alignment by defining clear API interfaces and technical contracts, facilitating effective communication between engineering and product teams to ensure seamless system integration.

Skills

Python

Asyncio

Real-time streaming

Latency optimization

AI tooling

Education

BS or MS in Computer Science or related field

Tools

Ray Serve

Triton

Docker

Kubernetes

vLLM

WebSocket

gRPC

Job description

LILT is building a new live translation product. We are seeking an ML Engineer to build the real-time speech translation backend that powers it.

You will own the backend end-to-end, from live audio input to translated output, on our production model serving platform with Ray Serve on GPU Kubernetes clusters and our in-house adaptive MT models. This hands-on backend role requires deep Python, ML stack knowledge, and collaboration with senior architects and language researchers to deliver a