Real-Time Speech Translation Backend Engineer

LILT; Boston (MA); On-site; USD 120,000 - 161,000; Full time

Job summary

LILT in Boston, MA is seeking an experienced backend/ML infra engineer to own the end-to-end live translation pipeline, fusing streaming speech recognition with adaptive machine translation in a GPU-accelerated Kubernetes environment.

You will optimize latency budgets, implement batching and autoscaling, monitor real-time performance, and collaborate with research, frontend, and platform teams to deliver robust translations with low latency.

Qualifications

• 3+ years building production backend or ML serving systems in Python with async skills.
• Experience with real-time streaming transport (WebSocket/gRPC bidirectional).
• Experience serving ML models on GPUs in production (Ray Serve/Triton/vLLM).
• Familiarity with Docker and Kubernetes in production environments.
• Experience integrating speech/NLP models into streaming systems.

Responsibilities

• Build high-throughput, real-time audio and text streaming services.
• Own session lifecycle management, concurrency, and backpressure handling.
• Integrate streaming ASR and MT models with latency budgets.
• Develop confidence scoring and routing for human intervention.
• Broadcast real-time updates and corrections to end users.
• Scale production ML infra on GPU-accelerated Kubernetes with Ray Serve.
• Implement batching, load balancing, and autoscaling for performance and cost.
• Set up instrumentation for real-time performance and observability.
• Identify bottlenecks and reduce end-to-end latency to production standards.
• Define API contracts for audio ingestion and downstream services.
• Collaborate with frontend and platform teams on robust integrations.

Skills

Python

Asyncio

WebSocket

gRPC

Distributed systems

GPU ML serving

Education

BS or MS in Computer Science

Tools

Ray Serve

Triton

vLLM

Docker

Kubernetes

RabbitMQ

Datadog

Prometheus

WebRTC

LiveKit

Job description

LILT in Boston, MA is seeking an experienced backend/ML infra engineer to own the end-to-end live translation pipeline, fusing streaming speech recognition with adaptive machine translation in a GPU-accelerated Kubernetes environment.

You will optimize latency budgets, implement batching and autoscaling, monitor real-time performance, and collaborate with research, frontend, and platform teams to deliver robust translations with low latency.