Senior ML Infra Engineer for AI Assistants (Hybrid)

Roku; Cambridge (MA); Hybrid; USD 180,000 - 260,000; Full time

Benefits offered by this job

Hybrid work model

401(k)

Healthcare benefits

Job summary

Roku is seeking a Senior Software Engineer to own production ML infrastructure for the Entertainment Assistant team. You’ll drive training, evaluation, deployment, and monitoring of models, with emphasis on latency, cost, and reliability across distributed services.

You’ll collaborate with ML, product, data and platform partners to turn ambiguous problems into measurable outcomes, expanding agent-based tooling and scalable evaluation pipelines across our streaming platform.

Qualifications

• Strong production software-engineering experience, including designing, testing, operating and debugging distributed services.
• Meaningful ownership of ML infrastructure or production model delivery, with clear evidence of your contribution and its operational or adoption impact.
• A practical understanding of the ML lifecycle: training, evaluation, versioning, deployment, online validation, monitoring and safe rollback.
• Experience designing for latency, resilience, observability and cost in production systems.
• Experience with LLMs, embeddings, semantic search, retrieval or agent systems, including how you evaluate quality and control failure modes.
• Experience with caching systems and the trade-offs involved in using caching for low-latency services.
• A commitment to automation, CI/CD, code quality and evidence-led engineering decisions.
• A collaborative approach and the ability to explain technical trade-offs to partners across disciplines.
• A degree in Computer Science, Electrical Engineering or a related field, or equivalent practical experience.

Responsibilities

• Own fulfilment-ranking pipelines from training orchestration and quality gates through deployment and online validation.
• Build and operate an offline-evaluation platform, combining LLM-as-judge harnesses with deterministic checks for answer quality.
• Develop the Roku Entertainment Assistant's LLM agent, including tool routing, retrieval, guardrails and answer caching.
• Design caching as an intentional latency and cost lever for high-volume services.
• Build observability for ML and LLM systems, including latency attribution, quality metrics, tracing and per-request cost.
• Improve the reliability and operability of distributed systems, and lead the diagnosis of difficult cross-service failures.
• Drive accountable AI-assisted engineering practices across the team, building on the agents already used in release workflows.
• Work across machine-learning, product, data and platform boundaries to turn ambiguous problems into measurable delivery outcomes.

Skills

Distributed services

ML infrastructure

LLM systems

Caching systems

CI/CD

Observability

Latency optimization

Go

Python

Education

BS in CS or related field

Tools

AWS

GCP

Kubernetes

SQL

Trino/Presto

Spark

Kafka

Job description

Roku is seeking a Senior Software Engineer to own production ML infrastructure for the Entertainment Assistant team. You’ll drive training, evaluation, deployment, and monitoring of models, with emphasis on latency, cost, and reliability across distributed services.

You’ll collaborate with ML, product, data and platform partners to turn ambiguous problems into measurable outcomes, expanding agent-based tooling and scalable evaluation pipelines across our streaming platform.