Staff Software Engineer — Model Infrastructure & Reliability

Speedrun Talent Network; San Francisco, Northern (CA, KY); Hybrid; USD 193,000 - 290,000; Full time

Job summary

Harvey is seeking a Staff Software Engineer to lead the design and development of the Model Infrastructure platform powering AI requests. You will collaborate with AI Research, Product Engineering, and external providers to build a reliable, scalable, and observable foundation for model inference.

You will own high-availability systems, optimize latency, and drive capacity planning while mentoring engineers and advancing the state of the art in frontier models and provider integrations.

Qualifications

• 4+ years of software engineering experience building large-scale distributed systems.
• Experience designing and operating highly available production services.
• Strong programming skills in Go, Java, Python, Rust, or C++.
• Deep understanding of distributed systems, cloud infrastructure, networking, and observability.
• Experience leading technical projects across multiple engineering teams.
• Ability to balance long-term architecture with pragmatic execution.
• Strong communication and collaboration skills.
• Passion for building foundational platforms that enable other engineering teams.

Responsibilities

• Lead the design and implementation of Harvey's Model Infrastructure platform.
• Build systems to ensure high availability, low latency, and operational excellence for AI inference.
• Design and improve Harvey's Unified Model Controller (UMC) and Model Selector platform to automatically detect model degradations and intelligently route traffic based on reliability, latency, quality, compliance, and cost.
• Develop systems for model provisioning, capacity management, failover, and traffic engineering across multiple AI providers.
• Integrate new model providers and maintain provider APIs and SDKs, enabling Harvey to rapidly adopt emerging frontier models.
• Improve observability through health dashboards, alerting, token usage analytics, cost reporting, and end-to-end telemetry.
• Partner with Product Engineering to support model launches, experimentation, and proactive monitoring of production AI workloads.
• Drive infrastructure efficiency through capacity planning, utilization optimization, and cost visibility.
• Collaborate with AI Research to build the infrastructure foundation for future model evaluation, training, and deployment.
• Lead cross-functional technical initiatives and mentor engineers across the organization.

Skills

Go

Java

Python

Rust

C++

Tools

Kubernetes

Kafka

Spark

Flink

Airflow

Iceberg

Job description

Harvey is seeking a Staff Software Engineer to lead the design and development of the Model Infrastructure platform powering AI requests. You will collaborate with AI Research, Product Engineering, and external providers to build a reliable, scalable, and observable foundation for model inference.

You will own high-availability systems, optimize latency, and drive capacity planning while mentoring engineers and advancing the state of the art in frontier models and provider integrations.