Staff Software Engineer, AI Model Infrastructure

Speedrun Talent Network; San Francisco, Northern (CA, KY); Hybrid; USD 231,000 - 340,000; Full time

Job summary

Harvey is seeking a Staff Software Engineer to lead the design and development of the Model Infrastructure powering AI requests. You will collaborate with AI Research, Product Engineering, and external providers to build a scalable, observable platform with high reliability and efficiency.

You will drive capacity planning, traffic routing, and provider integration, while mentoring engineers and advancing the foundational AI platform for future deployments.

Qualifications

• 7+ years of software engineering experience building large-scale distributed systems.
• Experience designing and operating highly available production services.
• Strong programming skills in Go, Java, Python, Rust, or C++.

Responsibilities

• Design and implement Harvey's Model Infrastructure platform.
• Ensure high availability and low latency for AI inference.
• Develop and improve Unified Model Controller (UMC) and Model Selector.
• Manage model provisioning, capacity, failover, and traffic engineering across providers.
• Integrate new model providers and maintain APIs/SDKs.
• Enhance observability with dashboards, alerts, and cost reporting.
• Collaborate with Product Engineering on model launches and monitoring.
• Drive infrastructure efficiency through capacity planning and cost visibility.
• Collaborate with AI Research on future model evaluation and deployment.
• Lead cross-functional technical initiatives and mentor engineers.

Skills

7+ years

Distributed systems

Go

Java

Python

Rust

C++

Leadership

Tools

Kubernetes

Job description

Harvey is seeking a Staff Software Engineer to lead the design and development of the Model Infrastructure powering AI requests. You will collaborate with AI Research, Product Engineering, and external providers to build a scalable, observable platform with high reliability and efficiency.

You will drive capacity planning, traffic routing, and provider integration, while mentoring engineers and advancing the foundational AI platform for future deployments.