Staff Software Engineer — AI Model Infrastructure

Harvey; San Francisco (CA); On-site; USD 236,000 - 340,000; Full time

Job summary

Harvey in San Francisco is seeking a Staff Software Engineer for the Model Infrastructure team to design and build the systems powering every AI request. You will partner with AI Research, Product Engineering, and external model providers to deliver a platform that is reliable, scalable, and observable.

You will lead the architecture for the Unified Model Controller and Model Selector, drive capacity provisioning and failover, integrate new providers, and mentor engineers while shaping the

Qualifications

• 10+ years in software engineering for large-scale distributed systems.
• Experience operating highly available production services.
• Strong skills in Go, Java, Python, Rust, or C++.
• Deep knowledge of distributed systems, cloud, networking, and observability.
• Experience leading technical projects across teams.
• Ability to balance long-term architecture with pragmatic execution.
• Strong communication and collaboration skills.

Responsibilities

• Lead design and implementation of Harvey's Model Infrastructure platform.
• Ensure high availability, low latency, and operational excellence for AI inference.
• Develop UMC and Model Selector for reliable, cost-aware routing.
• Integrate providers and maintain APIs/SDKs for frontier models.
• Improve observability through dashboards and analytics.
• Mentor engineers and collaborate with cross-functional teams.

Skills

Distributed systems

Production services

Go/Java/Python/Rust/C++

Cloud infrastructure

Leadership experience

Communication skills

Platform development

Tools

Kubernetes

Kafka

Spark

Flink

Job description

Harvey in San Francisco is seeking a Staff Software Engineer for the Model Infrastructure team to design and build the systems powering every AI request. You will partner with AI Research, Product Engineering, and external model providers to deliver a platform that is reliable, scalable, and observable.

You will lead the architecture for the Unified Model Controller and Model Selector, drive capacity provisioning and failover, integrate new providers, and mentor engineers while shaping the