Staff AI Infrastructure Engineer: Scalable Model Platform

Engg; San Francisco (CA); On-site; USD 231,000 - 340,000; Full time

Job summary

Harvey is seeking a Staff Software Engineer to lead the Model Infrastructure team in building the platform powering all AI requests. You will collaborate with AI Research, Product Engineering, and external providers to deliver a reliable, scalable, and observable system that handles model provisioning, routing, and deployment.

The role emphasizes strong distributed-systems expertise, leadership across teams, and a focus on cost-aware, high-performance infrastructure for frontier models.

Qualifications

• 7+ years of software engineering and large distributed systems experience.
• Experience designing and operating highly available production services.
• Strong programming skills in Go, Java, Python, Rust, or C++.
• Deep understanding of distributed systems, cloud infrastructure, networking, and observability.
• Experience leading technical projects across multiple teams.
• Ability to balance long-term architecture with pragmatic execution.
• Strong communication and collaboration skills.
• Passion for building foundational platforms enabling other teams.

Responsibilities

• Lead design and implementation of Harvey's Model Infrastructure platform.
• Ensure high availability, low latency, and operational excellence for AI inference.
• Develop Unified Model Controller and Model Selector for intelligent routing.
• Build provisioning, capacity management, failover, and traffic engineering across providers.
• Integrate new model providers and maintain APIs/SDKs.
• Improve observability with dashboards, alerts, analytics, and telemetry.
• Partner with Product Engineering to support model launches and monitoring.
• Drive infrastructure efficiency via capacity planning and cost visibility.
• Mentor engineers and drive cross-functional technical initiatives.

Skills

Distributed systems

Production services

Go

Java

Python

Rust

C++

Communication

Tools

Kubernetes

Cloud infra

Service mesh

Kafka

Spark

Flink

Airflow

Iceberg

Job description

Harvey is seeking a Staff Software Engineer to lead the Model Infrastructure team in building the platform powering all AI requests. You will collaborate with AI Research, Product Engineering, and external providers to deliver a reliable, scalable, and observable system that handles model provisioning, routing, and deployment.

The role emphasizes strong distributed-systems expertise, leadership across teams, and a focus on cost-aware, high-performance infrastructure for frontier models.