Staff ML Infrastructure Architect

Cacheflow; Palo Alto (CA); On-site; USD 218,000 - 285,000; Full time

Job summary

Quince is seeking a Staff Engineer, Machine Learning Infrastructure to join our growing team in Palo Alto. You will design and operate production-grade ML systems at scale, from training pipelines to high-throughput inference serving, emphasizing extensibility, observability, and reliability.

You will mentor engineers, shape platform standards (CI/CD for ML, IaC, model versioning), and drive cost-aware performance optimizations.

Qualifications

• 8+ years of industry experience, with at least 4+ years in ML Infrastructure, MLOps, or large-scale Data Platform engineering.
• Proven track record designing and building MLOps platforms that support the full model lifecycle—from data ingestion and distributed training to real-time inference and model governance.
• Deep expertise in cloud-native infrastructure (AWS), Kubernetes (EKS), Docker, and Infrastructure as Code tools (Terraform/Pulumi).
• Hands-on mastery of ML frameworks such as PyTorch, TensorFlow, Kubeflow, or SageMaker, with strong opinions on building a cohesive, high-leverage developer experience.
• Experience building Feature Stores and high-throughput data pipelines (Spark, Flink, Kafka).
• CI/CD for ML, including model versioning, experiment tracking, and deployment strategies such as blue-green and canary rollouts.
• Strong operational instincts with on-call discipline and reliability.
• Startup mindset; capable of handling ambiguity and rapid pace.

Responsibilities

• Architect the ML Infrastructure Foundation: end-to-end design of Quince’s ML platform, modular and scalable for long-term extensibility.
• Build the 'Paved Road' for Production: develop the core developer experience for Data Scientists and AI Researchers to move from idea to production with minimal friction.
• Drive Technical Excellence Across the Stack: set and uphold CI/CD for ML, IaC, model versioning, experiment tracking, and deployment strategies.
• Own High-Impact System Design Decisions: lead evaluation and selection of core platform components with build-vs-buy balance.
• Optimize Compute Performance & Cost: design GPU utilization optimizations, model batching, and cloud cost controls.
• Ensure Production Scalability & Reliability: ML serving infrastructure that handles traffic surges with monitoring and automated recovery.
• Mentor and Elevate the Engineering Team: code reviews and pairing to raise the technical bar.
• Champion Operational Excellence: RCA for production failures and a rigorous on-call culture.

Skills

ML Infrastructure

MLOps platforms

CI/CD for ML

Cloud-native infra

DevOps collaboration

Tools

AWS

Kubernetes (EKS)

Docker

Terraform

Pulumi

Kubeflow

TensorFlow

PyTorch

SageMaker

Spark

Kafka

Job description

Quince is seeking a Staff Engineer, Machine Learning Infrastructure to join our growing team in Palo Alto. You will design and operate production-grade ML systems at scale, from training pipelines to high-throughput inference serving, emphasizing extensibility, observability, and reliability.

You will mentor engineers, shape platform standards (CI/CD for ML, IaC, model versioning), and drive cost-aware performance optimizations.