Senior AI Engineer / Staff — Production-Scale AI Infra

Snorkel AI

San Francisco (CA)

On-site

USD 180,000 - 260,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Snorkel AI is hiring AI Engineers to design and operate the infrastructure for large-scale agentic workloads and AI experimentation. You will build synthetic data pipelines, evaluation systems, and distributed compute for thousands to millions of experiments, spanning environments, tools, and models.

You will work on observability, reproducibility, and performance while collaborating with research, product, and engineering teams to turn experiments into reusable platform capabilities.

Qualifications

  • 5+ years building production software systems.
  • Experience operating non-deterministic AI or ML workloads in production or at significant scale.
  • Experience building infrastructure for experimentation, evaluation, model development, synthetic data, agentic workflows, training, inference, or production ML systems.
  • Strong proficiency in Python and experience building production-quality APIs, services, and developer tooling.
  • Strong background in distributed systems and cloud platforms (AWS preferred).
  • Experience with workflow or distributed execution frameworks such as Prefect, Airflow, Dagster, Ray, Kubernetes, or similar systems.
  • Strong understanding of production system fundamentals - observability, telemetry, reliability, performance, debugging, incident response, and cost management.
  • Ability to reason about AI system quality beyond traditional service metrics, including evaluation design, experiment reproducibility, behavioral regressions, and model or agent variability.
  • Track record of leading complex engineering initiatives, influencing stakeholders, and delivering measurable impact.
  • Ability to work in a fast-paced environment with strong technical communication skills.
  • Fluency with modern AI and developer tooling and a willingness to rapidly evaluate and adopt new models, frameworks, infrastructure, and techniques as the ecosystem evolves.

Responsibilities

  • Design and build infrastructure for running large-scale agentic workloads, including multi-step agents interacting with tools, external services, sandboxes, and simulated environments.
  • Build scalable synthetic data generation and automated labeling systems that allow teams to create, refine, and evaluate high-quality training and evaluation datasets.
  • Design evaluation infrastructure for measuring AI system behavior across models, prompts, tools, environments, and multi-step trajectories - including reproducible experiments, benchmark execution, regression detection, and continuous evaluation.
  • Build orchestration and distributed compute systems for running thousands to millions of AI experiments and simulations reliably across heterogeneous compute environments.
  • Develop infrastructure for agent simulation environments, including environment provisioning, isolation, lifecycle management, and scalable execution.
  • Build and operate LLM infrastructure for routing, rate limiting, retries, caching, provider failover, cost attribution, and efficient execution across multiple model providers.
  • Instrument agent and model workloads so failures are observable and debuggable - capturing traces, model interactions, tool calls, environment state, evaluation results, latency, reliability, and cost.

Skills

5+ years production software
AI/ML in production
Python
distributed systems
cloud platforms (AWS)
workflow frameworks (Prefect, Airflow,
Dagster
Ray
production observability
tech leadership

Tools

AWS
Kubernetes
Dagster
Airflow
Ray
Prefect

Job description

Snorkel AI is hiring AI Engineers to design and operate the infrastructure for large-scale agentic workloads and AI experimentation. You will build synthetic data pipelines, evaluation systems, and distributed compute for thousands to millions of experiments, spanning environments, tools, and models.

You will work on observability, reproducibility, and performance while collaborating with research, product, and engineering teams to turn experiments into reusable platform capabilities.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Platform Engineer – Scale & Observability
Senior AI Platform Engineer – Scale & Observability

Fdetimes • New York (NY)

On-site
USD 180,000 - 240,000
Senior AI Platform Engineer — Scalable Agent & LLM Infra (Hybrid)
Senior AI Platform Engineer — Scalable Agent & LLM Infra (Hybrid)

Soter Insure • New York (NY)

Hybrid
USD 180,000 - 280,000
Senior AI Systems Engineer — Remote, Scalable Platforms
Senior AI Systems Engineer — Remote, Scalable Platforms

ContentBuffer • San Francisco (CA)

Hybrid
USD 208,000 - 315,000
Senior Software Engineer, AI Foundations & Scale
Senior Software Engineer, AI Foundations & Scale

Snorkel AI • United States

Remote
USD 180,000 - 230,000
Remote Research Scientist, AI Data & Agentic Environments
Remote Research Scientist, AI Data & Agentic Environments

Snorkel AI • San Francisco (CA)

Remote
USD 240,000 - 375,000
Senior Software Engineer — Remote AI Systems
Senior Software Engineer — Remote AI Systems

Snorkel AI • Oregon

Hybrid
USD 208,000 - 315,000
Senior Software Engineer - AI Data Systems (Remote)
Senior Software Engineer - AI Data Systems (Remote)

Snorkel AI • United States

Remote
USD 208,000 - 315,000
Senior Applied AI Engineer — Enterprise AI Solutions
Senior Applied AI Engineer — Enterprise AI Solutions

JobLarper • New York (NY), San Francisco (CA)

Hybrid
USD 180,000 - 260,000
Senior Software Engineer - AI Data Systems (Remote)
Senior Software Engineer - AI Data Systems (Remote)

Snorkel AI • United States

Remote
USD 208,000 - 315,000
Senior Manager, Forward Deployed AI Engineering — Remote
Senior Manager, Forward Deployed AI Engineering — Remote

JoinStartup • New York (NY), San Francisco (CA)

Hybrid
USD 260,000 - 350,000
Equity
Benefits