Senior AI Infrastructure Engineer - Scale & Observability

Snorkel AI; New York, San Francisco (NY, CA); Hybrid; USD 150,000 - 190,000; Full time

Job summary

Snorkel AI is hiring AI Engineers to design and operate the core infrastructure for large-scale, non-deterministic AI experiments and production workloads. You will build synthetic data pipelines, evaluation systems, and distributed compute across cloud environments.

Join a fast-paced team shaping agent-first AI platforms, with a focus on observability, cost attribution, and reliable delivery of AI workflows from development to production.

Qualifications

• 5+ years building production software systems, with AI/ML infra or distributed systems experience.
• Experience operating non-deterministic AI workloads at scale in production.
• Strong Python and API development skills.
• Proficiency with cloud platforms (AWS preferred) and orchestration tools.
• Experience with workflow systems (Prefect, Airflow, Dagster, Ray, Kubernetes).
• Ability to design observable, reliable, cost-aware AI platforms.

Responsibilities

• Design and build infra for large-scale agentic workloads and multi-step agents with tools, environments, and simulations.
• Develop synthetic data generation and labeling systems for high-quality datasets.
• Create evaluation infrastructure to measure AI behavior across models and prompts.
• Build orchestration and distributed compute across heterogeneous environments.
• Develop agent simulation environments with provisioning, lifecycle management, and scalable execution.
• Operate LLM infra for routing, caching, failover, and cost attribution across providers.
• Instrument workloads to capture traces, latency, cost, and reliability for debuggability.
• Make non-deterministic workloads reproducible and measurable for comparisons and improvements.
• Enhance developer experience with APIs, SDKs, and workflow abstractions.

Skills

Python

Distributed systems

AI/ML infrastructure

APIs

AWS

Kubernetes

Observability

Cost management

Communication

Tools

Prefect

Airflow

Dagster

Ray

Docker

Job description

Snorkel AI is hiring AI Engineers to design and operate the core infrastructure for large-scale, non-deterministic AI experiments and production workloads. You will build synthetic data pipelines, evaluation systems, and distributed compute across cloud environments.

Join a fast-paced team shaping agent-first AI platforms, with a focus on observability, cost attribution, and reliable delivery of AI workflows from development to production.