Senior AI Reliability Engineer — Platform & Observability

Flatiron Health; Berlin (NH); On-site; USD 140,000 - 190,000; Full time

Job summary

Flatiron Health seeks a Senior AI Reliability Engineer (Platform) to scale AI-enabled workflows across engineering, product, and business teams. You will help build evaluation infrastructure, observability, and guardrails for reliable AI production systems.

You will shape SLOs/SLIs, monitor risk, and collaborate with data science, security, and cloud/platform teams to drive pragmatic AI adoption decisions. Fluency in English and strong communication are essential.

Qualifications

• 5+ years of experience in platform engineering, SRE, machine learning, MLOps, or related field with Python production experience.
• Experience designing experiments, evaluation frameworks, statistical analyses, and quality metrics for ML/AI systems.
• Strong understanding of AI reliability, observability, including logging, tracing, monitoring, drift detection, and production health.

Responsibilities

• Design and continuously improve evaluation frameworks, benchmarks, and automated testing pipelines for AI, LLM-powered, and agentic workflows.
• Define and monitor quality, reliability, safety, performance, and cost metrics for AI systems, including observability and drift detection.
• Develop reliability engineering practices for AI-enabled systems, including SLOs, SLIs, monitoring, incident response, runbooks, and root-cause analysis of AI failure modes.
• Design orchestration, governance, and guardrails for multi-agent AI systems, including agent coordination, permissions, auditability, human oversight, and secure deployment patterns.
• Partner with platform, product, security, engineering, and data science teams to evaluate AI solutions, establish reusable standards, and guide build-vs-buy decisions.
• Support experimentation with emerging AI technologies while helping the organisation make pragmatic, scalable decisions across global teams.

Skills

Fluent in English

Strong communication

Tools

Python

AWS

Kubernetes

CI/CD

Datadog

OpenTelemetry

Databricks

Airflow

SageMaker

Job description

Flatiron Health seeks a Senior AI Reliability Engineer (Platform) to scale AI-enabled workflows across engineering, product, and business teams. You will help build evaluation infrastructure, observability, and guardrails for reliable AI production systems.

You will shape SLOs/SLIs, monitor risk, and collaborate with data science, security, and cloud/platform teams to drive pragmatic AI adoption decisions. Fluency in English and strong communication are essential.