Machine Learning Engineer, Reliability

DataJobs; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

fal builds generative media model APIs, and this role helps keep them dependable, secure, and safe. You will own the reliability and operations side of the platform, ensuring model availability, performance, secure serving, and consistently safe generations for production traffic at scale.

You will collaborate with a team that iterates quickly on AI breakthroughs, with an emphasis on making speed reliable and safe for end users.

Qualifications

• 5+ years of professional experience, including 2 years operating production ML or high-scale API systems.
• Experience supporting diffusion models in production.
• Strong systems fundamentals: distributed systems, networking, observability, and incident management.
• Working knowledge of modern generative models (diffusion, transformers) and their production failure modes.
• Familiarity with security and safety practices for ML systems, including abuse prevention or trust and safety engineering experience.
• A bias toward automation, measurement, and blameless postmortems.

Responsibilities

• Own availability, latency, and throughput SLOs across a large fleet of generative model APIs.
• Build monitoring, alerting, and observability to detect ML-specific failures and regressions.
• Harden deployment workflows with canary releases, shadow testing, and automated rollbacks.
• Drive security posture for the model fleet including rate limiting and abuse prevention.
• Operationalize safety systems for generative media at inference time without sacrificing performance.
• Lead incident response for outages and postmortems to prevent recurrence.
• Improve capacity planning, autoscaling, and GPU fleet efficiency.
• Collaborate with model and infra teams to embed reliability, security, and safety into onboarding.

Skills

5+ years experience

Production ML

Diffusion models

Systems fundamentals

Security & safety

Automation & metrics

Tools

Python

Torch

Diffusers

Kubernetes

Fal Python SDK

Job description

fal builds generative media model APIs, and this role helps keep them dependable, secure, and safe. You will own the reliability and operations side of the platform, ensuring model availability, performance, secure serving, and consistently safe generations for production traffic at scale. You will also work with a team that iterates quickly on new AI breakthroughs, with an emphasis on making speed reliable.

What you’ll be responsible for

• Own availability, latency, and throughput SLOs across a large fleet of generative media model APIs serving production traffic at scale.
• Build monitoring, alerting, and observability to detect ML-specific failures, output quality degradation, pipeline breakage, and model regressions before customers experience issues.
• Harden model deployment workflows using canary releases, shadow testing, automated rollbacks, and validation gates so new model versions ship safely.
• Drive the security posture for the model fleet with secure model serving, abuse and misuse detection, rate limiting, and protection against adversarial usage patterns.
• Operationalize safety systems for generative media, including content moderation pipelines, safety classifiers, and guardrails that run reliably at inference time without sacrificing performance.
• Lead incident response for model API outages and degradations, run postmortems, and drive engineering changes to prevent recurrence.
• Improve capacity planning, autoscaling, and GPU fleet efficiency for inference workloads under highly variable traffic.
• Partner with model and infrastructure teams so reliability, security, and safety requirements are built into how new models are onboarded to the platform.

What you’ll bring

• 5+ years of professional experience, including 2 years operating production ML or high-scale API systems, ideally with on-call ownership.
• Experience supporting diffusion models in production.
• Strong systems fundamentals: distributed systems, networking, observability, and incident management.
• Working knowledge of modern generative models (diffusion, transformers) and their production failure modes.
• Familiarity with security and safety practices for ML systems, including abuse prevention or trust and safety engineering experience (strong plus).
• A bias toward automation, measurement, and blameless postmortems.

Tools you’ll use

• Python, torch, diffusers, Kubernetes, fal Python SDK

Role details

• Department: EngineeringML
• Employment type: Full time
• Location: Remote in APAC (India, Australia, or New Zealand)

You’ll have access to fal’s massive GPU cluster for inference and evaluation, and you’ll work with a team focused on quickly iterating and deploying AI improvements while maintaining reliability.