Remote Reliability Engineer - Generative ML APIs

DataJobs; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

fal builds generative media model APIs, and this role helps keep them dependable, secure, and safe. You will own the reliability and operations side of the platform, ensuring model availability, performance, secure serving, and consistently safe generations for production traffic at scale.

You will collaborate with a team that iterates quickly on AI breakthroughs, with an emphasis on making speed reliable and safe for end users.

Qualifications

• 5+ years of professional experience, including 2 years operating production ML or high-scale API systems.
• Experience supporting diffusion models in production.
• Strong systems fundamentals: distributed systems, networking, observability, and incident management.
• Working knowledge of modern generative models (diffusion, transformers) and their production failure modes.
• Familiarity with security and safety practices for ML systems, including abuse prevention or trust and safety engineering experience.
• A bias toward automation, measurement, and blameless postmortems.

Responsibilities

• Own availability, latency, and throughput SLOs across a large fleet of generative model APIs.
• Build monitoring, alerting, and observability to detect ML-specific failures and regressions.
• Harden deployment workflows with canary releases, shadow testing, and automated rollbacks.
• Drive security posture for the model fleet including rate limiting and abuse prevention.
• Operationalize safety systems for generative media at inference time without sacrificing performance.
• Lead incident response for outages and postmortems to prevent recurrence.
• Improve capacity planning, autoscaling, and GPU fleet efficiency.
• Collaborate with model and infra teams to embed reliability, security, and safety into onboarding.

Skills

5+ years experience

Production ML

Diffusion models

Systems fundamentals

Security & safety

Automation & metrics

Tools

Python

Torch

Diffusers

Kubernetes

Fal Python SDK

Job description

fal builds generative media model APIs, and this role helps keep them dependable, secure, and safe. You will own the reliability and operations side of the platform, ensuring model availability, performance, secure serving, and consistently safe generations for production traffic at scale.

You will collaborate with a team that iterates quickly on AI breakthroughs, with an emphasis on making speed reliable and safe for end users.