Remote ML Reliability Engineer — Generative AI Platform

Fal; New Zealand; On-site; NZD 150,000 - 190,000; Full time

Benefits offered by this job

Interesting and challenging work

Competitive salary and equity

Visa sponsorship

Relocation to San Francisco (hybrid/US

Health, dental, and vision insurance (

Job summary

fal is building a world-class generative media platform and is seeking a Machine Learning Engineer, Reliability to own availability, latency, and security of our model APIs at scale. You will design monitoring, incident response, and deployment safeguards for a high-velocity GPU fleet.

You will collaborate with cross-functional teams to improve capacity, automate operations, and ensure safety without compromising performance. This remote-first role welcomes candidates in multiple regions.

Qualifications

• 3+ years of professional experience with production ML or high-scale API systems.
• Strong systems fundamentals: distributed systems, networking, observability, incident management.
• Working knowledge of modern generative models and their production failure modes.
• A bias toward automation, measurement, and blameless postmortems.

Responsibilities

• Own availability, latency, and throughput SLOs across a large fleet of generative media model APIs serving production traffic at scale
• Build the monitoring, alerting, and observability needed to catch ML-specific failures, output quality degradation, pipeline breakage, model regressions before customers do
• Harden model deployment workflows with canary releases, shadow testing, automated rollbacks, and validation gates so new model versions ship safely
• Drive the security posture of the model fleet: secure model serving, abuse and misuse detection, rate limiting, and protection against adversarial usage patterns
• Operationalize safety systems for generative media, content moderation pipelines, safety classifiers, and guardrails that run reliably at inference time without compromising performance
• Lead incident response for model API outages and degradations, run postmortems, and drive the engineering work that prevents recurrence
• Improve capacity planning, autoscaling, and GPU fleet efficiency for inference workloads under highly variable traffic
• Partner with model and infrastructure teams to make reliability, security, and safety requirements part of how new models get onboarded to the platform

Skills

Distributed systems

Networking

Observability

Incident management

On-call ownership

Tools

Python

PyTorch

diffusers

Kubernetes

fal Python SDK

Job description

fal is building a world-class generative media platform and is seeking a Machine Learning Engineer, Reliability to own availability, latency, and security of our model APIs at scale. You will design monitoring, incident response, and deployment safeguards for a high-velocity GPU fleet.

You will collaborate with cross-functional teams to improve capacity, automate operations, and ensure safety without compromising performance. This remote-first role welcomes candidates in multiple regions.