Senior Software Engineer, Machine Learning Infrastructure & Automation

fal-ai; United States; Remote; USD 150,000 - 210,000; Full time

Benefits offered by this job

Health insurance

Dental insurance

Vision insurance

Job summary

fal-ai seeks a seasoned software engineer to build automation and infrastructure that helps machine learning teams develop, test, and release generative AI models and inference services. The role covers CI/CD pipelines, model quality checks, performance benchmarking, deployment safeguards, and developer tools to reduce manual effort and improve reliability.

You will design scalable pipelines, accelerate CI with caching/parallelism, validate outputs across GPUs, benchmark latency/throughput, and

Qualifications

• 5+ years of software engineering experience including production infrastructure or developer tooling.
• Strong Python skills and experience with testing, builds, dependency management.
• Experience designing and operating CI/CD systems with caching and parallel execution.

Responsibilities

• Design and maintain automated testing, validation, and deployment pipelines for models and inference services.
• Speed up CI runs using parallel execution, caching, and efficient compute allocation.
• Validate model outputs across models, GPU architectures, and configurations, and catch quality regressions.
• Benchmark inference latency, throughput, GPU utilization, and cost to detect performance regressions.
• Add automated checks for pricing, billing settings, API schemas, and deployments before release.
• Improve release safety with verification, monitoring, rollback mechanisms, and AI-assisted tooling for diagnosing CI failures and reducing repetitive work.

Skills

Python

CI/CD pipelines

Distributed systems

Troubleshooting

Logging & monitoring

DevOps

Tools

Docker

Cloud platforms

Job description

Role overview

Build the automation and infrastructure that helps machine learning teams develop, test, and release generative AI models and inference services. The work spans CI/CD, model quality checks, performance benchmarking, deployment safeguards, and developer tools, with a focus on reducing manual effort and helping teams ship changes reliably.

Responsibilities

• Design and maintain automated testing, validation, and deployment pipelines for models and inference services.
• Speed up CI runs using techniques such as parallel execution, caching, and efficient compute allocation.
• Validate model outputs across models, GPU architectures, and configurations, and catch quality regressions.
• Benchmark inference latency, throughput, GPU utilization, and cost to detect performance regressions.
• Add automated checks for pricing, billing settings, API schemas, and deployments before release.
• Improve release safety with verification, monitoring, rollback mechanisms, and AI-assisted tools for diagnosing CI failures and reducing repetitive work.

Requirements

• At least five years of software engineering experience, including production infrastructure or developer tooling, with strong Python skills.
• Experience designing and operating CI/CD systems, plus a solid understanding of testing, builds, dependency management, caching, and parallel execution.
• Experience with containers, Docker, and cloud infrastructure.
• Ability to build reliable distributed systems, debug complex infrastructure issues, and use logs, metrics, tracing, and alerts effectively.
• Independence in identifying impactful problems and delivering end-to-end automation.

Nice to have

• Experience with ML infrastructure, PyTorch, GPU workloads, model serving, or NVIDIA multi-GPU systems.
• Background in inference benchmarking, ML quality evaluation, large-scale CI optimization, internal developer platforms, or infrastructure as code.
• Experience with AI coding agents or automated engineering workflows.

Benefits and work setup

The role offers challenging work, learning and growth opportunities, and regular team events and offsites. Health, dental, and vision coverage is available in the United States.