Machine Learning Engineer - Diffusion

Bagel Labs

Canada

Remote

CAD 167,014 - 250,522

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Top of the market compensation
Deeply technical culture
Full remote flexibility
Paid travel opportunities to ML conferences

Job summary

A leading machine learning research lab is looking for an expert in distributed systems to design scalable infrastructure for diffusion models. The ideal candidate will have substantial experience in ML serving and architectural knowledge of diffusion techniques. This role offers top compensation, full remote flexibility within North America, and opportunities for professional growth at leading conferences.

Qualifications

  • Minimum 5 years of experience with distributed systems and production ML serving.
  • Hands-on experience with diffusion model frameworks in production.
  • Deep understanding of diffusion model architectures.

Responsibilities

  • Design and implement distributed diffusion model inference systems.
  • Architect high-availability clusters for diffusion model serving.
  • Build monitoring and observability systems for distributed inference.

Skills

Distributed systems expertise
Production ML serving experience
Diffusion model frameworks familiarity
High-memory workload orchestration
Optimizing generation latency
Attention optimization techniques
Adaptation techniques knowledge
Variable-resolution generation

Job description

We are Bagel Labs - a distributed machine learning research lab working towards open-source superintelligence. We ignore years of experience and pedigree. If you have high agency - meaning your default assumption is that you can control the outcome of whatever situation you are in - we want to hear from you. Every requirement below is flexible for a candidate with high enough agency and tolerance for ambiguity.

Overview

You will design and optimize a distributed diffusion model training and serving system. Your focus is on building scalable, fault-tolerant infrastructure that can serve open-source diffusion models across multiple nodes and regions, with efficient support for adaptation techniques.

Key Responsibilities
  • Design and implement distributed diffusion model inference systems for image, video, and multimodal generation across multiple nodes and regions.
  • Architect high-availability clusters for diffusion model serving with automatic failover, load balancing, and dynamic batching for variable-resolution outputs.
  • Build monitoring and observability systems for distributed diffusion inference (denoising steps, memory usage, generation latency, CLIP score tracking).
  • Integrate with open-source diffusion frameworks (Diffusers, ComfyUI, Invoke AI) and optimize for production-scale serving.
  • Implement and optimize cutting-edge techniques: rectified flow models, consistency distillation, and progressive distillation for few-step generation.
  • Design distributed systems for ControlNet, IP-Adapter, and multi-modal conditioning at scale.
  • Build infrastructure for efficient LoRA/LyCORIS adaptation serving with hot-swapping and memory-efficient merging.
  • Optimize VAE decoding pipelines and implement tiled/windowed generation for ultra-high-resolution outputs.
  • Document architectural decisions, review code, and publish technical deep-dives on blog.bagel.com.
Who You Might Be

You have a deep understanding of distributed systems and diffusion model architectures. You\'re excited about the rapid evolution from DDPM to flow matching and consistency models. You enjoy architecting scalable infrastructure that can handle the unique challenges of diffusion models - from variable compute requirements per timestep to efficient caching of intermediate states.

Desired Skills
  • At least 5 years of experience with distributed systems and production ML serving.
  • Hands-on experience with diffusion model frameworks (Diffusers, ComfyUI, or similar) in production environments.
  • Deep understanding of diffusion model architectures (U-Net, DiT, rectified flows, consistency models).
  • Experience with distributed GPU orchestration for high-memory workloads.
  • Proven record of optimizing generation latency (classifier-free guidance, DDIM/DPM solvers, distillation techniques).
  • Experience with attention optimization techniques (Flash Attention, xFormers, memory-efficient attention).
  • Strong understanding of adaptation techniques (LoRA, LyCORIS, textual inversion, DreamBooth).
  • Expertise in handling variable-resolution generation and dynamic batching strategies.
What We Offer
  • Top of the market compensation.
  • A deeply technical culture where bold, frontier ideas are debated, stress-tested, and built.
  • Full remote flexibility within North American time zones.
  • Ownership of work that can set the direction for decentralized AI.
  • Paid travel opportunities to the top ML conferences around the world.

Please apply via our careers page. Note: we do not share application links here.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Distributed Diffusion ML Engineer — Remote
Senior Distributed Diffusion ML Engineer — Remote

Bagel Labs • Canada

Remote
CAD 167,014 - 250,522
Top of the market compensation
Deeply technical culture
Full remote flexibility
+1
Member of Technical Staff - World Models
Member of Technical Staff - World Models

Veeda AI • Toronto

On-site
CAD 120,000 - 180,000
Software Engineer: AI
Software Engineer: AI

Aloe • Vancouver

Hybrid
CAD 90,000 - 130,000
Competitive compensation
Benefits
Career growth opportunities
Senior Research Scientist
Senior Research Scientist

adaption • Toronto

Remote
CAD 120,000 - 180,000
Flexible work
Adaption Passport
Lunch Stipend
+1
Modelling Resident
Modelling Resident

adaption • Toronto

Remote
CAD 120,000 - 180,000
Staff + Senior Software Engineer, Inference
Staff + Senior Software Engineer, Inference

Anthropic Limited • Canada

Hybrid
CAD 140,000 - 210,000
Principal ML Engineer
Principal ML Engineer

HRB • Canada

On-site
CAD 140,000 - 210,000
RRSP matching
Software Engineer - GPU Networking & Distributed Systems
Software Engineer - GPU Networking & Distributed Systems

Baseten • Toronto

On-site
CAD 230,000 - 461,000
Competitive compensation with equity
Paid parental leave
Flexible PTO including winter break
+1
AI / ML Engineer
AI / ML Engineer

Testlify, Inc. • Canada

On-site
CAD 100,000 - 170,000
Health Insurance
Flexible Working Style
Staff + Senior Software Engineer, Inference
Staff + Senior Software Engineer, Inference

FDE Times • Canada

Hybrid
CAD 120,000 - 180,000