Senior ML Engineer - Production Inference & Low-Latency AI

Cloudflare, Inc.; New York (NY); Hybrid; USD 194,000 - 266,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

401(k) Retirement Savings Plan

Employee Stock Participation Plan

Job summary

Cloudflare, Inc. is building production-grade ML inference to power experiences across its global network. This hybrid role in New York, NY focuses on low latency, reliability, and efficient resource use for models across GPUs and accelerators, from frontier LLMs to customer-deployed models.

You’ll drive benchmarking, validation, and deployment workflows while mentoring engineers. You will collaborate with product and engineering teams to translate customer needs into scalable ML capabilities

Qualifications

• Experience building ML models in production environments.
• Strong Python skills and ML framework proficiency (PyTorch, TensorFlow, JAX or equivalent).
• Hands-on inference optimization for large-scale models (quantization, batching, caching, compilation, runtime tuning).
• Experience with large-scale inference serving frameworks/runtimes (e.g., SGLang, vLLM, TensorRT-LLM, ONNX Runtime, Triton, llama.cpp).
• Familiarity with LLMs, speech/vision/embeddings/multimodal models and retrieval-augmented generation.
• Experience optimizing models for GPUs or specialized accelerators.
• Strong production ML foundations: evaluation, monitoring, regressions, rollout safety, reliability.
• Ability to work across ML and systems boundaries; distributed systems and serverless familiarity.
• Track record leading complex technical projects and mentoring engineers.

Responsibilities

• Develop, optimize, and productionize ML models for Cloudflare’s serverless inference platform.
• Create benchmarking and evaluation frameworks for latency, throughput, cost, and model behavior.
• Improve inference performance using quantization, batching, caching, compilation, runtime tuning, and accelerator awareness.
• Collaborate with systems engineers to integrate models into distributed inference infrastructure across GPUs and accelerators.
• Advance deployment workflows including validation, rollout safety, observability, and testing.
• Partner with product and engineering teams to translate customer requirements into scalable ML capabilities for Workers AI.
• Mentor engineers and contribute to technical direction to raise production ML quality.

Skills

Machine learning models

Python

ML frameworks

Inference optimization

Distributed systems

GPU acceleration

Model benchmarking

Mentoring engineers

Tools

Python

PyTorch

TensorFlow

JAX

SGLang

vLLM

TensorRT-LLM

ONNX Runtime

Triton

llama.cpp

Workers AI

Job description

Cloudflare, Inc. is building production-grade ML inference to power experiences across its global network. This hybrid role in New York, NY focuses on low latency, reliability, and efficient resource use for models across GPUs and accelerators, from frontier LLMs to customer-deployed models.

You’ll drive benchmarking, validation, and deployment workflows while mentoring engineers. You will collaborate with product and engineering teams to translate customer needs into scalable ML capabilities