Senior AI Inference Systems Engineer

Cloudflare; New York (NY); Hybrid; USD 185,000 - 254,000; Full time

Benefits offered by this job

Equity plan

Health insurance

401(k) Retirement Savings Plan

Parental leave

Job summary

Cloudflare is building a world-class AI inference platform that powers real-time voice, LLMs, and user-deployed models across its global network. You will design core infrastructure and collaborate with AI/ML teams, hardware partners, and product groups to tackle foundational challenges in distributed systems.

As a senior systems engineer, you will drive sub-second model cold starts, multi-accelerator workload scheduling, and efficient KV cache management while shaping the platform's

Qualifications

• Proven experience in systems engineering, with a focus on distributed, high-performance systems.
• Expert proficiency in Rust, especially in asynchronous environments.
• Deep understanding of networking protocols (TCP, HTTP, WebSocket).
• Experience with scaling and performance optimization in distributed environments.

Responsibilities

• Platform Architecture: develop and maintain core components of the serverless inference platform for high availability and scalability.
• Optimization & Performance: improve model scheduling and routing to reduce latency and increase efficiency.
• System Reliability: drive measurable improvements in platform reliability and resilience.
• Observability: expand metrics/logging/tracing; refine alerts for proactive issue resolution.
• Technical Leadership: lead cross-functional projects from concept to deployment.
• Mentorship: guide junior engineers and foster collaborative culture.

Skills

Rust

Distributed systems

Networking protocols

Async I/O

Tools

Kubernetes

Nomad

gRPC

Job description

Cloudflare is building a world-class AI inference platform that powers real-time voice, LLMs, and user-deployed models across its global network. You will design core infrastructure and collaborate with AI/ML teams, hardware partners, and product groups to tackle foundational challenges in distributed systems.

As a senior systems engineer, you will drive sub-second model cold starts, multi-accelerator workload scheduling, and efficient KV cache management while shaping the platform's