Senior AI Inference Systems Engineer - Edge & Cloud

Cloudflare; Austin (TX); Hybrid; USD 180,000 - 230,000; Full time

Job summary

Cloudflare is seeking a high‑agency systems engineer to design and build the AI inference platform that powers real‑time workloads across Cloudflare’s global network. You will tackle sub‑second model cold starts, multi‑accelerator scheduling, and efficient KV cache management with a team of AI/ML engineers, hardware partners, and product groups.

We expect expertise in distributed systems, Rust, networking protocols, and performance tuning, plus leadership to guide cross‑functional projects and

Qualifications

• Proven experience in systems engineering, with a primary focus on distributed, high‑performance systems.
• Expert proficiency in Rust programming, particularly in an asynchronous environment.
• Deep understanding and hands‑on experience with relevant networking and application protocols (e.g., TCP, HTTP, WebSocket).
• Solid experience with scaling and performance optimization techniques, including load balancing and caching in a distributed environment.

Responsibilities

• Platform Architecture: Develop and maintain core components of the serverless inference platform to ensure high availability and scalability for Cloudflare users.
• Optimization & Performance: Optimize the model scheduling system to significantly increase efficiency and resource utilization across our inference infrastructure, and implement improvements to request routing logic to reduce latency for end‑users.
• System Reliability: Drive significant, measurable improvements in the platform’s reliability and resilience by identifying and mitigating systemic risks.
• Observability: Expand and refine the observability stack (metrics, logging, tracing) and fine‑tune alerts to proactively identify and resolve production issues.
• Technical Leadership: Lead complex, cross‑functional technical projects from initial concept and design through final deployment and operationalization.
• Mentorship: Act as a mentor to junior engineers and actively contribute to cultivating a strong, collaborative engineering culture within the team.

Skills

Distributed systems

Rust

Networking protocols

Performance optimization

Tools

Kubernetes

Nomad

Job description

Cloudflare is seeking a high‑agency systems engineer to design and build the AI inference platform that powers real‑time workloads across Cloudflare’s global network. You will tackle sub‑second model cold starts, multi‑accelerator scheduling, and efficient KV cache management with a team of AI/ML engineers, hardware partners, and product groups.

We expect expertise in distributed systems, Rust, networking protocols, and performance tuning, plus leadership to guide cross‑functional projects and