Senior AI Inference Platform Engineer

Cloudflare; Austin (TX); Hybrid; USD 180,000 - 240,000; Full time

Job summary

Cloudflare is seeking a highly autonomous systems engineer to design and build the core infrastructure powering AI inference across our global network. You will work with AI/ML engineers, hardware partners, and product teams to tackle sub-second model cold starts, multi‑accelerator scheduling, and efficient KV cache management.

We’re building an AI inference platform embedded in the internet fabric; you’ll lead cross‑functional projects, improve reliability, and mentor junior engineers in a

Qualifications

• Proven experience in distributed, high‑performance systems.
• Expert proficiency in Rust, especially in an asynchronous environment.
• Deep understanding of networking protocols (TCP, HTTP, WebSocket).
• Experience with scaling and performance optimization in distributed environments.

Responsibilities

• Platform Architecture: Develop and maintain core components of the serverless inference platform to ensure high availability and scalability for Cloudflare users.
• Optimization & Performance: Optimize the model scheduling system to significantly increase efficiency and resource utilization across our inference infrastructure, and implement improvements to request routing logic to reduce latency for end‑users.
• System Reliability: Drive significant, measurable improvements in the platform’s reliability and resilience by identifying and mitigating systemic risks.
• Observability: Expand and refine the observability stack (metrics, logging, tracing) and fine‑tune alerts to proactively identify and resolve production issues.
• Technical Leadership: Lead complex, cross‑functional technical projects from initial concept and design through final deployment and operationalization.
• Mentorship: Act as a mentor to junior engineers and actively contribute to cultivating a strong, collaborative engineering culture within the team.

Skills

Systems engineering

Rust programming

Networking protocols

Load balancing

Distributed systems

Tools

Kubernetes

Nomad

Job description

Cloudflare is seeking a highly autonomous systems engineer to design and build the core infrastructure powering AI inference across our global network. You will work with AI/ML engineers, hardware partners, and product teams to tackle sub-second model cold starts, multi‑accelerator scheduling, and efficient KV cache management.

We’re building an AI inference platform embedded in the internet fabric; you’ll lead cross‑functional projects, improve reliability, and mentor junior engineers in a