Staff Infra Engineer — LLM Serving

EduRankAI; Kolkata District; Presencial; INR 4.500.000 - 7.000.000; Jornada completa

Descripción de la vacante

EduRankAI is seeking a senior specialist to own the inference stack from model checkpoint to API response, including quantisation and KV-cache management. You will drive multi-region serving, autoscaling, and cost-per-token observability to inform finance decisions.

The role requires hands-on experience with CUDA, Triton, Kubernetes, and profiling tools, ensuring low latency and high throughput in production environments.

Formación

• Experience with production-grade AI inference systems.
• Ability to manage end-to-end inference stacks.
• Experience with multi-region deployment and autoscaling.

Responsabilidades

• Own the inference stack end-to-end (vLLM / SGLang / TensorRT-LLM).
• Own the quantisation pipeline (GPTQ / AWQ / FP8) and the regression eval that says it shipped safely.
• Continuous batching and KV-cache management — including the unpopular hard cases.
• Multi-region serving and autoscaling.
• Cost-per-token observability used by finance.
• SLO ownership for every AI feature that ships.

Conocimientos

Production inference

CUDA

Triton kernels

Networking

RDMA basics

Kubernetes

Helm

Cost-latency intuition

Profiler tools

Herramientas

nsys

py-spy

torch.profiler

CUDA toolkit

Descripción del empleo

Inference & Serving Systems

Own the inference fleet. Sub-100ms first-token latency at our scale, on infrastructure we control. Quantisation, batching, KV cache, scheduling, autoscaling — everything between "model checkpoint" and "API response".You will choose the inference stack, defend the choice, and own the cost-per-token reporting that finance uses to plan. When something goes wrong at 2am, you are who the on-call calls.

Responsibilities

• Own the inference stack end-to-end (vLLM / SGLang / TensorRT-LLM).
• Own the quantisation pipeline (GPTQ / AWQ / FP8) and the regression eval that says it shipped safely.
• Continuous batching and KV-cache management — including the unpopular hard cases.
• Multi-region serving and autoscaling.
• Cost-per-token observability used by finance.
• SLO ownership for every AI feature that ships.

Skills

• Production inference at >1000 RPS
• CUDA / Triton kernels
• Networking + RDMA basics
• Kubernetes + Helm
• Strong cost-vs-latency intuition
• Profiler-fluent (nsys, py-spy, torch.profiler)