Inference Optimization Engineer (Sr Engineer / Sta5 Engineer / Principal Engineer)

Rafay; India; Presencial; INR 3.000.000 - 4.500.000; Jornada completa

Descripción de la vacante

Rafay is seeking an inference optimization engineer to bridge ML research and low-level systems. The role focuses on serving optimized models under strict speed and cost constraints, balancing TTFT and tokens per second with accuracy and hardware cost.

You will work on model compression, runtime tuning, memory management, profiling, and distributed multi-GPU deployments. Strong Python/Go skills and CUDA/Triton kernel familiarity are valued.

Formación

• 4+ years in systems programming, HPC, or ML infra.
• Strong Python or Go, with CUDA/Triton kernel familiarity.
• Production experience with modern inference engines and profiling tools.
• Deep understanding of transformer architectures and roofline models.

Responsabilidades

• Model Compression: quantization, pruning, distillation.
• Runtime serving: tune high-throughput frameworks (vLLM, TensorRT-LLM, SGLang).
• Memory and caching: optimize KV cache, batching, and decoding.
• Profiling: use Nsight, PyTorch profiler, and Triton metrics to locate bottlenecks.
• Hardware co-optimization: tune kernels and multi-GPU distribution.

Conocimientos

Python

Go

CUDA kernels

vLLM

TensorRT-LLM

SGLang

Nsight

Herramientas

Triton Inference Server

Nsight Systems

Descripción del empleo

Full Time

India

Role Overview

This engineering role begins where model training ends. Once a machine learning or large language model (LLM) is trained, an inference optimization engineer figures out how to serve it efficiently to real-world users under strict speed and cost limits. The position sits between research and low-level systems engineering. It requires balancing speed—measured as Time-to-First-Token (TTFT) and tokens per second—with output accuracy and hardware costs.

Key Responsibilities

• Model Compression: Apply techniques like quantization (FP8, INT8, INT4), pruning, and distillation to shrink model size without losing accuracy.
• Runtime & Serving Tuning: Configure and optimize high-throughput serving frameworks such as vLLM, SGLang, TensorRT-LLM, or Triton Inference Server.
• Memory & Caching Management: Optimize KV cache usage, continuous batching, and speculative decoding to handle long context windows and multiple concurrent users.
• Profiling & Benchmarking: Use tools like Nsight Systems, PyTorch Profiler, and Triton Metrics to identify memory bandwidth or compute bottlenecks on GPU/TPU clusters.
• Hardware Co-Optimization: Tune custom kernels (CUDA/Triton) and manage distributed setups across multi-GPU clusters using tensor and pipeline parallelism

Key Requirements & Qualifications

• Experience: 4+ years in systems programming, high-performance computing (HPC), or ML infrastructure.
• Core Languages: Strong proficiency in Python or Go, with familiarity reading or writing CUDA/Triton kernels.
• Frameworks: Hands-on production experience with modern inference engines (vLLM, TensorRT-LLM, SGLang) and profiling tools (Nsight).
• Foundational Knowledge: Deep understanding of transformer architectures, memory hierarchies, arithmetic intensity, and roofline performance models.