Inference Server Engineer

Evollabs Tech; Anupgarh; On-site; INR 3,000,000 - 6,000,000; Full time

Job summary

Evollabs Tech is developing advanced AI accelerator hardware and software for large-scale inference. The team seeks engineers who understand LLM inference servers, runtime paths, and distributed AI systems to integrate our backend into modern serving frameworks.

You will optimize inference execution, support device registration, and extend operator paths across NPU-GPU deployments, contributing to performance and reliability at datacenter scale.

Qualifications

• 5+ years of relevant experience in CS/CE, with strong software fundamentals.
• Hands-on experience modifying or contributing to LLM inference servers.
• Experience running and scaling workloads on large-scale heterogeneous clusters.

Responsibilities

• Integrate our AI accelerator backend into LLM inference servers like vLLM, SGLang, and TensorRT-LLM.
• Implement backend support for device registration, runtime execution, memory management, and custom operators.
• Enable dense and MoE LLM inference on hardware, including attention and expert routing.

Skills

C/C++

Python

PyTorch

LLM inference

Distributed systems

Education

MS/PhD in CS/CE

Tools

vLLM

SGLang

TensorRT-LLM

Job description

We are a technology company focused on designing and developing advanced, customized server hardware solutions optimized for artificial intelligence workloads. Our mission is to accelerate AI innovation by delivering high-performance, scalable, and energy-efficient infrastructure for datacenter-scale inference.

Our chips in development are purpose-built for large-scale AI inference and will be deployed in rack-level systems where multiple devices collaborate to deliver optimal latency, throughput, and efficiency. We are building the next generation of AI infrastructure and are looking for engineers who deeply understand LLM inference servers, model execution runtimes, MoE inference, and distributed AI systems. This role is focused on integrating our AI accelerator platform as a backend into modern LLM inference servers such as vLLM, SGLang, TensorRT-LLM, or similar serving systems.

You will work on adding backend support for our hardware, enabling dense and MoE LLM inference, integrating custom operators and runtime paths, and optimizing the execution of large-scale models across NPU and heterogeneous NPU-GPU systems. You will work at the intersection of LLM serving, accelerator runtime, model execution, distributed inference, and hardware-software co-design.

Responsibilities

• Integrate our AI accelerator backend into LLM inference servers such as vLLM, SGLang, TensorRT-LLM, or similar frameworks.
• Implement backend support for device registration, runtime execution, memory management, custom operators, and KV cache handling.
• Enable dense and MoE LLM inference on our hardware, including attention execution, expert routing, expert execution, and distributed inference support.
• Profile, debug, and optimize inference performance while building correctness, performance, and stress tests for NPU and heterogeneous NPU-GPU deployments.

Requirements

• 5+ years of relevant experience with M.S./Ph.D. degree in CS/CE or equivalent experience.
• Strong C/C++ and Python programming skills.
• Hands-on experience modifying, extending, or contributing to any known LLM inference servers such as vLLM, SGLang, TensorRT-LLM etc.
• Experience running and scaling workloads on large-scale, heterogeneous clusters (CPU + GPU) using distributed training or inference strategies.
• Familiarity with PyTorch internals, custom op registration, tensor execution, model loading, and Hugging Face model integration.
• Familiarity with distributed inference techniques such as tensor parallelism, expert parallelism, data parallelism, multi-device execution, and multi-node serving.

Nice to Have

• Prior merged PRs or significant contributions to vLLM, SGLang, TensorRT-LLM or similar inference server repositories.
• Strong knowledge of accelerator runtime and compiler stack.
• Experience with GPU/NPU memory models, allocation, movement, and heterogeneous device execution.
• Familiarity with distributed communication libraries such as NCCL, RCCL, oneCCL, MPI, UCX, or libfabric.
• Experience with CUDA / Triton Kernels Experience

What We're Not Looking For

This role is not for LLM deployment engineers who have only used inference servers to serve models or has written low-level kernels. We are also not looking for candidates whose experience is limited to running, configuring, or deploying models using vLLM, SGLang, TensorRT-LLM, or similar frameworks. We are looking for engineers who have worked on inference server internals. Are familiar with modifying the codebase, adding backend support, integrating custom operators, changing execution paths, or contributing to the framework itsel

Why Join Us?

• Work on cutting-edge AI hardware designed specifically for large-scale inference.
• Build core inference server backend infrastructure for next-generation NPU and heterogeneous NPU-GPU systems.
• Solve challenging problems at the intersection of LLM serving, AI accelerators, distributed runtime, MoE inference, and datacenter-