Senior Inference Engineer

Aleroy; United Arab Emirates; On-site; AED 360,000 - 540,000; Full time

Benefits offered by this job

24 days annual leave

Health insurance

Modern office in Yas Creative Hub

Job summary

Aleroy aims to accelerate AI model deployment with a unified platform. This senior role focuses on building and optimising the inference serving stack, driven by hands-on work on the stack and measurement-driven improvements.

You will lead performance engineering across GPUs, MoE models, and distributed serving, setting standards and guiding engineers with a practical, outcome-oriented approach. The role is based in our Yas Creative Hub office in the UAE.

Qualifications

• Strong hands-on experience with LLM inference-serving in production.
• Deep knowledge of vLLM, SGLang, TensorRT-LLM.
• Experience with MoE models including expert parallelism and all-to-all communication.
• GPU performance engineering using CUDA or Triton.
• Experience with model quantization and performance optimisation.
• Strong benchmarking and measurement discipline across serving performance and cost.
• English level: Advanced (C1).

Responsibilities

• Own and improve the FAR Labs inference serving stack with optimisations.
• Improve throughput, latency, memory efficiency, and serving cost while maintaining model-quality standards.
• Identify and remove GPU performance bottlenecks across the stack using profiling and optimization.
• Drive efficient MoE serving including expert parallelism and load balancing.
• Implement and evaluate quantization and other model-optimisation techniques.
• Own rigorous, reproducible benchmarking across serving performance and cost.
• Build instrumentation to understand system behaviour and validate improvements.
• Make technical decisions around routing, scheduling, and distributed serving.

Skills

LLM serving

Performance tuning

Distributed systems

CUDA/Triton

MoE models

Quantization

Benchmarking

Leadership

Instrumentation

English (C1)

Tools

vLLM

SGLang

TensorRT-LLM

CUDA

Triton

Job description

About Aleroy

Aleroy is building a unified platform for high-performance AI inference, making it easier for companies to deploy and run AI models at scale.
Through a single API, Aleroy provides access to ready-to-use and custom AI models across cloud infrastructure, dedicated capacity, and customers' own hardware. Our platform optimizes performance, cost, and deployment flexibility while giving enterprises control over where and how their AI workloads run.
We work in small teams, collaborate directly, and focus on solving complex engineering challenges to build reliable, scalable AI infrastructure.

About the Role

You will work across inference serving, GPU performance, distributed systems, benchmarking, and model optimisation to make our serving infrastructure faster and more efficient while maintaining a clear quality bar.
This is the senior technical seat for AI serving at Aleroy. You will set the serving and measurement roadmap, establish engineering standards, and guide engineers working across serving, metrics, and testing.
The role is highly hands-on. You’ll be expected to work directly with the inference stack, identify performance bottlenecks, implement improvements, and demonstrate their impact through rigorous and reproducible measurement.

What You Will Own

• Inference Serving: Own and improve the FAR Labs inference serving stack, including prefill/decode disaggregation, continuous batching, KV-cache management, cache-aware routing, speculative decoding, and other serving optimisations.

• Serving Performance: Improve throughput, latency, memory efficiency, and serving cost while maintaining clearly defined model-quality standards.

• GPU Performance Engineering: Identify and eliminate GPU performance bottlenecks across the inference stack using CUDA, Triton, profiling, kernel optimisation, memory optimisation, and appropriate precision strategies.

• Mixture-of-Experts Serving: Drive efficient serving of large MoE models, including expert parallelism, all-to-all communication, expert-load balancing, and distributed execution.

• Quantization & Model Optimisation: Implement and evaluate quantization and other model optimisation techniques while measuring accuracy against appropriate full-precision references.

• Benchmarking & Measurement: Own rigorous, reproducible benchmarking across serving performance, quality, and cost. Establish measurement methodologies that can withstand external technical scrutiny.

• Performance Instrumentation: Build and improve the instrumentation required to understand system behaviour, identify bottlenecks, compare configurations, and validate improvements.

• Distributed Serving: Make technical decisions around routing, scheduling, caching, and workload distribution across diverse hardware environments.

About You

• Strong hands-on experience building, optimising, or operating LLM inference-serving systems in production.

• Deep understanding of inference stacks such as vLLM, SGLang, TensorRT-LLM, or similar technologies.

• Strong understanding of paged attention, continuous batching, disaggregated serving, KV-cache management, and modern inference architectures.

• Strong GPU performance engineering experience using CUDA or Triton.

• Strong understanding of GPU performance characteristics including memory bandwidth, compute utilisation, model bandwidth utilisation, FLOPs utilisation, and modern GPU precisions such as FP8 and FP4.

• Experience serving large Mixture-of-Experts (MoE) models, including expert parallelism, all-to-all communication, and expert-load balancing.

• Hands-on experience with model quantization and measuring quality against full-precision references.

• Strong benchmarking experience, including load generation, latency percentiles, throughput-at-SLO, quality measurement, and reproducibility.

• Strong understanding of distributed systems, particularly routing and scheduling workloads across varied hardware.

• Ability to analyse complex performance bottlenecks across models, serving infrastructure, networking, and hardware.

• Demonstrated technical leadership experience leading an inference, serving, ML infrastructure, or similarly complex engineering initiative.

• Ability to set technical direction while remaining hands-on with implementation and performance optimisation.

• Experience with speculative decoding and understanding its behaviour under production load is a strong plus.

• Open-source contributions to inference or serving frameworks are a strong plus.

• Experience serving reasoning models, long-context models, or agentic multi-turn workloads is a plus.

• Experience with energy-aware serving, including throughput-per-watt or power-constrained environments, is a plus.

• Experience with confidential computing or data-residency-aware serving is a plus.

• Production experience with model compression techniques such as structured sparsity, activation sparsity, or KV compression is a plus.

• English level: Advanced (C1).

What We Offer

• Real ownership of the backend that powers Aleroy inference platform

• Influence over backend architecture and product direction

• Hands-on work on hard problems in distributed systems, real-time streaming, and LLM serving

• A small, senior team with fast execution and low bureaucracy

• Competitive salary with performance-based incentives

• 24 days annual leave plus public holidays

• Health insurance

• Modern office in Yas Creative Hub