Lead Inference Engineer

Aleroy; United Arab Emirates; Hybrid; AED 360,000 - 620,000; Full time

Benefits offered by this job

Real ownership of backend

Performance-based incentives

24 days annual leave

Health insurance

Modern office in Yas Creative Hub

Job summary

Aleroy seeks a senior AI-inference engineer to own our serving stack across inference, metrics, and testing. You will lead performance improvements for large-scale LLM workloads, optimize GPU utilization, and drive MoE serving and quantization strategies.

The role combines hands-on engineering with technical leadership in distributed AI infrastructure, benchmarking, and production-grade improvements. Based in UAE with a strong focus on performance and reliability.

Qualifications

• Hands-on experience building or operating LLM inference-serving systems in production.
• Deep understanding of inference stacks (vLLM, SGLang, TensorRT-LLM) or similar.
• Knowledge of paged attention, continuous batching, and disaggregated serving.
• Experience with CUDA or Triton for GPU performance.
• Experience serving Mixture-of-Experts (MoE) models and quantization.
• Strong benchmarking, latency/throughput measurement, and reproducibility.
• Distributed systems expertise in routing and scheduling across hardware.
• Technical leadership in inference/ML infra projects.
• Ability to set direction while remaining hands-on.
• Preferred: experience with speculative decoding and open-source contributions.

Responsibilities

• Own and improve the INF serving stack, including batching and KV-cache management.
• Improve throughput, latency, memory efficiency, and cost while preserving model quality.
• Identify GPU bottlenecks and optimize using CUDA/Triton.
• Drive efficient MoE serving with expert parallelism and load balancing.
• Implement and evaluate quantization and other optimisations with accuracy checks.
• Own reproducible benchmarking across performance, quality, and cost.
• Build instrumentation to diagnose system behavior and validate improvements.
• Make technical decisions on routing, scheduling, and workload distribution.
• Define the AI serving roadmap and prioritize infra improvements.
• Set and enforce engineering standards for performance, benchmarking, and production readiness.
• Mentor engineers while contributing to implementation and problem solving.

Skills

LLM inference-serving

Performance tuning

Distributed systems

Engineering leadership

Benchmarking

CUDA/Triton

MoE serving

Quantization

Speculative decoding

KV-cache management

Model optimisation

Performance instrumentation

Distributed routing/scheduling

English (Advanced, C1)

Tools

vLLM

SGLang

TensorRT-LLM

CUDA

Triton

Job description

About Aleroy

Aleroy is building a unified platform for high-performance AI inference, making it easier for companies to deploy and run AI models at scale.
Through a single API, Aleroy provides access to ready-to-use and custom AI models across cloud infrastructure, dedicated capacity, and customers' own hardware. Our platform optimizes performance, cost, and deployment flexibility while giving enterprises control over where and how their AI workloads run.
We work in small teams, collaborate directly, and focus on solving complex engineering challenges to build reliable, scalable AI infrastructure.

About the Role

You will work across inference serving, GPU performance, distributed systems, benchmarking, and model optimisation to make our serving infrastructure faster and more efficient while maintaining a clear quality bar.
This is the senior technical seat for AI serving at Aleroy. You will set the serving and measurement roadmap, establish engineering standards, and guide engineers working across serving, metrics, and testing.
The role is highly hands-on. You’ll be expected to work directly with the inference stack, identify performance bottlenecks, implement improvements, and demonstrate their impact through rigorous and reproducible measurement.

What You Will Own

• Inference Serving: Own and improve the FAR Labs inference serving stack, including prefill/decode disaggregation, continuous batching, KV-cache management, cache-aware routing, speculative decoding, and other serving optimisations.

• Serving Performance: Improve throughput, latency, memory efficiency, and serving cost while maintaining clearly defined model-quality standards.

• GPU Performance Engineering: Identify and eliminate GPU performance bottlenecks across the inference stack using CUDA, Triton, profiling, kernel optimisation, memory optimisation, and appropriate precision strategies.

• Mixture-of-Experts Serving: Drive efficient serving of large MoE models, including expert parallelism, all-to-all communication, expert-load balancing, and distributed execution.

• Quantization & Model Optimisation: Implement and evaluate quantization and other model optimisation techniques while measuring accuracy against appropriate full-precision references.

• Benchmarking & Measurement: Own rigorous, reproducible benchmarking across serving performance, quality, and cost. Establish measurement methodologies that can withstand external technical scrutiny.

• Performance Instrumentation: Build and improve the instrumentation required to understand system behaviour, identify bottlenecks, compare configurations, and validate improvements.

• Distributed Serving: Make technical decisions around routing, scheduling, caching, and workload distribution across diverse hardware environments.

• Technical Roadmap: Define the AI serving and measurement roadmap and determine which performance problems and infrastructure improvements should be prioritised.

• Engineering Standards: Establish clear technical standards around inference performance, benchmarking, reproducibility, testing, and production readiness.

• Technical Leadership:Guide and mentor engineers working across serving, metrics, and testing while remaining deeply involved in implementation and technical problem solving.

About You

• Strong hands-on experience building, optimising, or operating LLM inference-serving systems in production.

• Deep understanding of inference stacks such as vLLM, SGLang, TensorRT-LLM, or similar technologies.

• Strong understanding of paged attention, continuous batching, disaggregated serving, KV-cache management, and modern inference architectures.

• Strong GPU performance engineering experience using CUDA or Triton.

• Strong understanding of GPU performance characteristics including memory bandwidth, compute utilisation, model bandwidth utilisation, FLOPs utilisation, and modern GPU precisions such as FP8 and FP4.

• Experience serving large Mixture-of-Experts (MoE) models, including expert parallelism, all-to-all communication, and expert-load balancing.

• Hands-on experience with model quantization and measuring quality against full-precision references.

• Strong benchmarking experience, including load generation, latency percentiles, throughput-at-SLO, quality measurement, and reproducibility.

• Strong understanding of distributed systems, particularly routing and scheduling workloads across varied hardware.

• Ability to analyse complex performance bottlenecks across models, serving infrastructure, networking, and hardware.

• Demonstrated technical leadership experience leading an inference, serving, ML infrastructure, or similarly complex engineering initiative.

• Ability to set technical direction while remaining hands-on with implementation and performance optimisation.

• Experience with speculative decoding and understanding its behaviour under production load is a strong plus.

• Open-source contributions to inference or serving frameworks are a strong plus.

• Experience serving reasoning models, long-context models, or agentic multi-turn workloads is a plus.

• Experience with energy-aware serving, including throughput-per-watt or power-constrained environments, is a plus.

• Experience with confidential computing or data-residency-aware serving is a plus.

• Production experience with model compression techniques such as structured sparsity, activation sparsity, or KV compression is a plus.

• English level: Advanced (C1).

What We Offer

• Real ownership of the backend that powers Aleroy inference platform

• Influence over backend architecture and product direction

• Hands-on work on hard problems in distributed systems, real-time streaming, and LLM serving

• A small, senior team with fast execution and low bureaucracy

• Competitive salary with performance-based incentives

• 24 days annual leave plus public holidays

• Health insurance

• Modern office in Yas Creative Hub