ML Systems Engineer, Inference Benchmarking

CommonAI CIC; Cambridge; Hybrid; GBP 50,000 - 75,000; Full time

Benefits offered by this job

Pension

Professional development

Networking opportunities

Office near Cambridge train station

Job summary

CommonAI CIC in Cambridge is seeking engineers to build reproducible LLM workloads, benchmark tools, and performance measures. You will characterise workloads, cache effects, and tool interactions, collaborating with measurement and simulator teams to ensure results reflect real-world conditions.

A strong focus on reproducibility and validating AI coding agents is essential. You will join a hybrid work environment with opportunities to influence UK AI infrastructure and policy-relevant research.

Qualifications

• Experience building or evaluating ML systems, inference services or benchmarks.
• Strong software skills for reliable benchmark tools and results analysis.
• Understanding of LLM inference and how serving choices affect performance.
• Care in designing reproducible comparisons and interpreting results.
• Experience using AI coding agents and validating code and data transformations.
• Experience with vLLM, SGLang or similar serving software would be valuable.

Responsibilities

• Build and maintain reproducible LLM serving workloads, configurations and benchmark tools.
• Characterise behaviour such as request arrivals, prompt/output lengths, caching, batching, tool interactions.
• Define performance measures alongside application-quality and service requirements.
• Collaborate with measurement engineers to reproduce workloads on physical systems.
• Work with simulator engineers to develop workload representations that preserve dependencies.
• Investigate how workload changes affect conclusions and keep benchmarks relevant.
• Document datasets, model and software versions, configurations and provenance.

Skills

Experience building ML systems

Benchmark tools development

LLM inference understanding

Reproducible comparisons

AI coding agents experience

Experience with vLLM/SGLang

Job description

About CommonAI

CommonAI CIC is a non-profit organisation built around collaborative engineering and is backed by a mix of UK Government and private funding in order to design, build and deploy innovative AI systems. Our Scaling Inference Lab (SIL) is developing simulation grounded in real measurements to help the UK invest in promising AI infrastructure. We connect proposed hardware designs to realistic workloads, helping investors and engineers understand their potential, their limitations and the next experiment worth doing.

The Opportunity

Build the workloads and benchmarks that reveal how AI infrastructure performs on useful applications. You will develop reproducible LLM serving workloads, connecting application behaviour and requirements to physical measurements and SIL's simulation platform. The initial focus includes long-context inference and agentic workloads. You will investigate how request patterns, context lengths, caching, repeated model calls and interactions with tools shape the demands placed on a system. Your benchmarks will make performance comparisons meaningful by specifying the work being done and the quality and service requirements it must meet. Working with measurement and simulation engineers and our partners, you will help preserve the workload behaviour that matters when moving between real systems and models. Your work will help engineers and investors understand which infrastructure choices suit particular applications, and how conclusions change when the workload changes.

What You Will Do

• Build and maintain reproducible LLM serving workloads, serving configurations and benchmark tools, including long-context and agentic workloads, using suitable existing software
• Characterise relevant behaviour such as request arrivals, prompt and output lengths, caching, batching, tool interactions and dependencies between requests
• Define performance measures alongside application-quality and service requirements, so comparisons reflect useful work under stated conditions
• Work with measurement engineers to reproduce workloads on physical systems, and with simulator engineers to develop workload representations that preserve the behaviour and dependencies that matter
• Investigate how changes in workloads and serving configurations affect conclusions, and keep the benchmark suite relevant as inference technology and workloads evolve
• Document datasets, model and software versions, configurations, provenance and permitted use so other engineers can reproduce and extend the work

Requirements

• Experience building or evaluating ML systems, inference services, performance benchmarks or comparable workload-driven systems
• Strong practical software skills for constructing reliable benchmark tools, analysing results and debugging interactions between applications and infrastructure
• Understanding of LLM inference and an interest in how serving choices affect performance and output quality
• Care in designing reproducible comparisons and interpreting results against stated application requirements
• Practical experience using AI coding agents, with the judgement to validate generated code, data transformations and evaluations

Experience with vLLM, SGLang or similar serving software, long-context workloads, agentic applications, trace analysis or quality evaluation would be valuable. Depth in a subset is sufficient. The initial remit is LLM serving; other forms of inference are possible later extensions.

Why join us

• Help build UK AI capability and improve how investment and compute serve the public good
• Build a network spanning UK hardware SMEs, major technology companies, venture and public investors, and leading universities
• Shape a growing team with startup pace and close academic links
• Use frontier models and coding agents with a substantial tooling budget

Benefits

• A collaborative and supportive work environment
• The opportunity to have a high impact in a growing organisation
• Competitive salary package, including pension and benefits
• Professional development opportunities
• Networking opportunities with influential figures from across the tech sector and academia
• A vibrant office environment located a few minutes' walk away from Cambridge train station

Working with us

Hybrid, with Cambridge attendance agreed individually around collaboration and the work; applicants across the UK are welcome. We are open to permanent employment, part-time arrangements and consultancy. We would like you to start as soon as practical and can be flexible for the right person. We particularly welcome experienced engineers and can also consider earlier-career candidates with strong evidence of ability. Relevant postgraduate study is useful, but qualifications are secondary to experience and the quality of your work.

CommonAI CIC is an equal opportunity employer and is committed to creating an inclusive and diverse workplace.