Remote AI Benchmarking Architect: Cost & Performance

Akka; San Francisco (CA); On-site; USD 140,000 - 210,000; Full time

Benefits offered by this job

Competitive salary

Comprehensive benefits

Flexible remote working

Professional development

Work-life balance

Job summary

Akka is seeking an Applied AI Research Engineer focused on benchmarking and performance economics. This role emphasizes producing decision-grade numbers, benchmarks, and cost models across token economics, routing, language model economics, and GPU/serving-stack modeling.

You will own the benchmark harness end-to-end, including data curation, evaluation, and drift retraining decisions. As part of the Engineering department, you will work remotely across North America or Europe, contributing

Qualifications

• Experience designing and running benchmark experiments with clear success criteria.
• Ability to translate measurements into actionable recommendations for product decisions.
• Strong skills in systems thinking and profiling for large-scale AI workloads.
• Proficiency in Python for harness development and data analysis.

Responsibilities

• Develop benchmarking frameworks with controlled cache states and reproducible runs.
• Model and analyze costs and performance across token, route, and serving pipelines.
• Validate estimates against ground truth measurements and provide defensible conclusions.
• Collaborate with engineers to deploy measurement tooling and explain results to non-technical stakeholders.
• Own the end-to-end benchmark harness as a product, with versioning and reproducibility.

Skills

Python

Experiment design

Systems engineering

Cost modeling

Tools

vLLM

TensorRT-LLM

Ray

AutoGen

Job description

Akka is seeking an Applied AI Research Engineer focused on benchmarking and performance economics. This role emphasizes producing decision-grade numbers, benchmarks, and cost models across token economics, routing, language model economics, and GPU/serving-stack modeling.

You will own the benchmark harness end-to-end, including data curation, evaluation, and drift retraining decisions. As part of the Engineering department, you will work remotely across North America or Europe, contributing