AI Benchmark Architect: Datasets & Evaluation

Pathway; Paris; On-site; EUR 90,000 - 130,000; Full time

Job summary

Pathway is seeking a research-focused professional to design and execute rigorous benchmarks for post-transformer models. You will collaborate with our R&D team to build evaluation infrastructure that guides model development and benchmarking across enterprise use cases.

You will curate datasets, evaluate methodologies, and produce benchmark specs that translate cutting-edge research into reproducible, business-friendly results. Fluent English is required.

Qualifications

• Experience in ML/LLM evaluation, data science, or technical product roles.
• Ability to read papers, leaderboards, and Github repos and translate into benchmark specs.
• Strong communication with engineers and customers; translate technical detail to business value.
• Commitment to high-quality data, reproducible experiments, and clear documentation.
• Fluent in English.

Responsibilities

• Identify and curate benchmarks across use cases and markets.
• Evaluate benchmarks for clarity, data quality, evaluation methodology, and fit with model roadmap.
• Run benchmarks with baseline models to validate setup and uncover edge cases.
• Hand off benchmark-ready packages to R&D (specs, data, evaluation scripts, metrics, constraints).
• Maintain a shared vocabulary and documentation around benchmarks, datasets, and evaluation formats for GTM and R&D use.
• Track benchmark results, model leaderboards, and what good looks like for different customers.
• Contribute to demos and public-facing proof points based on benchmark outcomes.

Skills

Benchmarking

ML/LLM evaluation

Data science

Technical product sense

English proficiency

Job description

Pathway is seeking a research-focused professional to design and execute rigorous benchmarks for post-transformer models. You will collaborate with our R&D team to build evaluation infrastructure that guides model development and benchmarking across enterprise use cases.

You will curate datasets, evaluate methodologies, and produce benchmark specs that translate cutting-edge research into reproducible, business-friendly results. Fluent English is required.