AI Benchmark Engineer | Remote Data & Evaluation

Pathway; Paris; Remote; EUR 90,000 - 130,000; Full time

Benefits offered by this job

Remote work

Job summary

Pathway is seeking a seasoned colleague to design and execute rigorous benchmarks for post-transformer models and to define dataset standards. You will work closely with our R&D team to build evaluation infrastructure guiding the evolution of Pathway's models.

You will curate benchmarks, assess data quality and methodology, and hand off benchmark-ready packages to R&D. You will help maintain shared benchmarks vocabulary and track model leaderboards for diverse customers.

Qualifications

• Experience in ML/LLM evaluation, data science, or technical product roles, ideally around benchmarks or experimentation.
• Ability to read papers, leaderboards, and Github repos, and translate them into clear benchmark specs.
• Strong communication with engineers and customers, translating technical detail into business value.
• Commitment to high‑quality data, reproducible experiments, and crisp documentation.
• Fluency in English.

Responsibilities

• Identify, prioritize, and curate relevant public and client-driven benchmarks across target use cases and markets.
• Evaluate benchmark clarity, data quality, evaluation methodology, and fit with model roadmap.
• Run benchmarks with baseline models to validate setup and uncover edge cases.
• Deliver benchmark-ready packages to R&D (specs, data, evaluation scripts, metrics, constraints).
• Maintain vocabulary and documentation around benchmarks, datasets, and evaluation formats used by GTM and R&D.
• Track benchmark results, model leaderboards, and what good looks like for different customers.

Skills

ML/LLM evaluation

Data science

Technical product roles

Reading papers and leaderboards

English fluent

Job description

Pathway is seeking a seasoned colleague to design and execute rigorous benchmarks for post-transformer models and to define dataset standards. You will work closely with our R&D team to build evaluation infrastructure guiding the evolution of Pathway's models.

You will curate benchmarks, assess data quality and methodology, and hand off benchmark-ready packages to R&D. You will help maintain shared benchmarks vocabulary and track model leaderboards for diverse customers.