AI Benchmark & Datasets Engineer / Researcher

Pathway; Paris; On-site; EUR 90,000 - 130,000; Full time

Job summary

Pathway is seeking a research-focused professional to design and execute rigorous benchmarks for post-transformer models. You will collaborate with our R&D team to build evaluation infrastructure that guides model development and benchmarking across enterprise use cases.

You will curate datasets, evaluate methodologies, and produce benchmark specs that translate cutting-edge research into reproducible, business-friendly results. Fluent English is required.

Qualifications

• Experience in ML/LLM evaluation, data science, or technical product roles.
• Ability to read papers, leaderboards, and Github repos and translate into benchmark specs.
• Strong communication with engineers and customers; translate technical detail to business value.
• Commitment to high-quality data, reproducible experiments, and clear documentation.
• Fluent in English.

Responsibilities

• Identify and curate benchmarks across use cases and markets.
• Evaluate benchmarks for clarity, data quality, evaluation methodology, and fit with model roadmap.
• Run benchmarks with baseline models to validate setup and uncover edge cases.
• Hand off benchmark-ready packages to R&D (specs, data, evaluation scripts, metrics, constraints).
• Maintain a shared vocabulary and documentation around benchmarks, datasets, and evaluation formats for GTM and R&D use.
• Track benchmark results, model leaderboards, and what good looks like for different customers.
• Contribute to demos and public-facing proof points based on benchmark outcomes.

Skills

Benchmarking

ML/LLM evaluation

Data science

Technical product sense

English proficiency

Job description

About Pathway

Pathway builds the first post-transformer frontier model that solves AI's fundamental memory problem. While transformers wake up in the same state every time—like Groundhog Day—our architecture enables true continuous learning, infinite context reasoning, and real-time adaptation. We're not optimizing yesterday's technology; we're building what comes after transformers.

About Pathway

Our breakthrough architecture outperforms Transformer and provides the enterprise with full visibility into how the model works. Combining the foundational model with the fastest data processing engine on the market, Pathway enables enterprises to move beyond incremental optimization and toward truly contextualized, experience-driven intelligence. We are trusted by organizations such as NATO, La Poste, and Formula 1 racing teams.

Pathway is led by co-founder & CEO Zuzanna Stamirowska, a complexity scientist who created a team consisting of AI pioneers, including CTO Jan Chorowski who was the first person to apply Attention to speech and worked with Nobel laureate Geoff Hinton at Google Brain, as well as CSO Adrian Kosowski, a leading computer scientist and quantum physicist who obtained his PhD at the age of 20.

The company is backed by leading investors and advisors, including Lukasz Kaiser, co-author of the Transformer ("the T" in ChatGPT) and a key researcher behind OpenAI's reasoning models. Pathway is headquartered in Palo Alto, California.

The Opportunity

You will design and execute rigorous benchmarks and define dataset standards. Collaborating closely with our R&D team, you will build the evaluation infrastructure that guides the evolution of Pathway's post‑transformer models.

You Will

• Proactively identify, prioritize, and curate relevant public and client-driven benchmarks across our target use cases and markets
• Evaluate candidate benchmarks for clarity, data quality, evaluation methodology, and fit with our model roadmap
• Run benchmarks with baseline models to validate setup, uncover edge cases, and de‑risk R&D runs
• Hand off "benchmark-ready" packages to R&D (specs, data, evaluation scripts, expected metrics, constraints)
• Maintain a shared vocabulary and documentation around benchmarks, datasets, and evaluation formats that GTM and R&D can both use
• Track and organize benchmark results, model leaderboards, and "what good looks like" for different customers and scenarios
• Contribute to demos and public‑facing proof points based on benchmark outcomes

Requirements

You are expected to meet at least one of the following criteria:

• You have published at least one paper at NeurIPS, ICLR, or ICML - where you were the lead author or made significant conceptual & code contributions.
• You have significantly contributed to an LLM training effort which became newsworthy (topped a Hugging Face benchmark, best in class model, etc.), preferably using multiple GPU's.
• You have spent at least 6 months working in a leading Machine Learning research center (e.g. at: Google Brain / Deepmind, Apple, Meta, Anthropic, Nvidia, MILA).
• You were an ICPC World Finalist, or an IOI, IMO, or IPhO medalist in High School.

You

• Have experience with ML/LLM evaluation, data science, or technical product roles, ideally around benchmarks or experimentation.
• Are comfortable reading papers, leaderboards, and Github repos, and turning them into clear, repeatable benchmark specs.
• Can talk comfortably with both engineers and customers, and translate between technical detail and business value.
• Care about high‑quality data, reproducible experiments, and crisp documentation.
• Are respectful of others.
• Are fluent in English.

Bonus Points

• Published or open‑sourced work on LLM evaluation, benchmarking