Research Scientist - LLM Evaluation & Routing

OpenRouter; United States; On-site; USD 150,000 - 230,000; Full time

Job summary

OpenRouter seeks a Research Scientist to conduct original research advancing how the world understands, evaluates, and routes large language models. You will analyze billions of LLM generations across providers and use cases.

You will own a research agenda, design experiments and evaluation frameworks, and produce work that influences model comparison, selection, and deployment. Collaboration with product and engineering is key, emphasizing depth and rigor.

Qualifications

• Advanced degree in ML, CS, statistics, or mathematics.
• Track record of original research with publications or notable impact.
• Strong statistical, experimental design, and causal inference skills.
• Experience with Python and large-scale experiments.

Responsibilities

• Own and pursue a research agenda focused on LLM evaluation, model quality, routing optimization, and AI usage patterns.
• Design novel evaluation frameworks and benchmarks using real-world generation data.
• Conduct large-scale empirical studies on LLM behavior across providers and time.
• Develop the statistical and mathematical foundations behind routing systems and model selection.
• Identify opportunities to apply research findings to OpenRouter's product and platform.
• Collaborate with external researchers and model providers to advance LLM capabilities.
• Translate research insights into platform improvements with product/engineering teams.

Skills

Python

SQL

Statistics

Experiment design

Research publications

LLM techniques

Education

MS or PhD in a quantitative field

Tools

ClickHouse

BigQuery

Job description

OpenRouter seeks a Research Scientist to conduct original research advancing how the world understands, evaluates, and routes large language models. You will analyze billions of LLM generations across providers and use cases.

You will own a research agenda, design experiments and evaluation frameworks, and produce work that influences model comparison, selection, and deployment. Collaboration with product and engineering is key, emphasizing depth and rigor.