Research Scientist: LLM Evaluation & Routing Innovator

OpenRouter; New York (NY); On-site; USD 170,000 - 250,000; Full time

Job summary

OpenRouter in New York seeks a Research Scientist to advance how the world understands, evaluates, and routes large language models. You will design experiments, build evaluation frameworks, and publish findings that influence rankings and routing decisions.

You will collaborate with product and engineering teams, work with external researchers, and prioritize depth and rigor over shipping production code.

Qualifications

• MS or PhD in a quantitative field (machine learning, statistics, CS, math, computational linguistics)
• Original research demonstrated by publications, open-source contributions, or equivalent impact
• Strong statistical and experimental design knowledge with attention to validity, bias, and generalizability
• Strong Python programming skills for data pipelines and large-scale experiments
• Proficiency in SQL for large-scale analytical databases (e.g., ClickHouse, BigQuery)
• Experience with modern ML/NLP techniques (LLM evaluation, fine-tuning, embeddings, RLHF)
• Familiarity with current LLM landscape and provider ecosystems

Responsibilities

• Own and pursue a research agenda on LLM evaluation, model quality, routing optimization, and AI usage patterns
• Design novel evaluation frameworks and benchmarks using real-world generation data
• Conduct large-scale empirical studies comparing models across providers and time
• Develop statistical foundations for routing systems and model/provider selection
• Translate research findings into product and platform improvements
• Collaborate with external researchers and the open-source community

Skills

Statistics

Experiment design

Research leadership

Scientific writing

Education

MS or PhD in quantitative field

Tools

Python

SQL

BigQuery

ClickHouse

Job description

OpenRouter in New York seeks a Research Scientist to advance how the world understands, evaluates, and routes large language models. You will design experiments, build evaluation frameworks, and publish findings that influence rankings and routing decisions.

You will collaborate with product and engineering teams, work with external researchers, and prioritize depth and rigor over shipping production code.