LLM Benchmark Architect — Research Lead

Vibehackers; San Francisco, Northern (CA, KY); Hybrid; USD 140,000 - 185,000; Full time

Benefits offered by this job

Relocation assistance

Housing stipend (within 1 mile)

Health and dental insurance

Meals provided (lunch & dinner)

Unlimited PTO

Job summary

Vibehackers in San Francisco is seeking a Senior Staff Researcher to design and implement novel benchmarks that evaluate real-world LLM capabilities. You will lead research and engineering efforts, publish results, and collaborate with model labs and enterprise partners to shape how foundation models are measured.

The role emphasizes rigorous experimental design, statistical analysis, and scalable benchmark deployment, with relocation support, a competitive salary, health and dental insurance,

Qualifications

• Advanced research experience (Master’s or PhD in Computer Science, NLP, Machine Learning, or related field)
• Publication track record in NeurIPS, ICML, ACL, EMNLP
• Strong understanding of experimental design, statistical analysis, and evaluation frameworks
• Proficiency in Python for research and experimentation
• Strong communication skills for technical and non-technical audiences
• Experience collaborating in research teams and integrating feedback
• Demonstrated track record of impactful research work / portfolio

Responsibilities

• Design and implement novel benchmarks that evaluate real-world LLM capabilities
• Conduct research to ensure benchmarks are valid, reliable, and meaningful
• Analyze model performance across benchmarks and articulate results
• Collaborate with foundation model labs, enterprises, and infrastructure teams to implement benchmarks at scale
• Publish research findings and contribute to the evaluation research community
• Stay current with developments in LLM capabilities and evaluation methodologies

Skills

Python programming

Strong communication

Research collaboration

Experimental design

Statistical analysis

Data analysis

Project ownership

Rapid learning

Problem solving

Education

Master’s or PhD in CS/NLP/ML

Tools

Django

React

AWS CDK

AWS

Job description

Vibehackers in San Francisco is seeking a Senior Staff Researcher to design and implement novel benchmarks that evaluate real-world LLM capabilities. You will lead research and engineering efforts, publish results, and collaborate with model labs and enterprise partners to shape how foundation models are measured.

The role emphasizes rigorous experimental design, statistical analysis, and scalable benchmark deployment, with relocation support, a competitive salary, health and dental insurance,