Research Scientist

Wintermeyer Ventures; San Francisco (CA); On-site; USD 150,000 - 200,000; Full time

Benefits offered by this job

Visa sponsorship

Job summary

Wintermeyer Ventures is seeking a research-focused AI scientist to evaluate new models and build robust benchmarks for enterprise use. You will lead dataset development, coordinate labelers, and advance scalable evaluation methods for generated text while partnering with engineers to deploy in production.

The role emphasizes publications where applicable, collaboration with AI labs, and close work with customers to shape research priorities. On-site in San Francisco, visa sponsorship available.

Qualifications

• Experience in benchmarking, evaluation methods, NLP or language models.
• Strong publication record and practical systems work valued.
• Hands-on Python and clean software development abilities.

Responsibilities

• Evaluate new AI models across use cases and report capabilities.
• Build benchmarks from dataset development to labeling coordination.
• Advance automated evaluation methods for scalable accuracy.
• Partner with engineers to scale evaluation in production.
• Collaborate with AI labs and enterprise customers to shape research priorities.
• Share findings through publications when appropriate.

Skills

Python

NLP

Benchmarking

PyTorch

TensorFlow

Education

Master's degree in CS/ML

PhD in CS/ML (preferred)

Bachelor's with 1–3 years applied experience

Tools

Django/Flask

Git workflows

CI/CD

Job description

About the Role

Join a research and engineering team developing benchmarks and evaluation methods for advanced language models. Your work will help AI labs and enterprise teams assess model capabilities and make informed decisions before deploying models in production.

What You'll Do

• Evaluate newly released AI models across a range of use cases and report on their capabilities.
• Build benchmarks from the ground up, including dataset development and coordination with labelers.
• Advance automated evaluation methods for generated text to improve scalability and accuracy.
• Partner with engineers to implement and scale evaluation approaches in production systems.
• Work with AI labs and enterprise customers to understand evaluation needs and shape research priorities.
• Share research findings, including through publications where appropriate.

What We're Looking For

• At least 3 years of full-time AI/ML research experience, particularly in benchmarking, evaluation methods, NLP, or language models.
• A record of published research, with applied research experience and practical systems work valued.
• Strong Python skills and experience building clean, maintainable software; familiarity with Django, Flask, or similar HTTP frameworks.
• Hands-on experience with PyTorch or TensorFlow, language modeling, LLM infrastructure, and relevant AI/ML techniques.
• Experience creating benchmarks or evaluation methodologies and translating research into scalable systems.
• A master's or PhD in computer science, machine learning, or a related field is preferred; candidates with a bachelor's degree and 1 to 3 years of applied experience may also be considered.
• Experience collaborating in engineering teams, including Git workflows, code review, and incorporating feedback.

Compensation & Benefits

Base salary: $150,000 to $200,000 annually. Visa sponsorship is available.

Location

On-site in San Francisco, California, United States.