Member of Technical Staff

Thomas Talent Network, LLC; San Francisco (CA); Vor Ort; USD 140.000 - 210.000; Vollzeit

Zusammenfassung

Our client, an AI healthcare software startup, seeks a Member of Technical Staff (Research Scientist) to develop benchmarks and evaluation methodologies for large language models. You will shape how enterprise AI systems are tested before deployment, collaborating with engineering and top AI labs.

Qualifications include 0-3 years in AI/ML research, strong Python, and experience with PyTorch/TF. Familiarity with Django/Flask is a plus; MBA not required.

Qualifikationen

• 0-3 years of AI/ML research experience focusing on benchmarking and evaluation.
• Experience in team environments, including development sprints, Git best practices, PR reviews.
• Valued: previous AI startup or research lab experience; founder/early employee background.
• Publications in NLP or benchmarking valued but not required; applied work prioritized.
• Experience building new benchmarks and evaluation methodologies preferred.
• Academic background in CS/ML; Masters/PhD preferred; Bachelor's + 0-3 years considered.
• Background from top tech/AI programs preferred.
• NLP research experience with publications preferred.
• Strong Python programming skills for clean, maintainable code.
• Experience with PyTorch/TF and diffusion models and language modeling.
• Interest in LLM infrastructure.
• Experience with Django, Flask, or other Python-based HTTP servers is a plus.
• Strong communication skills; able to give input and accept feedback.

Aufgaben

• Evaluate new AI models as they're released.
• Create fresh benchmarks, hire labelers, construct datasets, and write white papers.
• Improve auto-evaluation methods for generated text.
• Collaborate with engineering to implement and scale evaluation methodologies.
• Work with top AI labs and enterprise customers to understand evaluation needs.

Kenntnisse

Python

PyTorch

TensorFlow

NLP research

Benchmarking

Communication

Team collaboration

Research publication

Ausbildung

Master's or PhD preferred

Bachelor's degree

Tools

Git

GitHub

Django

Flask

Jobbeschreibung

Our client, an AI healthcare software startup, is seeking a Member of Technical Staff (Research Scientist) to join their team. In this role, you'll be at the forefront of developing benchmarks and evaluation methodologies for large language models, helping shape how the most advanced AI systems are tested and validated before deployment in enterprise environments.

Key Responsibilities:

• Evaluate new AI models as they're released (e.g., DeepSeek, Gemini)
• Create new benchmarks from scratch, including hiring labelers, constructing datasets, and writing white papers
• Improve underlying methods for auto-evaluation of generated text
• Work closely with engineering to implement and scale evaluation methodologies
• Collaborate with top AI labs and enterprise customers to understand evaluation needs

Qualifications:

• 0-3 years of research experience in AI/ML, particularly in benchmarking, evaluation methodologies, or language models
• Experience working in team environments, including development sprints, Git best practices, and pull request review
• Previous experience at AI startups, research labs (e.g., Anthropic, DeepMind), or as a founder/early employee at your own company is highly valued
• Published research (especially in NLP or benchmarking) is valued but not required; applied work is prioritized over purely academic publications
• Experience building new benchmarks and evaluation methodologies preferred
• Academic background in Computer Science, Machine Learning, or a related field (Master's or PhD preferred); candidates with a Bachelor's degree and 0-3 years of relevant experience will also be considered
• Background from top tech/AI programs preferred
• NLP research experience with publications in reputable journals preferred
• Strong Python programming skills, with the ability to build clean, maintainable code
• Experience with deep learning frameworks (PyTorch, TensorFlow) and techniques such as diffusion models and language modeling
• Interest in and familiarity with LLM infrastructure
• Experience with Django, Flask, or other Python-based HTTP servers is a plus
• Strong communication skills; able to give input to others and receive/integrate feedback effectively

Job Reference: X985X343

Package Details: Salary & equity

Visa Sponsorship: Open to visa transfers (e.g. OPT, H1B transfers), NOT new sponsorship. Will support relocation or transportation as needed.