Frontier Language Model Evaluation Engineer

Artificial Analysis, Inc.; San Francisco (CA); On-site; USD 170,000 - 230,000; Full time

Benefits offered by this job

Equity

Frontier AI exposure

Job summary

Artificial Analysis is seeking a Member of Technical Staff to design frontier evaluations for language models and publish results used by AI labs and enterprises. You will build datasets, scoring systems, and evaluation infrastructure applicable across major models released by labs worldwide.

Based in San Francisco or flexible to other major cities, this role offers equity and the opportunity to shape how industry measures frontier AI capability while collaborating with leading researchers and

Qualifications

• 3+ years of relevant professional experience in industry or research.
• Strong Python skills and experience running evaluation harnesses and building datasets.
• Deep familiarity with the LLM evaluation landscape, failure modes, contamination, and agentic evaluation.
• Strong statistical grounding to distinguish signal from noise.
• Genuine interest and knowledge of Frontier AI with informed opinions on its future direction.

Responsibilities

• Design and ship next-generation frontier evaluations across models.
• Build datasets, harnesses, and scoring systems for frontier scale.
• Contribute to the AI Index and platform metrics through analyses.
• Publish reports and data visualizations shaping industry understanding of progress.
• Evaluate pre-release models and maintain result integrity across releases.
• Maintain an AI-native workflow with cutting-edge tools.

Skills

Python

Analytical thinking

Critical thinking

Frontier AI knowledge

Tools

Evaluation harnesses

Job description

Artificial Analysis is seeking a Member of Technical Staff to design frontier evaluations for language models and publish results used by AI labs and enterprises. You will build datasets, scoring systems, and evaluation infrastructure applicable across major models released by labs worldwide.

Based in San Francisco or flexible to other major cities, this role offers equity and the opportunity to shape how industry measures frontier AI capability while collaborating with leading researchers and