Product Evaluations Engineer, Learning Tech (ChatGPT)

OpenAI, Inc.; San Francisco (CA); Hybrid; USD 180,000 - 240,000; Full time

Job summary

OpenAI seeks a Product Evals Engineer in San Francisco to lead systems that evaluate response quality across learning experiences. You will define criteria for excellent learner interactions and build infrastructure to evaluate performance at scale.

You will combine human judgment, automated evaluation, experimentation, and deep analysis across retrieval, ranking, and model behavior. This role partners with model measurement, data science, engineering, and education experts to ensure accurate,

Qualifications

• Strong software engineer capable of designing reliable data pipelines and evaluation infrastructure.
• Experience with human-labeled data, relevance judgments, or model-based evaluation.
• Ability to analyze qualitative and quantitative data and influence product decisions.

Responsibilities

• Build and evolve evaluation frameworks for quality and information-grounded experiences.
• Define quality metrics for relevance, factual accuracy, sources, coverage, and safety.
• Develop scalable evaluation pipelines combining human judgment, signals, grading, and checks.
• Create datasets and benchmarks spanning learner, educator, and lifelong-learning use cases.
• Design experiments to identify quality gaps and measure progress.
• Analyze evaluation results and user behavior to uncover trends and opportunities.
• Improve reliability of automated evaluators through calibration and validation.
• Partner with engineering and research to translate findings into product improvements.
• Build tools to make evaluation results accessible and actionable.

Skills

Software engineering

Data pipelines

Evaluation infrastructure

Internal tools

Information retrieval concepts

Human-labeled data

Model-based evaluation

Qualitative analysis

Quantitative analysis

Job description

OpenAI seeks a Product Evals Engineer in San Francisco to lead systems that evaluate response quality across learning experiences. You will define criteria for excellent learner interactions and build infrastructure to evaluate performance at scale.

You will combine human judgment, automated evaluation, experimentation, and deep analysis across retrieval, ranking, and model behavior. This role partners with model measurement, data science, engineering, and education experts to ensure accurate,