AI Evaluation Engineer — Evals & Benchmark Platform

Fdetimes; San Francisco (CA); Hybrid; USD 200,000 - 280,000; Full time

Benefits offered by this job

Home office stipend

Education stipend

Wellness stipend

Daily lunches

Job summary

Glean is seeking an experienced software engineer to join the Data & Evaluation Science team in the SF Bay Area. You will turn data and evals into production-ready tooling, build evalsets and benchmarks, and partner with R&D, product, and GTM to ship measurable improvements.

You'll own pipelines for experiment orchestration, tracing, and dashboards, and work on agent verifiers, model routing, and post-training tasks to improve product quality and customer outcomes.

Qualifications

• 5+ years of software engineering experience with Go and/or Python.
• Experience building datasets and benchmarks for LLM use-cases, RAG, or agents.
• Familiarity with eval methodologies and their tradeoffs.

Responsibilities

• Build evalsets and benchmarks from expert-curated examples and production data.
• Scale the eval platform and agent observability with pipelines and dashboards.
• Create verifiers for correctness, groundedness, citation quality, and task success.
• Run evals across CI/CD, pre-release, and production traffic with confidence intervals.
• Develop enterprise simulation environments for multi-step agent tasks.
• Diagnose failures by slice and trace and implement concrete fixes.

Skills

Go

Python

APIs

Data modeling

Testing

Distributed pipelines

TypeScript

Tools

OpenTelemetry

Job description

Glean is seeking an experienced software engineer to join the Data & Evaluation Science team in the SF Bay Area. You will turn data and evals into production-ready tooling, build evalsets and benchmarks, and partner with R&D, product, and GTM to ship measurable improvements.

You'll own pipelines for experiment orchestration, tracing, and dashboards, and work on agent verifiers, model routing, and post-training tasks to improve product quality and customer outcomes.