Senior AI Evaluation & Benchmark Engineer (Hybrid)

Glean; San Francisco (CA); Hybrid; USD 200,000 - 280,000; Full time

Benefits offered by this job

Medical, Vision, Dental

Generous time-off

401k plan

Home office stipend

Education stipend

Wellness stipend

Healthy lunches

Job summary

Glean, the Work AI platform redefining enterprise search and knowledge work, seeks a seasoned software engineer to build eval and benchmarking systems for LLM-driven products. You will craft datasets, design scalable eval pipelines, and ensure high-quality, measurable results across wide data sets.

Join a hybrid team in the SF Bay Area, work on agentic AI capabilities, observability, and CI/CD integrations, and help ship reliable AI-powered features that accelerate productivity for thousands of

Qualifications

• 5+ years of software engineering experience with Go and/or Python.
• Experience building datasets and benchmarks for LLM use-cases or agents.
• Familiar with CI/CD and observability tooling.

Responsibilities

• Build eval sets and benchmarks from expert-curated examples, production data, synthetic data, and past failures, covering edge cases and key user slices.
• Build and scale the eval platform and agent observability: experiment pipelines, trace capture and replay, and dashboards, with reproducible, affordable runs.
• Build agentic LLM verifiers for correctness, groundedness, citation quality, and task success, calibrating them with human labels using clear rubrics.
• Run evals at every stage, from CI and pre-release suites to production traffic, and report results with confidence intervals.
• Build enterprise simulation environments with simulated users and realistic company data to test multi-step agent tasks.
• Diagnose failures by slice and trace, and turn eval results into concrete fixes, weighing quality against latency and cost.

Skills

Go

Python

APIs

Data modeling

Testing

Distributed pipelines

OpenTelemetry

CI/CD

SQL

Tools

OpenTelemetry

Job description

Glean, the Work AI platform redefining enterprise search and knowledge work, seeks a seasoned software engineer to build eval and benchmarking systems for LLM-driven products. You will craft datasets, design scalable eval pipelines, and ensure high-quality, measurable results across wide data sets.

Join a hybrid team in the SF Bay Area, work on agentic AI capabilities, observability, and CI/CD integrations, and help ship reliable AI-powered features that accelerate productivity for thousands of