Senior AI Evaluation Engineer (Hybrid: Bay Area)

Glean; Mountain View (CA); Hybrid; USD 200,000 - 280,000; Full time

Benefits offered by this job

Medical, Vision, Dental

401k plan

Home office stipend

Education stipend

Wellness stipend

Free lunches

Job summary

Glean is seeking an experienced engineer to advance its AI evals roadmap. You will build datasets, benchmarks, and evaluation infrastructure to ensure high-quality AI products across an enterprise platform.

You will collaborate with R&D, product, and GTM to translate eval findings into product improvements and measurable customer outcomes. The role is hybrid with Mountain View or SF offices and offers strong compensation and benefits.

Qualifications

• 5+ years of software engineering experience with Go and/or Python and backend fundamentals.
• Experience building datasets and benchmarks for LLM use-cases, RAG, or agents.
• Familiarity with eval methodologies and their tradeoffs (deterministic metrics, LLM-as-a-judge, human labeling).
• Experience building eval or experiment infrastructure that runs reliably at scale, including observability tooling.

Responsibilities

• Build evalsets and benchmarks from expert-curated examples, production data, synthetic data, and past failures.
• Scale the eval platform and agent observability with pipelines and dashboards.
• Build agentic LLM verifiers and calibrate them against human labels.
• Run evals at all stages and report results with confidence intervals.
• Create enterprise simulation environments to test multi-step agent tasks.
• Diagnose failures by slice and trace to drive concrete fixes.

Skills

Go

Python

Backend fundamentals

SQL

TypeScript

Tools

OpenTelemetry

CI/CD

Job description

Glean is seeking an experienced engineer to advance its AI evals roadmap. You will build datasets, benchmarks, and evaluation infrastructure to ensure high-quality AI products across an enterprise platform.

You will collaborate with R&D, product, and GTM to translate eval findings into product improvements and measurable customer outcomes. The role is hybrid with Mountain View or SF offices and offers strong compensation and benefits.