AI Engineer - Data and Evals

Glean Technologies; San Francisco (CA); Hybrid; USD 200,000 - 280,000; Full time

Benefits offered by this job

Medical, Vision, and Dental coverage

401(k) contribution

Home office stipend

Annual education and wellness stipends

Healthy lunches daily

Equity may be offered

Job summary

Glean Technologies is seeking an AI Engineer - Data and Evals to design evaluation systems that link offline benchmarks to production behavior and translate results into engineering improvements.

This is a hybrid role in the San Francisco Bay Area, with 4 days per week in Mountain View or San Francisco, and a base salary range of USD 200,000 - 280,000 per year.

Qualifications

• 5+ years of software engineering experience with Go and/or Python.
• Experience creating datasets and benchmarks for LLM use cases.
• Familiarity with eval methodologies, calibration, and judge systems.
• Experience building scalable eval/experiment infra with CI/CD and observability.

Responsibilities

• Build evalsets and benchmarks from expert and production data.
• Create and scale an eval platform with agent observability and dashboards.
• Develop agentic verifiers for correctness and groundedness.
• Run evaluations across CI, pre-release, and production traffic.
• Build enterprise simulation environments to test multi-step workflows.
• Diagnose failures by slice and trace, balancing quality, latency and cost.

Skills

Go

Python

TypeScript

Tools

SQL

OpenTelemetry

CI/CD

APIs

Job description

Glean Technologies builds AI products and agents, and this role focuses on making them more reliable and useful through disciplined measurement. As an AI Engineer - Data and Evals, you will design and run evaluation systems that connect offline benchmarks to production behavior, then translate results into engineering improvements.

This is a hybrid position in the San Francisco Bay Area, CA, with 4 days per week in the Mountain View or San Francisco office. The base salary range is USD 200,000 - 280,000 per year, with compensation determined by factors such as location, level, and relevant experience.

What you’ll do

• Build evalsets and benchmarks from expert-curated examples, production data, synthetic data, and past failures, ensuring coverage of edge cases and key user slices.
• Create and scale an eval platform and agent observability, including experiment pipelines, trace capture and replay, and dashboards that support reproducible, affordable runs.
• Develop agentic LLM verifiers for correctness, groundedness, citation quality, and task success, then calibrate them against human labels using clear rubrics.
• Run evaluations across the lifecycle, from CI and pre-release suites to sampled production traffic, and communicate outcomes with confidence intervals.
• Build enterprise simulation environments using simulated users and realistic company data to test multi-step agent workflows.
• Diagnose failures by slice and trace, then convert eval results into concrete fixes while balancing quality against latency and cost.

What you bring

• 5+ years of software engineering experience with strong Go and/or Python and backend fundamentals, including APIs, data modeling, testing, and distributed pipelines. TypeScript is a plus.
• Experience creating datasets and benchmarks for LLM use cases such as RAG or agents, including evalset curation and synthetic data generation.
• Familiarity with eval methodologies and tradeoffs, including deterministic metrics, LLM-as-a-judge, pointwise vs pairwise grading, human labeling, and judge calibration.
• Experience building eval or experiment infrastructure that runs reliably at scale, including CI/CD integrations and observability tooling such as OpenTelemetry.
• Strong analytical skills: ability to work through traces and failure cases, write SQL, and use data and metrics to explain quality movement.
• Nice to have: experience with simulation environments and/or model post-training techniques.

Tools you may use

• Go, Python, TypeScript, SQL
• OpenTelemetry, CI/CD

Compensation and benefits

• Base salary range: $200,000 - $280,000 annually
• Medical, Vision, and Dental coverage
• Generous time-off policy
• 401(k) contribution opportunity
• Home office improvement stipend
• Annual education and wellness stipends
• Healthy lunches daily
• Certain roles may be eligible for variable compensation, equity, and benefits

AI-first mindset at Glean

During the interview process, you will complete a brief AI-focused exercise or discussion to help Glean understand how you think about, design, and apply AI to drive impact in your role.