Staff Software Engineer - AI Agent Evaluation & DataQuality

engineeringjobs.net, Inc.; Town of Montana (WI); On-site; USD 207,000 - 300,000; Full time

Job summary

Google DeepMind is seeking a Staff Software Engineer to advance Agent Data Quality. You will build systems for agent evaluation, manage data pipelines, and create visualizations and leaderboards to compare approaches at scale.

The role requires a strong software engineering background and experience with LLMs/NLP. Collaboration with ML researchers is essential, with a focus on robust, scalable testing of AI agents.

Qualifications

• Bachelor's degree in Computer Science or Engineering, or equivalent practical experience.
• Experience with Large Language Models (LLMs), NLP, or Generative AI.

Responsibilities

• Construct rigorous quantitative benchmarks and automated evaluation frameworks to measure agent capabilities in reasoning, planning, and tool use.
• Develop end-to-end data flywheel on agent evalset curation, and automate and productionize the flywheel with agentic data pipelines.
• Develop a dedicated platform for "Self‑Service Judges" to rapidly prototype and test new LLM‑based classifiers and annotations against historical trajectory data.
• Establish a formal ecosystem for metric lifecycle management, including a sandbox for validating new metrics against a library of high‑confidence calibration experiments.
• Productionize automated insights tools to identify the top movers of metric deltas across complex slices such as task type and user cohort.

Skills

LLMs/NLP

Software engineering

AI evaluation

Education

Bachelor's degree in Computer Science or Engineering

MBA or Master's in CS

Job description

Google DeepMind is seeking a Staff Software Engineer to advance Agent Data Quality. You will build systems for agent evaluation, manage data pipelines, and create visualizations and leaderboards to compare approaches at scale.

The role requires a strong software engineering background and experience with LLMs/NLP. Collaboration with ML researchers is essential, with a focus on robust, scalable testing of AI agents.