Senior AI Agent Data Quality Engineer

Google DeepMind; Mountain View (CA); On-site; USD 207,000 - 300,000; Full time

Job summary

Google DeepMind in Mountain View is seeking a Software Engineer with a strong software foundation to work on cutting-edge AI agents, testing systems for agent reasoning in games, and visualization of results.

You will collaborate with ML and neuroscience researchers to build scalable pipelines, dashboards, and benchmarks that advance general-purpose AI while ensuring safety and ethics are prioritized.

Qualifications

• Bachelor's degree or equivalent practical experience in CS or engineering.
• Experience with LLMs, NLP, or Generative AI.
• MBA or Master's degree in CS preferred.

Responsibilities

• Construct rigorous benchmarks and automated evaluation frameworks for agent reasoning, planning, and tool use.
• Develop end-to-end data flywheel on agent evalset curation, and automate and productionize the flywheel with agentic data pipelines.
• Develop a dedicated platform for 'Self-Service Judges,' allowing researchers to prototype and test new LLM-based classifiers and annotations against historical trajectory data.
• Establish a formal ecosystem for metric lifecycle management, including a sandbox for validating new metrics against a library of high-confidence calibration experiments.
• Productionize automated insights tools to identify the top movers of metric deltas across complex slices such as task type and user cohort.

Skills

LLMs / NLP / Generative AI experience

Education

Bachelor's degree in Computer Science or Engineering, or equivalent practical experience.

MBA or Master's degree in Computer Science.

Job description

Google DeepMind in Mountain View is seeking a Software Engineer with a strong software foundation to work on cutting-edge AI agents, testing systems for agent reasoning in games, and visualization of results.

You will collaborate with ML and neuroscience researchers to build scalable pipelines, dashboards, and benchmarks that advance general-purpose AI while ensuring safety and ethics are prioritized.