Staff AI Engineer — LLM & Agent Evaluation

Socket.dev; Mountain View (CA); On-site; USD 207,000 - 300,000; Full time

Job summary

Google DeepMind seeks a Software Engineer to advance general-purpose AI agents, building testing systems, visualization tools, and evaluation pipelines. You will work across 2D/3D environments, physics simulators, and robot testing while shaping scalable data platforms and metric pipelines.

The role focuses on rigorous engineering, experimentation, and collaboration with ML and neuroscience researchers to push AI capabilities and safe product innovation for billions of users.

Qualifications

• Bachelor's degree in Computer Science or Engineering or equivalent practical experience.
• Experience with Large Language Models (LLMs), NLP, or Generative AI.

Responsibilities

• Construct rigorous quantitative benchmarks and automated evaluation frameworks for agent capabilities.
• Develop data flywheel for agent evalset curation and productionize data pipelines.
• Create a platform for Self-Service Judges to prototype LLM-based classifiers and annotations.
• Establish metric lifecycle management with a sandbox for validating new metrics.
• Productionize automated insights tools to identify metric deltas across slices like task type and user cohort.

Skills

LLMs experience

Bachelor's degree requirement

Education

Bachelor's degree in CS or engineering

MBA or CS Master’s degree

Job description

Google DeepMind seeks a Software Engineer to advance general-purpose AI agents, building testing systems, visualization tools, and evaluation pipelines. You will work across 2D/3D environments, physics simulators, and robot testing while shaping scalable data platforms and metric pipelines.

The role focuses on rigorous engineering, experimentation, and collaboration with ML and neuroscience researchers to push AI capabilities and safe product innovation for billions of users.