Senior Distributed Systems Engineer, GenAI Eval Platform

Socket.dev; San Diego (CA); On-site; USD 207,000 - 300,000; Full time

Job summary

Google DeepMind is seeking a Software Engineer to build scalable evaluation platforms for frontier AI models. You will design and optimize distributed engines, create abstractions for evaluating LLM loops and tool use, and implement robust error handling and observability to ensure high reliability.

You will collaborate with research scientists and data teams, mentor engineers, and uphold strict code quality standards while tackling diverse problems across AI safety and performance domains.

Qualifications

• Bachelor's degree in Computer Science, Electrical Engineering, a related technical field, or equivalent practical experience.
• 8 years of experience with software development or 5 years of experience with an advanced degree.
• 5 years of experience with data structures and algorithms.
• Experience with distributed systems, software architecture, and system design.

Responsibilities

• Design and optimize distributed evaluation execution engines capable of orchestrating large volumes of inference steps across TPU and GCU pools with high throughput and low latency.
• Build foundational abstractions to evaluate complex Large Language Model (LLM) agent loops, tool use, and automated LLM-as-a-judge rating systems.
• Design robust error classification, automated retry policies, and observability dashboards to maintain strict Service Level Objectives (SLOs) for evaluation pipeline success rates.
• Partner closely with GDM research scientists and Data Science teams to anticipate frontier model evaluation requirements and translate them into elegant infrastructure solutions.
• Mentor fellow engineers, set high standards for code quality (Python in Google3), and advocate testing and system design practices.

Skills

Data structures & algorithms

Distributed systems

Software architecture

System design

Education

Bachelor's degree in Computer Science / Electrical Engineering or related field

MBA or Master's degree in CS/Software Engineering or related field

Job description

Google DeepMind is seeking a Software Engineer to build scalable evaluation platforms for frontier AI models. You will design and optimize distributed engines, create abstractions for evaluating LLM loops and tool use, and implement robust error handling and observability to ensure high reliability.

You will collaborate with research scientists and data teams, mentor engineers, and uphold strict code quality standards while tackling diverse problems across AI safety and performance domains.