Senior Software Engineer - Large Language Model Evaluation Infrastructure

Google India Pvt Ltd; Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Google is seeking a Senior Software Engineer to lead the architecture and implementation of LLM evaluation platforms and scalable benchmarking pipelines for Search, focusing on quality, safety, and latency across model architectures.

You will transform complex evaluation workflows into a unified back-end infrastructure operating at a massive global scale, driving reliability and developer velocity in a fast-moving environment.

Qualifications

• Bachelor’s degree or equivalent practical experience.
• 5 years of experience with software development in one or more programming languages.
• 3 years of experience working with embedded operating systems.
• 3 years of experience testing, maintaining, or launching software products.
• 1 year of experience with software design and architecture.
• Master's degree or PhD in Computer Science or a related technical field.
• 5 years of experience with data structures and algorithms.
• 1 year of experience in a technical leadership role.
• Experience developing accessible technologies.

Responsibilities

• Architect, build, and maintain high-throughput, low-latency platform services and orchestration pipelines that execute automated evaluation, benchmark suites, and metric computations for Generative Artificial Intelligence (GenAI) Search experiences.
• Design robust evaluation systems to detect metric drift, optimize evaluation throughput, and scale model-assisted rating (e.g., LLM-as-a-judge/Autorater) workflows while maintaining compute efficiency.
• Drive the modernization and consolidation of fragmented, bespoke evaluation pipelines into a unified core platform, identifying capability gaps and creating reusable, extensible frameworks.
• Optimize system throughput, execution turnaround times, and resource efficiency for continuous, high-volume evaluation workloads supporting production-critical Search models.
• Integrate modern AI tooling, automated validation harnesses, and AI-assisted workflows into daily engineering practices to boost developer velocity, evaluation excellence, and system robustness.

Skills

Software development

Embedded OS

Testing & maintenance

Software design & architecture

Data structures & algorithms

Technical leadership

Accessible technologies

Education

Bachelor’s degree or equivalent practical experience

Master’s degree or PhD in Computer Science or related field

Job description

Minimum qualifications

• Bachelor’s degree or equivalent practical experience.
• 5 years of experience with software development in one or more programming languages.
• 3 years of experience working with embedded operating systems.
• 3 years of experience testing, maintaining, or launching software products.
• 1 year of experience with software design and architecture.

Preferred qualifications

• Master's degree or PhD in Computer Science or a related technical field.
• 5 years of experience with data structures and algorithms.
• 1 year of experience in a technical leadership role.
• Experience developing accessible technologies.

About the job

Google's software engineers develop the next-generation technologies that change how billions of users connect, explore, and interact with information and one another. Our products need to handle information at massive scale, and extend well beyond web search. We're looking for engineers who bring fresh ideas from all areas, including information retrieval, distributed computing, large-scale system design, networking and data storage, security, artificial intelligence, natural language processing, UI design and mobile; the list goes on and is growing every day. As a software engineer, you will work on a specific project critical to Google’s needs with opportunities to switch teams and projects as you and our fast-paced business grow and evolve. We need our engineers to be versatile, display leadership qualities and be enthusiastic to take on new problems across the full-stack as we continue to push technology forward. At Google, our mission is to organize the world's information and make it universally accessible and useful.

The Search Platforms Large Language Model (LLM) Evaluation Infrastructure team is at the forefront of this mission, building the next-generation core infrastructure and benchmarking platforms that power robust, automated, and scalable LLM evaluations across Google Search. As a Senior Software Engineer, you will lead the architecture, design, and implementation of LLM evaluation platforms and continuous benchmarking pipelines. You will transform complex, bespoke evaluation workflows into a unified, high-throughput platform capable of assessing quality, factual accuracy, safety, and latency across advanced model architectures. You will be responsible for creating and evolving scalable, reproducible back-end evaluation infrastructure for systems operating at a massive global scale. In Google Search, we're reimagining what it means to search for information – any way and anywhere. To do that, we need to solve complex engineering challenges and expand our infrastructure, while maintaining a universally accessible and useful experience that people around the world rely on. In joining the Search team, you'll have an opportunity to make an impact on billions of people globally.

Responsibilities

• Architect, build, and maintain high-throughput, low-latency platform services and orchestration pipelines that execute automated evaluation, benchmark suites, and metric computations for Generative Artificial Intelligence (GenAI) Search experiences.
• Design robust evaluation systems to detect metric drift, optimize evaluation throughput, and scale model-assisted rating (e.g., LLM-as-a-judge/Autorater) workflows while maintaining compute efficiency.
• Drive the modernization and consolidation of fragmented, bespoke evaluation pipelines into a unified core platform, identifying capability gaps and creating reusable, extensible frameworks.
• Optimize system throughput, execution turnaround times, and resource efficiency for continuous, high-volume evaluation workloads supporting production-critical Search models.
• Integrate modern AI tooling, automated validation harnesses, and AI-assisted workflows into daily engineering practices to boost developer velocity, evaluation excellence, and system robustness.

Experience Level

Senior Level