Staff Software Engineer, AI Evaluation & Quality (Equity)

Google; United States; On-site; USD 207,000 - 300,000; Full time

Job summary

Google Staff Software Engineer in the Search Evaluation and Quality Team is based in New York or Mountain View, focusing on Evaluation-Driven Development for generative AI features across Search Verticals. You will drive objective, metrics-based quality from day one of product definition.

Role emphasizes maximizing evaluation model quality, building scalable rating infrastructure, and aligning metrics with user growth while coordinating with multiple engineering teams to deliver high-impact

Qualifications

• Bachelor’s degree or equivalent practical experience.
• 8 years of experience in software development.
• Experience integrating generative AI tools or LLM interfaces into workflows.
• Experience with machine learning model training or fine-tuning (e.g., LLMs, generative AI).
• Experience developing or executing evaluation frameworks or metrics for machine learning models.

Responsibilities

• Maximize fine-tuned evaluation model quality and establish precise AutoRater alignment across teams.
• Lead the core vertical AutoRater hillclimb and manage data contributions while scaling evaluation capabilities.
• Align evaluation metrics with long-term user growth metrics and direct user feedback signals.
• Develop scalable automated rating infrastructure and high-quality evaluation rubrics.
• Shift evaluations left to minimize subjective quality assessments and implement metrics-based expertise.

Skills

Software development

LLM integration

Model training

Evaluation metrics

Education

Bachelor's degree or equivalent

Job description

Google Staff Software Engineer in the Search Evaluation and Quality Team is based in New York or Mountain View, focusing on Evaluation-Driven Development for generative AI features across Search Verticals. You will drive objective, metrics-based quality from day one of product definition.

Role emphasizes maximizing evaluation model quality, building scalable rating infrastructure, and aligning metrics with user growth while coordinating with multiple engineering teams to deliver high-impact