Staff AI Engineer, Model Quality & Evaluation

Google; Ionia (NY); On-site; USD 207,000 - 300,000; Full time

Job summary

Google is seeking a Staff Software Engineer for Applied AI in New York, NY to own evaluation and quality for model outputs across Workspace tools. You will shape evaluation frameworks and collaborate across product and research teams to advance AI capabilities at scale.

The role focuses on building robust evaluation infrastructure, improving grounding, safety, and usefulness, while delivering scalable tooling for rapid iteration.

Qualifications

• Bachelor's degree or equivalent practical experience.
• 8 years of experience in software development.
• 5 years experience with speech/audio, reinforcement learning, ML infrastructure, or related ML field.
• 5 years experience with ML design and ML infrastructure (model deployment, evaluation, data processing, debugging, fine tuning).
• 5 years experience testing and launching software products, and 3 years experience with design/architecture.
• Experience integrating generative AI tools or LLM interfaces into workflows.

Responsibilities

• Build human-powered and LLM-powered automated evaluation systems to assess model performance.
• Establish clear metrics for grounding, coherence, safety, and helpfulness.
• Utilize platforms and tools to run evaluations across models and datasets.
• Provide actionable insights from evaluations to improve model quality with cross-functional teams.
• Create tools and systems to make the evaluation process more efficient and effective.

Skills

Speech/Audio

Reinforcement learning

ML infrastructure

Model deployment

LLM integration

Education

Bachelor's degree or equivalent

Master's or PhD in Engineering/CS or related

Tools

GemPix

Gemini

LLM interfaces

Job description

Google is seeking a Staff Software Engineer for Applied AI in New York, NY to own evaluation and quality for model outputs across Workspace tools. You will shape evaluation frameworks and collaborate across product and research teams to advance AI capabilities at scale.

The role focuses on building robust evaluation infrastructure, improving grounding, safety, and usefulness, while delivering scalable tooling for rapid iteration.