Staff ML Engineer, GenAI & Evaluation Systems

Socket.dev; New York (NY); On-site; USD 207,000 - 300,000; Full time

Job summary

Google Pics, an AI-powered visual editor integrated across Google Workspace, seeks a software engineer to drive evaluation systems for model performance. You will build human-powered and LLM-powered evaluation tools and establish metrics for grounding, coherence, safety, and helpfulness.

You will work with Gemini and GemPix within a fast-moving roadmap for a 2026 GA launch, collaborating with research and cross-functional teams to deliver scalable, reliable evaluation platforms across Workspace

Qualifications

• Bachelor’s degree or equivalent practical experience.
• 8 years of experience in software development.
• 5 years of experience with speech/audio, reinforcement learning, ML infrastructure, or related ML fields.
• 5 years of experience with ML design and ML infrastructure (model deployment, evaluation, data processing, debugging, fine tuning).
• 5 years of experience testing and launching software products; 3 years in software design and architecture.
• Experience integrating generative AI tools or LLM interfaces into workflows.

Responsibilities

• Build both human-powered and Large Language Model (LLM)-powered automated evaluation systems to assess model performance.
• Establish clear metrics to measure grounding, coherence, safety, and helpfulness.
• Utilize platforms and tools to efficiently run evaluations across different models and datasets.
• Provide actionable insights from evaluations to improve model quality, often in collaboration with research, and cross-functional teams.
• Create tools and systems that make the evaluation process more efficient and effective.

Skills

Software development

ML infrastructure

LLM integration

Leadership

System design

Education

Bachelor’s degree or equivalent practical experience

Master’s degree or PhD in Engineering, Computer Science, or a related technical field

Job description

Google Pics, an AI-powered visual editor integrated across Google Workspace, seeks a software engineer to drive evaluation systems for model performance. You will build human-powered and LLM-powered evaluation tools and establish metrics for grounding, coherence, safety, and helpfulness.

You will work with Gemini and GemPix within a fast-moving roadmap for a 2026 GA launch, collaborating with research and cross-functional teams to deliver scalable, reliable evaluation platforms across Workspace