AI Evaluation Engineer — Shape Real-World Agent Performance

Harrison Clarke; San Francisco (CA); Sur place; USD 150 000 - 230 000; Plein temps

Avantages offerts par ce poste

Meaningful equity

Competitive salary

Résumé du poste

Harrison Clarke is seeking a Member of Technical Staff - AI Evaluation Engineer to study how engineers and customers use AI agents, replicate real-world tasks, and develop robust evaluation methodologies. You will work with engineers, product teams, SMEs, and external partners to ensure evaluations reflect real workflows and guide improvements.

The role emphasizes turning observations into concrete evaluation work and ensuring automated judges align with expert human judgment, with a focus on

Qualifications

• Technical background in a physical engineering domain (e.g., power systems, power electronics, mechanical, aerospace, robotics) is required.
• Strong understanding of agentic AI systems to design and reason about evaluations.
• Experience with AI evaluations, including creating tasks, designing judges, and comparing automated evaluation to human judgment.
• Comfort working directly with customers, engineers, SMEs, and internal technical teams to extract what matters from workflows.
• Strong product judgment to translate observations into concrete evaluation work.

Responsabilités

• Study how engineers and customers use the agent in the field and identify successes/failures from an engineering perspective.
• Recreate real-world engineering tasks and failure modes in reusable environments for development teams.
• Build and refine evaluation methodologies, including automated judges and human evaluation processes.
• Assess alignment between automated judges and expert human judgment and improve them.
• Collaborate across engineering, product, customers, SMEs, and external partners to reflect real workflows in evaluations.
• Use evaluation results to show where the agent needs to improve and what problems matter most to users.

Connaissances

Engineering background

Agentic AI understanding

AI evaluation experience

Customer-facing

Description du poste

Harrison Clarke is seeking a Member of Technical Staff - AI Evaluation Engineer to study how engineers and customers use AI agents, replicate real-world tasks, and develop robust evaluation methodologies. You will work with engineers, product teams, SMEs, and external partners to ensure evaluations reflect real workflows and guide improvements.

The role emphasizes turning observations into concrete evaluation work and ensuring automated judges align with expert human judgment, with a focus on