AI Evaluation Engineer — Shape Real-World Agent Performance

Harrison Clarke; San Francisco (CA); On-site; USD 150,000 - 230,000; Full time

Benefits offered by this job

Meaningful equity

Competitive salary

Job summary

Harrison Clarke is seeking a Member of Technical Staff - AI Evaluation Engineer to study how engineers and customers use AI agents, replicate real-world tasks, and develop robust evaluation methodologies. You will work with engineers, product teams, SMEs, and external partners to ensure evaluations reflect real workflows and guide improvements.

The role emphasizes turning observations into concrete evaluation work and ensuring automated judges align with expert human judgment, with a focus on

Qualifications

• Technical background in a physical engineering domain (e.g., power systems, power electronics, mechanical, aerospace, robotics) is required.
• Strong understanding of agentic AI systems to design and reason about evaluations.
• Experience with AI evaluations, including creating tasks, designing judges, and comparing automated evaluation to human judgment.
• Comfort working directly with customers, engineers, SMEs, and internal technical teams to extract what matters from workflows.
• Strong product judgment to translate observations into concrete evaluation work.

Responsibilities

• Study how engineers and customers use the agent in the field and identify successes/failures from an engineering perspective.
• Recreate real-world engineering tasks and failure modes in reusable environments for development teams.
• Build and refine evaluation methodologies, including automated judges and human evaluation processes.
• Assess alignment between automated judges and expert human judgment and improve them.
• Collaborate across engineering, product, customers, SMEs, and external partners to reflect real workflows in evaluations.
• Use evaluation results to show where the agent needs to improve and what problems matter most to users.

Skills

Engineering background

Agentic AI understanding

AI evaluation experience

Customer-facing

Job description

Harrison Clarke is seeking a Member of Technical Staff - AI Evaluation Engineer to study how engineers and customers use AI agents, replicate real-world tasks, and develop robust evaluation methodologies. You will work with engineers, product teams, SMEs, and external partners to ensure evaluations reflect real workflows and guide improvements.

The role emphasizes turning observations into concrete evaluation work and ensuring automated judges align with expert human judgment, with a focus on