Member of Technical Staff, AI Evaluation

Harrison Clarke; San Francisco (CA); On-site; USD 150,000 - 230,000; Full time

Benefits offered by this job

Meaningful equity

Competitive salary

Job summary

Harrison Clarke is seeking a Member of Technical Staff - AI Evaluation Engineer to study how engineers and customers use AI agents, replicate real-world tasks, and develop robust evaluation methodologies. You will work with engineers, product teams, SMEs, and external partners to ensure evaluations reflect real workflows and guide improvements.

The role emphasizes turning observations into concrete evaluation work and ensuring automated judges align with expert human judgment, with a focus on

Qualifications

• Technical background in a physical engineering domain (e.g., power systems, power electronics, mechanical, aerospace, robotics) is required.
• Strong understanding of agentic AI systems to design and reason about evaluations.
• Experience with AI evaluations, including creating tasks, designing judges, and comparing automated evaluation to human judgment.
• Comfort working directly with customers, engineers, SMEs, and internal technical teams to extract what matters from workflows.
• Strong product judgment to translate observations into concrete evaluation work.

Responsibilities

• Study how engineers and customers use the agent in the field and identify successes/failures from an engineering perspective.
• Recreate real-world engineering tasks and failure modes in reusable environments for development teams.
• Build and refine evaluation methodologies, including automated judges and human evaluation processes.
• Assess alignment between automated judges and expert human judgment and improve them.
• Collaborate across engineering, product, customers, SMEs, and external partners to reflect real workflows in evaluations.
• Use evaluation results to show where the agent needs to improve and what problems matter most to users.

Skills

Engineering background

Agentic AI understanding

AI evaluation experience

Customer-facing

Job description

Role: Member of Technical Staff - AI Evaluation Engineer

The role:

• Study how engineers and customers use the agent in the field, identify where it succeeds or fails from an engineering perspective, and turn those observations into actionable evaluations
• Recreate real-world engineering tasks and failure modes in reusable environments for the development teams
• Build and refine evaluation methodologies, including automated judges and human evaluation processes
• Assess whether automated judges align with expert human judgment, and improve them when they don't
• Work across engineering, product, customers, SMEs, and external partners to ensure evaluations reflect real engineering workflows rather than abstract benchmarks
• Use evaluation results to show where the agent needs to improve and which problems matter most to users

What they're looking for:

• Technical background in a physical engineering domain such as power systems, power electronics, mechanical engineering, aerospace, robotics, or a closely related field
• Strong enough understanding of agentic AI systems to design and reason about evaluations
• Experience with AI evaluations, including creating evaluation tasks, designing judges, and comparing automated evaluation against human judgment
• Comfort working directly with customers, engineers, SMEs, and internal technical teams to extract what matters from ambiguous workflows
• Strong product judgment, with the ability to turn observations about how a system is used or fails into concrete evaluation work
• Motivation to make AI systems genuinely capable in the physical world

What is on offer:

• Competitive salary + meaningful equity
• The chance to define how a frontier engineering agent is evaluated and improved