AI Evaluation Engineer

raydar; United States; Remote; USD 200,000 - 300,000; Full time

Benefits offered by this job

Equity

Remote-friendly

Job summary

Raydar is recruiting for a role to lead the creation of a measurement approach for an AI system that carries out technical design work. You will draw on engineering judgment and evaluation know-how, collaborating with domain experts and other teams to turn findings into clear improvement priorities.

You will establish a new assessment framework, reproduce realistic design tasks, create and tune automated graders, and translate results into a ranked list of system improvements.

Qualifications

• About five or more years of professional experience in hardware engineering, AI, or a blend of both, with depth in one and practical familiarity with the other.
• Hands-on background assessing AI systems, including authoring test tasks and benchmarking with human review.
• Solid grasp of autonomous, agent-style AI systems and hardware design principles.
• Comfort working with external users and cross-functional partners.
• Willingness to travel to an office for one week every 6 weeks (US or Canada).

Responsibilities

• Establish a new assessment framework that defines quality for design work where several answers can be valid.
• Collaborate with domain experts, external users and internal teams to observe how seasoned engineers operate and where the system struggles.
• Reproduce realistic design tasks and typical failure cases in repeatable test setups for developers to run again.
• Create and tune automated graders alongside human review processes and ensure graders align with expert opinion.
• Convert assessment results into a ranked list of improvements to the system’s reasoning.

Skills

Hardware Eng

AI

Test Task Authoring

Automated Graders

Stakeholder Leadership

Job description

About the company

Our client is a software company.

The role

Raydar is recruiting for this role on behalf of our client. Lead the creation of a measurement approach for an AI system that carries out technical design work. Draw on both hands-on engineering judgment and evaluation know-how, and collaborate with domain experts and other teams to convert findings into clear improvement priorities.

What you’ll do

• Establish a new assessment framework that defines quality for design work where several answers can be valid.
• Collaborate with domain experts, external users and internal teams to see how seasoned engineers operate and where the system struggles.
• Reproduce realistic design tasks and typical failure cases in repeatable test setups that developers can run again.
• Create and tune automated graders alongside human review processes, and confirm that the graders agree with expert opinion.
• Convert assessment results into a ranked list of improvements to the system’s reasoning.

Requirements

What we’re looking for

• About five or more years of professional experience in hardware engineering, AI, or a blend of both, with depth in one and practical familiarity with the other.
• Substantial hands-on background assessing AI systems, including authoring test tasks, building automated graders and benchmarking them against human review.
• Solid grasp of how autonomous, agent-style AI systems behave.
• Grounding in a hardware engineering field, enough to tell sound design from weak design.
• Comfort working directly with external users, specialists, partners and technical colleagues across functions.
• Self-assurance to guide work among many stakeholders in a role with high ownership.
• Based in the US or Canada, and able to travel to an office for one week every 6 weeks.

Bonus points

• Prior time at a robotics or physical AI company.
• Experience building things from zero in an early-stage startup.

Benefits

Compensation and benefits

• Base salary: USD 200,000 to 300,000 per year
• Equity

Location and work model

• Remote (US or Canada)
• Full-time