Member of Technical Staff

Raydar; United States; Hybrid; USD 200,000 - 300,000; Full time

Benefits offered by this job

Medical, dental and vision

401(k) with up to 3% match

Unlimited PTO

Job summary

Raydar is recruiting for an AI evaluation engineer to own how the company measures its AI engineer, building the evaluation framework that demonstrates growth from junior to senior levels. You will design tasks, recreate real-world failure modes, and align automated judges with expert judgment.

You will collaborate with engineering SMEs, customers and internal teams to identify strengths and weaknesses, turning results into actionable product improvements.

Qualifications

• 5 to 15 years across physical engineering and AI work; deep strength in one with working knowledge of the other.
• Extensive hands‑on AI evaluation experience: building eval tasks, designing judges and running human evaluation for agentic systems.
• A physical engineering background (mechanical, electrical, aerospace, robotics or power systems).
• Experience working directly with customers, SMEs, partners and cross‑functional teams.
• Based in the US or Canada, able to spend one week every six weeks in San Mateo, CA; US citizen or green card holder.

Responsibilities

• Own the evaluation framework from scratch, defining what good engineering design looks like when there is no single right answer.
• Work with engineering SMEs, customers, partners and internal teams to learn where the system succeeds or fails.
• Recreate real-world engineering tasks and failure modes in rerunnable evaluation environments.
• Design and refine automated judges and human evaluation, and check that judges match expert judgment.
• Turn eval results into clear priorities for improving engineering reasoning.

Skills

AI evaluation experience

Physical engineering

Customer interaction

Cross‑functional teamwork

Evaluation framework design

Job description

Our client is building engineering AGI for the physical world. Its AI engineer is already in customers' hands and works at the level of a junior mechanical and electrical engineer. The company recently closed a large Series A led by a top-tier venture firm, with angels from leading AI labs.

The role

Raydar is recruiting for this role on behalf of our client. You will own how the company measures its AI engineer: building the evaluation framework that shows how it compares to human engineers and what it takes to grow it from junior to senior engineer and beyond.

What you'll do

• Own the evaluation framework from scratch, defining what good engineering design looks like when there is no single right answer
• Work with engineering SMEs, customers, partners and internal teams to learn where the system succeeds or fails
• Recreate real-world engineering tasks and failure modes in rerunnable evaluation environments
• Design and refine automated judges and human evaluation, and check that judges match expert judgment
• Turn eval results into clear priorities for improving engineering reasoning

What we're looking for

• 5 to 15 years across physical engineering and AI work; deep strength in one with working knowledge of the other is fine
• Extensive hands‑on AI evaluation experience: building eval tasks, designing judges and running human evaluation for agentic systems
• A physical engineering background (mechanical, electrical, aerospace, robotics or power systems)
• Experience working directly with customers, SMEs, partners and cross‑functional teams
• Based in the US or Canada, able to spend one week every six weeks in San Mateo, CA
• US citizen or green card holder (no visa sponsorship)

Tech stack

agentic AI systems, AI evaluation frameworks, LLM-as-a-judge, human-in-the-loop evaluation, SFT, RLVR

• Experience at physical AI or robotics companies
• Zero-to-one work in early‑stage startups

Compensation and benefits

• $200K-$300K base
• Meaningful equity (4-year vest, 1-year cliff)
• Medical, dental and vision, 401(k) with up to 3% match, unlimited PTO

Location and work model

• Remote, United States or Canada
• On-site one week every six weeks in San Mateo, CA