Remote AI Evaluation Architect for Physical Engineering

Raydar; United States; Hybride; USD 200 000 - 300 000; Plein temps

Avantages offerts par ce poste

Medical, dental and vision

401(k) with up to 3% match

Unlimited PTO

Résumé du poste

Raydar is recruiting for an AI evaluation engineer to own how the company measures its AI engineer, building the evaluation framework that demonstrates growth from junior to senior levels. You will design tasks, recreate real-world failure modes, and align automated judges with expert judgment.

You will collaborate with engineering SMEs, customers and internal teams to identify strengths and weaknesses, turning results into actionable product improvements.

Qualifications

• 5 to 15 years across physical engineering and AI work; deep strength in one with working knowledge of the other.
• Extensive hands‑on AI evaluation experience: building eval tasks, designing judges and running human evaluation for agentic systems.
• A physical engineering background (mechanical, electrical, aerospace, robotics or power systems).
• Experience working directly with customers, SMEs, partners and cross‑functional teams.
• Based in the US or Canada, able to spend one week every six weeks in San Mateo, CA; US citizen or green card holder.

Responsabilités

• Own the evaluation framework from scratch, defining what good engineering design looks like when there is no single right answer.
• Work with engineering SMEs, customers, partners and internal teams to learn where the system succeeds or fails.
• Recreate real-world engineering tasks and failure modes in rerunnable evaluation environments.
• Design and refine automated judges and human evaluation, and check that judges match expert judgment.
• Turn eval results into clear priorities for improving engineering reasoning.

Connaissances

AI evaluation experience

Physical engineering

Customer interaction

Cross‑functional teamwork

Evaluation framework design

Description du poste

Raydar is recruiting for an AI evaluation engineer to own how the company measures its AI engineer, building the evaluation framework that demonstrates growth from junior to senior levels. You will design tasks, recreate real-world failure modes, and align automated judges with expert judgment.

You will collaborate with engineering SMEs, customers and internal teams to identify strengths and weaknesses, turning results into actionable product improvements.