Remote AI Evaluation Engineer for Engineering Systems

P-1 AI; United States; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Equity

Healthcare

Dental

Vision

401(k) match

Unlimited PTO

Job summary

P-1 AI is building an AI engineer agent for the physical world named Archie. In this role you’ll work with engineers, customers, partners, and SMEs to challenge Archie on tough engineering work, understand where it fails, and capture those failures in evaluations that guide product development.

You will design evaluation tasks, recreate real-world failure modes, build automated judges, and assess alignment with expert judgment, ensuring our evaluations reflect real engineering workflows rather

Qualifications

• Technical background in a physical engineering domain such as power systems, power electronics, mechanical engineering, aerospace, robotics, or closely related field.
• Strong understanding of agentic AI systems to design and reason about evaluations.
• Experience with AI evaluations, including creating evaluation tasks, designing or crafting judges, and comparing automated evaluation against human judgment.
• Comfortable working directly with customers, engineers, SMEs, partners, and internal technical teams to extract what matters from ambiguous workflows.
• Strong product judgment, turning observations into concrete evaluation work.
• Motivated by building superintelligence for engineering and excited by making AI systems capable in the physical world.

Responsibilities

• Study how engineers and customers use Archie in the field and identify where Archie succeeds or fails from an engineering perspective.
• Recreate real-world engineering tasks and failure modes for repeated use by development teams.
• Build and refine evaluation methodologies for Archie, including automated judges and human evaluation processes.
• Assess whether automated judges align with expert human judgment and improve them.
• Work across engineering, product, customers, SMEs, and external partners to reflect real workflows in evaluations.
• Use evaluation results to help the team understand where Archie needs to improve and which problems matter most to users.

Skills

Physical engineering background

Agentic AI understanding

AI evaluation experience

Customer & partner engagement

Product judgment

Motivation to build superintelligence

Job description

P-1 AI is building an AI engineer agent for the physical world named Archie. In this role you’ll work with engineers, customers, partners, and SMEs to challenge Archie on tough engineering work, understand where it fails, and capture those failures in evaluations that guide product development.

You will design evaluation tasks, recreate real-world failure modes, build automated judges, and assess alignment with expert judgment, ensuring our evaluations reflect real engineering workflows rather