Member of Technical Staff, AI Evaluation

Fdetimes; United States; Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Healthcare

Dental

Vision

401(k) match

Equity ownership

Unlimited PTO

Job summary

P-1 AI seeks an AI evaluation engineer to sit at the frontier of AI for physical engineering. You will work with engineers, customers, partners, and SMEs to challenge Archie on tough tasks, understand where it fails, and capture those failures in actionable evaluations guiding product development.

You will design and run evaluations, recreate real-world engineering challenges, and refine methods to measure alignment with expert human judgment. Remote-friendly with on-site options.

Qualifications

• Technical background in a physical engineering domain (e.g., power systems, robotics, mechanical/electrical).
• Experience with AI evaluations, including creating tasks and comparing automated vs human judgment.
• Comfortable working with customers, SMEs, partners, and internal teams.

Responsibilities

• Study how engineers and customers use Archie; identify success/failure from an engineering view.
• Recreate real-world tasks and failure modes for reusable evaluation environments.
• Build and refine evaluation methodologies, incl. automated judges and human evaluation processes.
• Assess alignment between automated judges and expert human judgment.

Skills

Physical engineering background

Agentic AI understanding

AI evaluation experience

Customer-facing

Product judgment

Superintelligence mission

Education

Bachelor's degree in engineering or related field

Tools

Evaluation design

Job description

TL;DR: If you

• have mastered extreme systems in a physical engineering domain to know what peak engineering actually looks like;
• have built or worked with AI evaluations, including judges and human evaluation;
• can work comfortably with customers, subject matter experts, partners, and technical teams;
• are motivated by the goal of building superintelligence for engineering…

About P-1 AI

At P-1 AI, we are building an AI engineer agent for the physical world named Archie. We maximize Archie’s anthropomorphism so that he fits seamlessly into existing engineering teams and workflows in the form factor of a human engineer. Archie today is at the level of a junior mechanical and electrical engineer, with a quantitative intuition over the product design space and the ability to use complex engineering tools—the same tools his human teammates use. Archie's tech stack includes a custom agentic harness, structured design representation, continual skills learning, and small custom post-trained models (SFT and RLVR) using proprietary semi-synthetic training data sets and environments which create a deep competitive moat. Our ultimate aim is to build engineering ASI. We recently announced a $50 million Series A financing led by NEA, which added former General Electric CEO Jeff Immelt to our board. The round also included the addition of several AI luminaries from Anthropic and Nominal to our existing angel investors from Google and OpenAI.

About the opportunity

In this role, you’ll sit at the forefront of AI for physical engineering, working with subject matter experts, customers, partners, and our internal teams to challenge and evaluate Archie on tough engineering work, understand where it fails, and capture those failures in evaluations that guide product development. You need to understand not only whether outlook is plausible, but whether the underlying engineering reasoning and decisions make sense.

About the role

• Study how engineers and customers use Archie in the field, identify where Archie succeeds or fails from an engineering perspective, and turn those observations into actionable evaluations.
• Recreate real-world engineering tasks and failure modes in environments that can be repeatedly used by our development teams.
• Build and refine evaluation methodologies for Archie, including automated judges and human evaluation processes.
• Assess whether automated judges actually align with expert human judgment and improve them when they do not.
• Work across engineering, product, customers, SMEs, and external partners to make sure our evaluations reflect real engineering workflows rather than abstract benchmarks.
• Use evaluation results to help the team understand where Archie needs to improve and which problems matter most to users.

About you

• Technical background in a physical engineering domain such as power systems, power electronics, mechanical engineering, aerospace, robotics, or a closely related field.
• Strong enough understanding of agentic AI systems to design and reason about evaluations.
• Experience with AI evaluations, including creating evaluation tasks, designing or crafting judges, and comparing automated evaluation against human judgment.
• Comfortable working directly with customers, engineers, subject matter experts, partners, and internal technical teams to extract what matters from ambiguous workflows.
• Strong product judgment, with the ability to turn observations about how a system is being used or failing into concrete evaluation work.
• Motivated by the mission of building superintelligence for engineering and excited by the challenge of making AI systems genuinely capable in the physical world.

Location

Remote (US/Canada) or San Mateo, CA. Remote employees spend one week out of six working together on-site in our San Mateo office. Relocation support available.

Benefits

Competitive salary, meaningful equity ownership, healthcare, dental, vision, 401(k) match, and unlimited PTO.

Interview process

• Introductory call (30 mins)
• Biographical/behavioural interview (45 mins)
• Technical interview (60 mins)
• CEO interview (30 mins)