Member of Technical Staff, AI Evaluation

P-1 AI; United States; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Equity

Healthcare

Dental

Vision

401(k) match

Unlimited PTO

Job summary

P-1 AI is building an AI engineer agent for the physical world named Archie. In this role you’ll work with engineers, customers, partners, and SMEs to challenge Archie on tough engineering work, understand where it fails, and capture those failures in evaluations that guide product development.

You will design evaluation tasks, recreate real-world failure modes, build automated judges, and assess alignment with expert judgment, ensuring our evaluations reflect real engineering workflows rather

Qualifications

• Technical background in a physical engineering domain such as power systems, power electronics, mechanical engineering, aerospace, robotics, or closely related field.
• Strong understanding of agentic AI systems to design and reason about evaluations.
• Experience with AI evaluations, including creating evaluation tasks, designing or crafting judges, and comparing automated evaluation against human judgment.
• Comfortable working directly with customers, engineers, SMEs, partners, and internal technical teams to extract what matters from ambiguous workflows.
• Strong product judgment, turning observations into concrete evaluation work.
• Motivated by building superintelligence for engineering and excited by making AI systems capable in the physical world.

Responsibilities

• Study how engineers and customers use Archie in the field and identify where Archie succeeds or fails from an engineering perspective.
• Recreate real-world engineering tasks and failure modes for repeated use by development teams.
• Build and refine evaluation methodologies for Archie, including automated judges and human evaluation processes.
• Assess whether automated judges align with expert human judgment and improve them.
• Work across engineering, product, customers, SMEs, and external partners to reflect real workflows in evaluations.
• Use evaluation results to help the team understand where Archie needs to improve and which problems matter most to users.

Skills

Physical engineering background

Agentic AI understanding

AI evaluation experience

Customer & partner engagement

Product judgment

Motivation to build superintelligence

Job description

TL;DR:

If you:

• have mastered extreme systems in a physical engineering domain to know what peak engineering actually looks like;
• have built or worked with AI evaluations, including judges and human evaluation;
• can work comfortably with customers, subject matter experts, partners, and technical teams;
• are motivated by the goal of building superintelligence for engineering…

About P-1 AI:

At P-1 AI, we are building an AI engineer agent for the physical world named Archie. We maximize Archie’s anthropomorphism so that he fits seamlessly into existing engineering teams and workflows in the form factor of a human engineer. Archie today is at the level of a junior mechanical and electrical engineer, with a quantitative intuition over the product design space and the ability to use complex engineering tools—the same tools his human teammates use. Archie's tech stack includes a custom agentic harness, structured design representation, continual skills learning, and small custom post-trained models (SFT and RLVR) using proprietary semi-synthetic training data sets and environments which create a deep competitive moat. Our ultimate aim is to build engineering ASI. We recently announced a $50 million Series A financing led by NEA, which added former General Electric CEO Jeff Immelt to our board. The round also included the addition of several AI luminaries from Anthropic and Nominal to our existing angel investors from Google and OpenAI.

About the opportunity:

In this role, you’ll sit at the forefront of AI for physical engineering, working with subject matter experts, customers, partners, and our internal teams to challenge and evaluate Archie on tough engineering work, understand where it fails, and capture those failures in evaluations that guide product development. You need to understand not only whether outlook is plausible, but whether the underlying engineering reasoning and decisions make sense.

About the role:

• Study how engineers and customers use Archie in the field, identify where Archie succeeds or fails from an engineering perspective, and turn those observations into actionable evaluations.
• Recreate real-world engineering tasks and failure modes in environments that can be repeatedly used by our development teams.
• Build and refine evaluation methodologies for Archie, including automated judges and human evaluation processes.
• Assess whether automated judges actually align with expert human judgment and improve them when they do not.
• Work across engineering, product, customers, SMEs, and external partners to make sure our evaluations reflect real engineering workflows rather than abstract benchmarks.
• Use evaluation results to help the team understand where Archie needs to improve and which problems matter most to users.

About you:

• Technical background in a physical engineering domain such as power systems, power electronics, mechanical engineering, aerospace, robotics, or a closely related field.
• Strong enough understanding of agentic AI systems to design and reason about evaluations.
• Experience with AI evaluations, including creating evaluation tasks, designing or crafting judges, and comparing automated evaluation against human judgment.
• Comfortable working directly with customers, engineers, subject matter experts, partners, and internal technical teams to extract what matters from ambiguous workflows.
• Strong product judgment, with the ability to turn observations about how a system is being used or failing into concrete evaluation work.
• Motivated by the mission of building superintelligence for engineering and excited by the challenge of making AI systems genuinely capable in the physical world.

Location:

Remote (US/Canada) or San Mateo, CA. Remote employees spend one week out of six working together on-site in our San Mateo office. Relocation support available.

Benefits:

Competitive salary, meaningful equity ownership, healthcare, dental, vision, 401(k) match, and unlimited PTO.

Interview process:

• Introductory call (30 mins)

• Biographical/behavioural interview (45 mins)
• Technical interview (60 mins)
• CEO interview (30 mins)