AI Automation Engineer, Real-World Test Lab

niantic-spatial

United States

On-site

USD 120,000 - 170,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Niantic Spatial in the United States is hiring an AI Automation Engineer to design and own the automation that runs evaluation workflows end to end for the Real-World Test Lab. You will ensure repeatable, scalable processes and provide reliable data to guide product decisions.

The role emphasizes building infrastructure that executes evaluations autonomously across releases, departments, and devices, with a focus on trust and maintainability. Join a frontier team advancing physical AI.

Qualifications

  • Production-grade automation experience.
  • Experience evaluating systems where quality, accuracy, or latency thresholds matter.
  • Clear written communication of technical findings for action.

Responsibilities

  • Build the Evaluation Machine end-to-end: environment setup, run orchestration, artifact capture, result collection.
  • Make results comparable across product versions, devices and conditions.
  • Automate agent-driven workflows for priority customer use cases at scale.
  • Eliminate manual work by turning experiments into standing protocols.
  • Produce actionable diagnostics with reproducible data attached.

Skills

Python
Orchestration
CI/CD pipelines
Cloud storage
Reproducible environments

Education

Bachelor's degree in a relevant field

Tools

APIs integration

Job description

About Niantic Spatial

At Niantic Spatial, we're building the future of physical AI. Powered by a proprietary database of over 30 billion posed images, our groundbreaking mapping technology unlocks a new dimension of interaction and spatial intelligence that helps both humans and machines better understand, represent, navigate, and engage with the real environment.

Our reconstruction technology captures environments with geometric accuracy and extreme detail from any standard camera, and our Visual Positioning System delivers precise positioning almost anywhere in the world. We serve customers across robotics, the public sector, and energy and industrial markets - building for the 80% of economic activity that takes place beyond our screens.

About the Real-World Test Lab

Physical AI doesn't get graded on a leaderboard. It gets graded on a factory floor at shift change, in a substation with no GPS, on a site where the lighting is wrong and the stakes are real.

The Real-World Test Lab closes the gap between the benchmark and the field. We bring the customer's world inside our walls - their devices, their environments, their hardest conditions, and their definition of success - and make it the bar every release has to clear.

As Niantic Spatial's first and most demanding customer, we push our reconstruction, localization, and spatial understanding to their limits, find where they shine and where they break, and turn that into evidence that shapes what we build next. It's a new team at the frontier of physical AI, and you'll help invent how the job is done.

About the Role

We're hiring an AI Automation Engineer, reporting to the Director of the Real-World Test Lab, to build the system that produces our evidence. Today our evaluations are manual, inconsistent, and slow. You'll design and own the automation that runs them end to end on every relevant release, with no human driving it, turning one-off experiments into an always-on service the whole company relies on.

This role is about owning the evidence, not executing a test plan. You'll decide how each workflow gets exercised, and you'll be measured on whether the company can trust the results, not on how much automation exists. The work counts when it keeps running correctly months later, without you in the loop.

You believe evaluation is engineering, not process. You've built systems that test other systems, and you know the difference between a script that works on your laptop and infrastructure a team can trust. When a result couldn't be reproduced, you fixed the tooling instead of arguing about the number.

What You'll Do

Build the Evaluation Machine - Own the automation that executes Lab evaluations end to end: environment setup, run orchestration, artifact capture, and result collection. Make reruns free so we test constantly rather than occasionally.

Make Results Comparable - Instrument scorecards so a result can be compared across product versions, devices, and capture conditions. A number without its lineage is not evidence.

Automate the Agent-Driven Layer - Build the agent workflows that exercise priority customer use cases at realistic scale, across the graded difficulty spectrum from easy to frontier, and be honest about where agents cannot yet replace human judgment.

Kill Manual Work Permanently - Convert one-off experiments into standing protocols that run on every relevant release. Automate recurring inspection wherever it can be automated, and route what genuinely cannot to scalable human review.

Make Failures Actionable - Produce diagnostics precise enough that a finding reaches its owner with a reproducible case and data attached. Findings that need re-investigation before anyone can act on them are half-finished.

Keep Data From Being the Bottleneck - Work with the AI Data Manager so every run is reproducible from a known dataset state, without anyone downloading and re-uploading data by hand.

What Success Looks Like

30 days: One priority workflow evaluated end to end with no manual steps, with results in a comparable scorecard.

60 days : Evaluations trigger automatically on relevant releases, with diagnostics that route failures to the right owners.

90 days: Three priority workflows under standing automated evaluation, with version-over-version comparison for leadership.

What You'll Bring

Built and maintained production-grade automation or test infrastructure that other engineers relied on daily.

Experience evaluating systems where correctness is graded rather than binary, meaning quality, accuracy, or latency thresholds rather than pass/fail assertions.

Strong Python, with fluency in orchestration, CI-style pipelines, cloud storage, and reproducible environments.

Worked with backend services and APIs you did not own, integrating against them without becoming a bottleneck for their team.

Written up a technical finding clearly enough that a non-author could act on it without a meeting.

A bachelor's degree in a relevant field, or e

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Automation Engineer, Real-World Test Lab
AI Automation Engineer, Real-World Test Lab

SLAS (Society for Laboratory Automation and Screening) • San Francisco (CA)

On-site
USD 158,000 - 210,000
Medical insurance
Dental coverage
Vision coverage
+1
AI Automation Engineer, Real-World Test Lab
AI Automation Engineer, Real-World Test Lab

Niantic • San Francisco (CA)

Hybrid
USD 158,000 - 210,000
AI Automation Engineer, Real-World Test Lab
AI Automation Engineer, Real-World Test Lab

Niantic Spatial, Inc. • San Francisco (CA)

On-site
USD 158,000 - 210,000
Annual bonus
Equity
Medical coverage
+3
Applied Research — Evaluations & Data
Applied Research — Evaluations & Data

Human Intuition Inc. • New York (NY)

On-site
USD 120,000 - 180,000
Member of Technical Staff, Evaluations
Member of Technical Staff, Evaluations

Bake AI Inc. • San Mateo (CA)

On-site
USD 180,000 - 240,000
QA AI Automation Engineer
QA AI Automation Engineer

Dynasty Financial Partners • Saint Petersburg (FL)

On-site
USD 120,000 - 150,000
AI Evaluation Engineer
AI Evaluation Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 170,000
Staff Software Engineer, Agent Eval Platform
Staff Software Engineer, Agent Eval Platform

ServiceNow • Santa Clara (CA)

On-site
USD 180,000 - 320,000
Hardware Operations Engineer - Robotics, Real-World Test Lab
Hardware Operations Engineer - Robotics, Real-World Test Lab

Niantic Spatial, Inc. • San Francisco (CA)

On-site
USD 142,000 - 193,000
Medical, Dental, Vision
401(k)
AI Automation Engineer - End-to-End Evaluation
AI Automation Engineer - End-to-End Evaluation

Niantic • San Francisco (CA)

Hybrid
USD 158,000 - 210,000