AI Automation Engineer - End-to-End Evaluation

Niantic; San Francisco (CA); Hybrid; USD 158,000 - 210,000; Full time

Job summary

Niantic Spatial is hiring an AI Automation Engineer to design and own the end-to-end evaluation system that produces reproducible evidence across releases. You will automate evaluations, instrument scorecards, and build agent workflows to simulate real-world use cases at scale.

You will work with a data manager to ensure reproducible datasets and drive automated diagnostics across the platform. The role emphasizes engineering-first evaluation, building scalable automation, and delivering trusted

Qualifications

• Built and maintained production-grade automation or test infrastructure that other engineers relied on daily.
• Experience evaluating systems where correctness is graded rather than binary, meaning quality, accuracy, or latency thresholds rather than pass/fail assertions.
• Strong Python, with fluency in orchestration, CI-style pipelines, cloud storage, and reproducible environments.
• Worked with backend services and APIs you did not own, integrating against them without becoming a bottleneck for their team.
• Written up a technical finding clearly enough that a non-author could act on it without a meeting.
• A bachelor's degree in a relevant field, or equivalent experience.

Responsibilities

• Build the Evaluation Machine - Own the automation that executes Lab evaluations end to end: environment setup, run orchestration, artifact capture, and result collection.
• Make Results Comparable - Instrument scorecards so a result can be compared across product versions, devices, and capture conditions.
• Automate the Agent-Driven Layer - Build the agent workflows that exercise priority customer use cases at realistic scale, across the graded difficulty spectrum from easy to frontier.
• Kill Manual Work Permanently - Convert one-off experiments into standing protocols that run on every relevant release.
• Make Failures Actionable - Produce diagnostics precise enough that a finding reaches its owner with a reproducible case and data attached.
• Keep Data From Being the Bottleneck - Work with the AI Data Manager so every run is reproducible from a known dataset state.

Skills

Python

Orchestration

CI/CD

Cloud storage

APIs

Education

Bachelor's degree

Job description

Niantic Spatial is hiring an AI Automation Engineer to design and own the end-to-end evaluation system that produces reproducible evidence across releases. You will automate evaluations, instrument scorecards, and build agent workflows to simulate real-world use cases at scale.

You will work with a data manager to ensure reproducible datasets and drive automated diagnostics across the platform. The role emphasizes engineering-first evaluation, building scalable automation, and delivering trusted