Senior ML Evaluation & Introspection Infra Engineer

Aurora; Mountain View (CA); On-site; USD 162,000 - 260,000; Full time

Job summary

Aurora is seeking a Senior Software Engineer, ML Evaluation & Introspection Infrastructure to join the Behavior Analysis team. You will own evaluation pipelines, embedding/search, and model introspection tools, collaborating with ML engineers to make analysis faster and more trustworthy.

You’ll develop end-to-end systems for offline evals, model-version comparisons, and data-driven insights on driving scenarios, enabling safer and scalable self-driving tech.

Qualifications

• 5+ years of professional software engineering experience building backend, data or ML infrastructure.
• Strong Python, and proficiency with SQL.
• Designed and operated large-scale batch or data pipelines on cloud infrastructure, including backfills, retries and cost management.
• Built systems that measure or evaluate something: metrics, experimentation, evaluation or analytics platforms.
• Experience with distributed compute and container orchestration (e.g. Kubernetes, AWS Batch or similar).

Responsibilities

• Build and scale evaluation pipelines that compare model versions on large driving datasets: offline evals, slice-based metrics, regression detection, and model-vs-model reports.
• Own embedding and vector-search pipelines over driving data, versioned per model, enabling search, clustering and comparison of representations.
• Build model introspection infrastructure (attention, attribution, ablations) to turn 'why' into a query.
• Help define how frontier LLMs, VLMs and agents affect ML analysis and data curation.
• Collaborate with ML engineers to ensure measurements are trustworthy and cost-effective.
• Develop the web app and CLI for datasets, training runs, evals and costs, with agents as first-class users.

Skills

Python

SQL

Data pipelines

Kubernetes

AWS Batch

Backend engineering

Job description

Aurora is seeking a Senior Software Engineer, ML Evaluation & Introspection Infrastructure to join the Behavior Analysis team. You will own evaluation pipelines, embedding/search, and model introspection tools, collaborating with ML engineers to make analysis faster and more trustworthy.

You’ll develop end-to-end systems for offline evals, model-version comparisons, and data-driven insights on driving scenarios, enabling safer and scalable self-driving tech.