Senior ML Evaluation & Introspection Engineer

Aurora; Pittsburgh (Allegheny County); On-site; USD 146,000 - 234,000; Full time

Job summary

Aurora is hiring a Senior Software Engineer for ML Evaluation & Introspection Infrastructure to join Behavior Analysis. You’ll own core pipelines that compare model versions using large driving datasets and build embeddings, vector search, and model introspection tools.

You will collaborate with ML engineers to define measurements, ensure data quality, and deliver scalable analysis platforms, plus an end-to-end web app and CLI for dataset and run discovery.

Qualifications

• 5+ years of professional software engineering experience building backend, data or ML infrastructure.
• Strong Python, and proficiency with SQL.
• Designed and operated large-scale batch or data pipelines on cloud infrastructure, including backfills, retries and cost management.
• Built systems that measure or evaluate something: metrics, experimentation, evaluation or analytics platforms.
• Experience with distributed compute and container orchestration (e.g. Kubernetes, AWS Batch or similar).

Responsibilities

• Build and scale evaluation pipelines that compare model versions on large driving datasets: offline evals, slice-based metrics, regression detection, and model-vs-model reports.
• Own embedding and vector-search pipelines over driving data, versioned per model, so engineers can search, cluster and compare how any model represents the world.
• Build model introspection infrastructure (attention, attribution, ablations) that turns “why did the model do that?” from a research project into a query.
• Help define how frontier LLMs, VLMs and agents change ML analysis here: triaging, labeling, clustering and summarizing driving scenarios and eval results.
• Work side by side with the ML engineers training these models to decide what to measure, then make those measurements trustworthy, observable and cheap enough to run on every candidate model.
• Build the web app and CLI that engineers and AI agents use to find datasets, training runs, evals and their costs, designed with agents as first-class users.

Skills

Python

SQL

Batch processing

Distributed compute

Kubernetes

Tools

Kubernetes

AWS Batch

Docker

Job description

Aurora is hiring a Senior Software Engineer for ML Evaluation & Introspection Infrastructure to join Behavior Analysis. You’ll own core pipelines that compare model versions using large driving datasets and build embeddings, vector search, and model introspection tools.

You will collaborate with ML engineers to define measurements, ensure data quality, and deliver scalable analysis platforms, plus an end-to-end web app and CLI for dataset and run discovery.