Senior ML Evaluation & Introspection Infra Engineer

Auror Limited; Mountain View (CA); Presencial; USD 162.000 - 260.000; Jornada completa

Descripción de la vacante

Aurora is seeking a Senior Software Engineer for ML Evaluation & Introspection Infrastructure. You will join the Behavior Analysis team and own end-to-end systems that evaluate model versions on large driving datasets and provide searchable embeddings for driving scenarios.

You will help shape how frontier LLMs, VLMs and agents are used to automate analysis, labeling, and data curation, working closely with ML engineers across the company.

Formación

• Typically 5+ years of professional software engineering experience building backend, data or ML infrastructure.
• Strong Python, and proficiency with SQL.
• Designed and operated large-scale batch or data pipelines on cloud infrastructure, including backfills, retries and cost management.
• Built systems that measure or evaluate something: metrics, experimentation, evaluation or analytics platforms.
• Experience with distributed compute and container orchestration (e.g. Kubernetes, AWS Batch or similar).

Responsabilidades

• Build and scale evaluation pipelines that compare model versions on large driving datasets: offline evals, slice-based metrics, regression detection, and model-vs-model reports.
• Own embedding and vector-search pipelines over driving data, versioned per model, so engineers can search, cluster and compare how any model represents the world.
• Build model introspection infrastructure (attention, attribution, ablations) that turns “why did the model do that?” from a research project into a query.
• Help define how frontier LLMs, VLMs and agents change ML analysis here: triaging, labeling, clustering and summarizing driving scenarios and eval results.
• We already have a prototype using LLMs for data curation, and you’d help decide where it goes next.
• Work side by side with the ML engineers training these models to decide what to measure, then make those measurements trustworthy, observable and cheap enough to run on every candidate model.
• Build the web app and CLI that engineers and AI agents use to find datasets, training runs, evals and their costs, designed with agents as first-class users.

Conocimientos

Python

SQL

Distributed compute

Kubernetes

AWS Batch

Herramientas

Kubernetes

AWS Batch

Descripción del empleo

Aurora is seeking a Senior Software Engineer for ML Evaluation & Introspection Infrastructure. You will join the Behavior Analysis team and own end-to-end systems that evaluate model versions on large driving datasets and provide searchable embeddings for driving scenarios.

You will help shape how frontier LLMs, VLMs and agents are used to automate analysis, labeling, and data curation, working closely with ML engineers across the company.