Staff Engineer, Autonomous Driving Data Platform & Curation New Mountain View, CA

CARIAD, Inc.; Mountain View (CA); Hybrid; USD 162,000 - 234,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

401k match

Disability insurance

Life insurance

AD&D insurance

Tuition reimbursement

Student loan repayment

Paid time off

Vehicle lease program

Job summary

CARIAD, Inc. in Mountain View, CA is seeking a Staff Engineer to own the path from raw multimodal vehicle data to ready datasets. You’ll lead ingestion, schema, storage, and curation for 10M+ records, enabling efficient retrieval and model-ready data for autonomous driving.

The role requires deep ML/data engineering expertise, experience with Parquet/Lance/Arrow, and ability to design scalable pipelines, governance, and observability across cloud storage and on-prem systems.

Qualifications

• 8+ years of experience in applied machine learning or deep learning.
• 4+ years of experience in reinforcement learning, computer vision, or AD/ADAS systems.
• Master’s in Computer Science, Robotics, Electrical Engineering, Applied Mathematics, or a related field.
• PhD in Computer Science, Robotics, Electrical Engineering, Applied Mathematics, or a related field.

Responsibilities

• Design canonical representations for drives, scenarios, frames, trajectories, and dataset manifests.
• Build and maintain validated, versioned datasets containing 10M+ records on local or cloud storage.
• Evaluate Parquet, Apache Arrow, Lance; own partitioning, indexing, schema evolution, lineage, and reproducibility.
• Optimize filtering, projection, joins, scenario slicing, and data-loading performance.
• Measure and improve ingestion throughput, query latency, storage utilization, and cost per usable sample.
• Lead data quality monitoring, labeling workflows, and reproducibility across teams.

Skills

Autonomous driving data

Production data pipelines

Python

SQL

Parquet/Lance/Arrow

Education

Master's in Computer Science, Robotics, Electrical Engineering, Applied Mathematics, or related field

Tools

Parquet

Lance

Apache Arrow

Spark

Beam

Airflow

Job description

Staff Engineer, Autonomous Driving Data Platform & Curation

Mountain View, CA

We are CARIAD, an automotive software development team with the Volkswagen Group. Our mission is to make the automotive experience safer, more sustainable, more comfortable, more digital, and more fun. To achieve that we are building the leading tech stack for the automotive industry and creating a unified software platform for over 10 million new vehicles per year. We’re looking for talented, digital minds like you to help us create code that moves the world. Together with you, we’ll build outstanding digital experiences and products for all Volkswagen Group brands that will transform mobility. Join us as we shape the future of the car and everyone around it.

Role Summary:

The Staff Engineer, Autonomous Driving Data Platform & Curation is a hands‑on staff-level individual contributor who owns the path from raw multimodal vehicle data to reliable, versioned, and model‑ready datasets. The ideal candidate combines production data or ML systems experience with practical knowledge of autonomous‑driving data and can lead ingestion, schema, storage, query, curation, quality, and delivery. This engineer designs and operates datasets containing more than 10 million records or samples, evaluates formats such as Parquet and Lance, and enables efficient filtering, slicing, random access, and sequential retrieval. The role also advances data‑quality monitoring, statistical and out‑of‑distribution detection, rule‑based and model‑based tagging, model‑in‑the‑loop and human‑in‑the‑loop labeling, and future data preparation for imitation learning and reinforcement learning.

Role Responsibilities:

• Design canonical representations for drives, scenarios, clips, frames, trajectories, sensor references, vehicle state, map context, labels, predictions, and dataset manifests.
• Build and maintain validated, versioned datasets containing more than 10 million records or samples on local or cloud object storage.
• Evaluate Parquet, Apache Arrow, Lance, and related technologies; own partitioning, indexing, file sizing, compaction, schema evolution, lineage, and reproducibility.
• Optimize filtering, projection, joins, scenario slicing, random sampling, shuffling, sequential retrieval, and model data‑loading performance.
• Measure and improve ingestion throughput, query latency, training throughput, storage utilization, reliability, and cost per usable sample.

Autonomous Dataset Lifecycle, Curation & Training Readiness

• Work effectively with synchronized camera and other sensor data, ego state, localization, calibration, coordinate frames, map context, control actions, clips, and trajectories.
• Translate perception, planning, VLA, and evaluation needs into schemas, searchable attributes, scenario definitions, sampling strategies, and reproducible dataset splits.
• Build reliable batch or distributed pipelines for ingestion, transformation, enrichment, validation, cataloging, and publication, including retries, backfills, idempotency, and observability.
• Enable self‑service discovery and composition for lane keeping, lane changes, long‑tail scenarios, hard examples, balanced datasets, and leakage‑resistant train, validation, and test splits.

Data Quality, Statistics & Distribution Monitoring

• Define automated quality gates, dashboards, and alerts for completeness, validity, freshness, duplication, synchronization, calibration, corruption, label integrity, coverage, balance, and cost.
• Apply statistics, sampling, and distribution comparisons to detect drift, anomalies, under‑represented conditions, and out‑of‑ distribution data.
• Use deterministic checks, heuristic rules, geometry and metadata queries, embeddings, VLMs, and learned models to assess quality and create searchable scenario tags.
• Quarantine suspicious data and lead root‑cause analysis across collection, synchronization, schema, transformation, storage, annotation, sampling, and model‑consumer failures.

Auto‑Labeling, Closed‑Loop Data Automation & RL Readiness

• Design model‑in‑the‑loop and human‑in‑the‑loop labeling workflows with confidence thresholds, review routing, audit sampling, disagreement handling, and label provenance.
• Close the loop from model failures and edge cases through selection, annotation, quality review, dataset publication, training, and evaluation.
• Evaluate auto‑labeled and synthetic data using label quality, coverage, distributional impact, and downstream model performance.
• Prepare replayable trajectory data for imitation learning and offline RL, including observations, actions, timestamps, policy versions, interventions, rewards, termination conditions, and alignment checks.

Cross‑Functional Technical Leadership, Reliability & Cost

• Serve as the staff‑level owner and escalation point for dataset architecture, storage and query performance, data quality, reliability, reproducibility, and cost.
• Align vehicle collection, autonomy metadata, model interfaces, annotation, storage, governance, and training requirements across partner teams.
• Communicate risks, trade‑offs, decisions, and recovery plans with clear evidence; maintain traceable schemas, data contracts, lineage, operating procedures, and incident records.
• Mentor engineers and strengthen design reviews, testing, observability, reproducibility, cost awareness, and data engineering standards.

General Skills:

• Deep production experience with Parquet and/or Lance, including dataset migration, indexing, compaction, schema evolution, version maintenance, cloud storage, random access, or training‑loader optimization.
• Experience with large‑scale multimodal autonomous‑driving data from cameras, LiDAR, radar, maps, localization, vehicle state, simulation, or other sensor‑rich robotic systems.
• Experience building auto‑labeling, active‑learning, model‑in‑the‑loop, or human‑in‑the‑loop systems, including confidence calibration, annotation routing, review workflows, audit sampling, label agreement, and annotation QA.
• Strong statistical knowledge for quality monitoring, distribution comparison, drift measurement, anomaly detection, and out‑of‑distribution detection.
• Experience using rule‑based logic, geometry and metadata heuristics, embeddings, semantic search, VLMs, classifiers, or other learned methods to tag data, assess quality, mine edge cases, detect hard examples, or prioritize review.
• Experience connecting dataset metrics, composition, label quality, drift, cost, and downstream model performance through dashboards and evaluation.
• Experience optimizing distributed ML data loading, caching, sampling, preprocessing, and the data‑to‑GPU path.
• Experience preparing sequential or trajectory datasets for imitation learning, offline reinforcement learning, preference or ranking data, reward modeling, or policy evaluation, including observations, actions, rewards, policy provenance, and replay semantics.
• Familiarity with data governance, privacy, retention, redaction, access control, auditability, and security requirements for fleet‑collected sensor data.

Required Specialized Skills:

• Demonstrated ownership of production data pipelines and large datasets, including systems with more than 10 million records or samples or comparable multi‑terabyte scale.
• Strong dataset and storage foundation, including schema design, columnar storage, partitioning, file and fragment sizing, compression, predicate pushdown, projection, indexing, compaction, schema evolution, versioning, lineage, and reproducibility.
• Hands‑on experience with Parquet, Lance, Apache Arrow, or a comparable analytical dataset technology, with the ability to reason about format selection and benchmark storage, query, update, and model‑loading trade‑offs.
• Strong Python and SQL skills with production software‑engineering practices, including testing, code review, version control, profiling, debugging, maintainable APIs, and operational documentation.
• Experience building distributed or parallel data pipelines and orchestration using technologies such as Ray, Spark, Beam, Dask, Airflow, Dagster, or comparable systems.
• Experience with local or cloud object storage, such as S3 or GCS, and practical understanding of throughput, request patterns, caching, data movement, reliability, access control, and cost.
• Basic autonomous‑driving or robotics data knowledge, including temporal sensor data, camera and other sensor modalities, timestamps and synchronization, calibration, ego and vehicle state, coordinate frames, clips, trajectories, scenarios, and annotations.
• Ability to design automated data validation and observability and translate ML training, sampling, evaluation, and failure‑analysis needs into reliable data products.

Desired Skills:

• Deep production experience with Parquet and/or Lance, including dataset migration, indexing, compaction, schema evolution, version maintenance, cloud storage, random access, or training‑loader optimization.
• Experience with large‑scale multimodal autonomous‑driving data from cameras, LiDAR, radar, maps, localization, vehicle state, simulation, or other sensor‑rich robotic systems.
• Experience building auto‑labeling, active‑learning, model‑in‑the‑loop, or human‑in‑the‑loop systems, including confidence calibration, annotation routing, review workflows, audit sampling, label agreement, and annotation QA.
• Strong statistical knowledge for quality monitoring, distribution comparison, drift measurement, anomaly detection, and out‑of‑distribution detection.
• Experience using rule‑based logic, geometry and metadata heuristics, embeddings, semantic search, VLMs, classifiers, or other learned methods to tag data, assess quality, mine edge cases, detect hard examples, or prioritize review.
• Experience connecting dataset metrics, composition, label quality, drift, cost, and downstream model performance through dashboards and evaluation.
• Experience optimizing distributed ML data loading, caching, sampling, preprocessing, and the data‑to‑GPU path.
• Experience preparing sequential or trajectory datasets for imitation learning, offline reinforcement learning, preference or ranking data, reward modeling, or policy evaluation, including observations, actions, rewards, policy provenance, and replay semantics.
• Familiarity with data governance, privacy, retention, redaction, access control, auditability, and security requirements for fleet‑collected sensor data.

Years of Relevant Experience:

• 8+ years of experience in applied machine learning or deep learning
• 4+ years ofexperience reinforcement learning, computer vision, or AD/ADAS systems.
• Strong candidates with equivalent industry experience will be considered

Required Education:

Master’s in Computer Science, Robotics, Electrical Engineering, Applied Mathematics, or a related field

Desired Education:

PhD in Computer Science, Robotics, Electrical Engineering, Applied Mathematics, or a related field

Compensation

Salary range is dependent on factors such as geographical differentials, credentials or certifications, industry-based experience, qualification and training. In the city of Mountain View, CA, the salary range for this position is $161,710 - $234,325.

CARIAD, Inc. provides performance based merits and annual bonus along with a competitive benefits package. Benefits include medical, dental, vision, 401k with employer match and defined contribution plan, short and long term disability, basic life and AD&DD insurance, employee assistance program, tuition reimbursement and student loan repayment plans, maternity and non-primary caregiver leave, adoption assistance, employee referral program and vacation and paid holidays. We also offer a unique vehicle lease program that covers registration and insurance fees.

CARIAD is an Equal Opportunity Employer. We welcome and encourage applicants from all backgrounds, and do not discriminate based on race, sex, age, disability, sexual orientation, nationalorigin, religion, color, gender identity/expression, marital status, veteran status, or any other characteristics protected by applicable laws.