Data Platform Engineer — Quality & Provenance at Scale

Meyandy LLC; Northern (KY); Hybrid; USD 110,000 - 170,000; Full time

Job summary

Wayve is seeking a Data Engineer focused on Data Quality & Provenance to build the data foundation for autonomous-driving development. You will turn fleet and simulation data into trusted datasets for ML, autonomy, simulation and safety teams to use with confidence.

You will own data quality, lineage, observability, and reproducibility for production data workflows, collaborating with ML, autonomy and safety engineers to define schemas, APIs and data contracts at petabyte scale.

Qualifications

• Strong Python and SQL with production software engineering fundamentals.
• Experience designing and operating large-scale distributed data systems.
• Hands-on experience with distributed processing and workflow orchestration technologies (Spark, Flyte, Airflow).
• Experience building and operating cloud-based data platforms using object storage, governance and cost management.
• Proven ownership of data quality, lineage, observability, reproducibility and incident response for production data workflows.
• Ability to translate ambiguous requirements from ML, data-science, robotics or similar teams into reusable platform capabilities.
• Comfort operating in ambiguity and helping define boundaries and standards for a growing data platform.

Responsibilities

• Design, build and operate scalable batch and streaming pipelines for multimodal fleet and simulation data.
• Create data models, catalogs, indexes and query capabilities that make sensor, vehicle-state, map and event data easy to discover and use.
• Build versioned, reproducible datasets for training, evaluation, replay, scenario mining and safety analysis.
• Develop robust workflows for data ingestion, synchronisation, transformation, curation, labelling and data-quality validation.
• Partner with autonomy, ML, simulation and safety engineers to define schemas, APIs and data contracts.
• Establish strong standards for lineage, observability, access controls, retention and cost management across the data platform.
• Improve the performance, reliability and unit economics of large-scale storage and compute workloads.

Skills

Python

SQL

Distributed data systems

Spark

Airflow

Tools

Flyte

Spark

Cloud storage & data catalogs

Job description

Wayve is seeking a Data Engineer focused on Data Quality & Provenance to build the data foundation for autonomous-driving development. You will turn fleet and simulation data into trusted datasets for ML, autonomy, simulation and safety teams to use with confidence.

You will own data quality, lineage, observability, and reproducibility for production data workflows, collaborating with ML, autonomy and safety engineers to define schemas, APIs and data contracts at petabyte scale.