Senior Data Pipeline & Orchestration Engineer

Wayve; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Wayve is seeking a Senior Software Engineer to join the Data and Orchestration team in London. You will help build scalable data pipelines and a DAG-based orchestration system powering data ingestion, processing and inference at scale.

You will contribute to the Harness revamp, develop new infrastructure, and collaborate with partner teams to ensure reliability, performance and ease of use across the data platform.

Qualifications

• 7+ years of professional software engineering experience.
• Strong Python programming skills.
• Experience with distributed systems and large data volumes.
• Familiarity with Apache Kafka.

Responsibilities

• Design, build and evolve scalable data pipeline and orchestration infrastructure for Wayve's next-generation autonomous driving data platform.
• Contribute to the revamp of the existing Harness system and the development of a new DAG-based workflow platform.
• Build tooling and libraries that make data ingestion, processing, inference and evaluation workflows simple, reliable and consistent across teams.
• Ensure the platform can run hundreds of parallel pipeline steps efficiently, with correct outputs, no conflicts and effective resource prioritisation.
• Support batch inference and performance optimisation workflows, helping teams run evaluations in a cost-effective and scalable way.
• Improve platform reliability, quality and usability, including incident response workflows aligned to business-defined SLOs.
• Collaborate with partner teams to understand their pipeline requirements and provide the underlying infrastructure that enables them to move quickly

Skills

7+ years experience

Python

Unit testing

Distributed systems

Tools

Kafka

Postgres

ClickHouse

Job description

Wayve is seeking a Senior Software Engineer to join the Data and Orchestration team in London. You will help build scalable data pipelines and a DAG-based orchestration system powering data ingestion, processing and inference at scale.

You will contribute to the Harness revamp, develop new infrastructure, and collaborate with partner teams to ensure reliability, performance and ease of use across the data platform.