Senior Data Pipeline Engineer — Scalable Orchestration

Wayve; Greater London; Hybrid; GBP 90,000 - 150,000; Full time

Benefits offered by this job

Meaningful equity

Job summary

Wayve is building the leading AI platform for autonomous driving. The Data and Orchestration team in London builds the platforms behind data engineering, orchestration and inference workflows, enabling scalable pipelines across the fleet and partner teams.

You will revamp our Harness system and build a new DAG-based orchestration platform, while ensuring reliability and performance for high-volume data runs and evaluations.

Qualifications

• Strong software engineering fundamentals
• Solid Python programming skills
• Experience with distributed systems and high-throughput data
• Experience with testing practices (unit/integration)
• Familiarity with Kafka and event streaming
• Knowledge of databases and data warehouses (Postgres/ClickHouse)

Responsibilities

• Design, build and evolve scalable data pipeline and orchestration infrastructure
• Revamp Harness system and develop a DAG-based workflow platform
• Create tooling to simplify ingestion, processing, inference and evaluation workflows
• Ensure hundreds of parallel pipeline steps run efficiently with correct outputs
• Support batch inference and performance optimisation for evaluations
• Improve platform reliability and incident response aligned to SLOs

Skills

Python programming

Distributed systems

Data pipelines

Incident response

Performance optimisation

Collaboration with partners

Tools

Ray

Flyte

Spark

Job description

Wayve is building the leading AI platform for autonomous driving. The Data and Orchestration team in London builds the platforms behind data engineering, orchestration and inference workflows, enabling scalable pipelines across the fleet and partner teams.

You will revamp our Harness system and build a new DAG-based orchestration platform, while ensuring reliability and performance for high-volume data runs and evaluations.