Senior Data Engineer — PB-scale Data Transfer Hub (Hybrid)

Wayve; London (KY); Hybrid; USD 119,000 - 172,000; Full time

Job summary

Wayve is seeking a Senior Software Engineer, Data to build the Data Transfer Hub, a globally distributed system for PB-scale sensor data exchange with partners. You will design and operate data pipelines across video, LiDAR and other sensor data, using Flyte, Kafka, and Azure technologies.

You will join a small, autonomous team in London, contributing to architecture, reliability and observability while maintaining high engineering standards in a fast-changing environment.

Qualifications

• Proven experience building and operating large-scale data transfer, ingestion or data processing pipelines, ideally at multi-TB or PB scale.
• Strong software engineering skills and production systems delivery experience.
• Experience with workflow orchestration for parallelised jobs (Flyte preferred; other tools also relevant).
• Experience building reliable, observable data systems with retry, backfills, data integrity checks and alerting.
• Strong cloud skills, Azure preferred; AWS/GCP a plus.
• Proficiency in Python and Terraform.
• Experience with blob/object storage, large-scale file movement, networking or cloud-to-cloud data transfer.
• Product mindset: focus on customer problems and deliver real value.
• Comfort in fast-changing environments and ownership mentality.

Responsibilities

• Build and operate large-scale data transfer pipelines for video, LiDAR and sensor data from global partners at PB/day scale.
• Contribute to Data Transfer Hub design, balancing UX, throughput, cost and observability.
• Develop distributed pipelines using Flyte, Kafka/event-driven patterns, and Azure storage/infrastructure.
• Implement hub-and-spoke data transfer model for global data sharing with partners.
• Write infrastructure-as-code in Terraform and pipeline logic in Python.
• Create reliable operational foundations including retries, backfills, data integrity checks and dashboards.

Skills

Data pipelines

Python

Terraform

Azure

Observability

Production systems

Cloud platforms

Ownership

Fast-changing environment

Tools

Flyte

Kafka

Airflow

Prefect

Job description

Wayve is seeking a Senior Software Engineer, Data to build the Data Transfer Hub, a globally distributed system for PB-scale sensor data exchange with partners. You will design and operate data pipelines across video, LiDAR and other sensor data, using Flyte, Kafka, and Azure technologies.

You will join a small, autonomous team in London, contributing to architecture, reliability and observability while maintaining high engineering standards in a fast-changing environment.