Senior Data Platform Engineer — PB-Scale Pipelines

EngineersOfAI; Greater London; Hybrid; GBP 90.000 - 120.000; Vollzeit

Zusammenfassung

Wayve is seeking a Senior Software Engineer, Data to build and operate the Data Transfer Hub, a globally distributed system handling petabytes of sensor data daily. You’ll work with partners like Uber, Stellantis and Nissan to enable AI model training at scale.

You will design data pipelines, implement orchestration with Flyte and Kafka, and deploy on Azure, writing Terraform and Python to ensure reliability and observability across a cloud-first environment.

Qualifikationen

• Proven experience building and operating large-scale data transfer pipelines.
• Strong software engineering skills.
• Experience delivering & maintaining production systems.
• Experience with workflow orchestration (Flyte preferred).
• Experience building reliable, observable data systems.

Aufgaben

• Build and operate large-scale data transfer pipelines for video, LiDAR and sensor data from global partners at up to PB/day scale.
• Contribute to the technical design of the Data Transfer Hub with trade-offs on UX, reliability, throughput, cost, partner constraints, observability and operational support.
• Build parallelised, distributed data transfer pipelines using Flyte for workflow orchestration, Kafka/event-driven patterns for lifecycle tracing, and Azure for storage and transfer infrastructure.
• Help implement and evolve a globally distributed hub-and-spoke data transfer model to retrieve and share sensor data at scale with partners.
• Write infrastructure-as-code in Terraform and pipeline logic primarily in Python.
• Build reliable operational foundations, including retry strategies, backfills, data integrity checks, dashboards and alerting.
• Work on cloud-to-cloud and cross-cloud networking solutions, with Azure as the primary cloud platform.
• Once established, participate in an out-of-hours on-call and support rotation.
• Collaborate with product, commercial and partner-facing teams to understand constraints and deliver solutions that solve real customer and partner problems.
• Work in a fast-changing environment where requirements and partner deliverable dates evolve, while maintaining high engineering standards.

Kenntnisse

Large-scale pipelines

Software engineering

Production systems

Workflow orchestration

Observability

Tools

Flyte

Kafka

Azure

Terraform

Python

Jobbeschreibung

Wayve is seeking a Senior Software Engineer, Data to build and operate the Data Transfer Hub, a globally distributed system handling petabytes of sensor data daily. You’ll work with partners like Uber, Stellantis and Nissan to enable AI model training at scale.

You will design data pipelines, implement orchestration with Flyte and Kafka, and deploy on Azure, writing Terraform and Python to ensure reliability and observability across a cloud-first environment.