Data Ingestion Engineer: High-Volume Spark Pipelines

Wayve; Sunnyvale (CA); Hybrid; USD 170,000 - 230,000; Full time

Benefits offered by this job

Hybrid working

Equity

Relocation assistance

Learning budget

Health insurance

Job summary

Wayve is seeking a data engineer to join the Ingestion Systems team in Sunnyvale. You will debug and optimize high-volume data pipelines that feed annotation, model training and evaluation.

You’ll work with Spark, Python, Airflow and Databricks to build resilient, cost-efficient data flows across partners and data sources. The role demands independent problem-solving and a passion for reliable data infrastructure.

Qualifications

• Experience with Spark and Python in production.
• Built, operated or debugged large-scale ingestion pipelines.
• Diagnose production failures involving corrupt data, stalled queues, retries and complex dependencies.
• Optimise high-throughput systems for reliability, performance and cost.
• Able to learn an existing system quickly, work independently and deliver practical improvements in a fast-moving environment.
• Experience with Airflow, Flyte, Databricks, Delta Lake, Scala or Java is a plus.

Responsibilities

• Debug and unblock failing ingestion pipelines, stalled jobs and processing queues.
• Investigate corrupt, malformed or unexpected data from a variety of sources.
• Optimise Spark jobs and batch-processing pipelines for throughput, reliability and compute efficiency.
• Design and implement more resilient pipelines so individual bad data segments do not block wider workflows.
• Improve monitoring, retries and failure recovery across multi-step ingestion workflows.
• Help prioritise and unblock important datasets for downstream annotation, data science and model-training teams, and contribute to the technical direction of Wayve's ingestion platform.
• High-volume pipelines that process real-world driving data for annotation, data science, model training and evaluation.

Skills

Apache Spark

Python

Ingestion pipelines

ETL processes

Data processing

System debugging

Tools

Airflow

Flyte

Databricks

Delta Lake

Scala

Java

Job description

Wayve is seeking a data engineer to join the Ingestion Systems team in Sunnyvale. You will debug and optimize high-volume data pipelines that feed annotation, model training and evaluation.

You’ll work with Spark, Python, Airflow and Databricks to build resilient, cost-efficient data flows across partners and data sources. The role demands independent problem-solving and a passion for reliable data infrastructure.