Data and Ingestion Engineer - Robotics Sector

Cognitive Group | Part of the Focus Cloud Group; Greater London; On-site; GBP 85,000 - 130,000; Full time

Job summary

Cognitive Group | Part of the Focus Cloud Group is seeking an experienced Data Ingestion Engineer to join a cutting-edge AI and robotics organisation. You will work on large-scale data infrastructure supporting AI model training and continuous improvement, ingesting terabyte-scale real-world driving, vehicle and sensor data from automotive providers.

The role focuses on Python, PySpark/Spark and SQL, with exposure to robotics/automotive datasets including LiDAR, ADAS and ADAS-related systems,

Qualifications

• Experience building and operating large-scale ingestion/ETL platforms.
• Proven ability to process terabyte-scale data daily.
• Hands-on Python, Spark/PySpark and SQL with automotive/robotics data.

Responsibilities

• Design, build and operate high-throughput data ingestion pipelines processing terabyte-scale datasets.
• Ingest real-world driving, vehicle, camera, sensor and robotics data from multiple external providers.
• Process and transform datasets associated with ADAS, SLAM, LiDAR and autonomous vehicle/robotics systems.
• Build scalable ETL/ELT processes for collecting, aggregating, transforming and moving data across complex environments.
• Develop production data-processing solutions using Python, Spark/PySpark and SQL.
• Debug and resolve failing or blocked ingestion pipelines.
• Investigate corrupt, malformed, incomplete and inconsistent datasets without allowing individual failures to stall wider processing workflows.
• Improve pipeline resilience through validation, retries, queue management, observability and automated recovery.

Skills

Python

PySpark

SQL

Spark/Databricks

Databricks

Kafka

Airflow

AWS Glue

DLT

dbt

Dagster

S3

Iceberg

Snowflake

Redshift

PostgreSQL

Oracle

Robotics data

ADAS

Tools

Databricks

Airflow

Kafka

AWS Glue

Job description

We are looking for an experienced Data Ingestion Engineer to join a cutting-edge AI and robotics organisation, working on the large-scale data infrastructure that underpins the training and continuous improvement of advanced AI models.

This is a highly technical, hands-on engineering role focused on ingesting and processing huge volumes of real-world driving, vehicle and sensor data from automotive manufacturers, fleet operators, dashcam providers and other external data sources.

This is not a generalist cloud data engineering role. The successful candidate must have strong hands-on Python, PySpark/Spark and SQL expertise combined with demonstrable experience working with robotics or automotive data, including technologies and datasets associated with LiDAR, SLAM, ADAS, cameras, vehicle sensors and autonomous systems.

We are looking for engineers who have personally built or operated pipelines processing terabyte-scale data volumes per day and can clearly quantify the largest ingestion and data-processing environments they have worked with.

Key Responsibilities

• Design, build and operate high-throughput data ingestion pipelines processing terabyte-scale datasets
• Ingest real-world driving, vehicle, camera, sensor and robotics data from multiple external providers
• Process and transform datasets associated with ADAS, SLAM, LiDAR and autonomous vehicle/robotics systems
• Build scalable ETL/ELT processes for collecting, aggregating, transforming and moving data across complex environments
• Develop production data-processing solutions using Python, Spark/PySpark and SQL
• Debug and resolve failing or blocked ingestion pipelines
• Investigate corrupt, malformed, incomplete and inconsistent datasets without allowing individual failures to stall wider processing workflows
• Improve pipeline resilience through validation, retries, queue management, observability and automated recovery

Essential Skills & Experience

• Strong experience building and operating large-scale production data ingestion, ETL or data-processing platforms
• Demonstrable experience processing terabyte-scale data volumes per day
• Strong hands-on programming experience with Spark/PySpark, Python and SQL
• Strong experience with distributed data processing and technologies such as Spark and Databricks
• Experience with data engineering technologies such as DLT, dbt, Dagster, Airflow, Kafka and AWS Glue
• Experience working with large-scale cloud data platforms and storage technologies including S3 and Apache Iceberg
• Experience with databases and data warehouses such as Snowflake, Redshift, PostgreSQL and Oracle
• Previous experience within robotics, autonomous vehicles, ADAS, automotive AI or another real-world AI/ML environmentExperience processing vehicle, camera, sensor, telemetry or robotics datasets at significant scaleExperience working with data supplied by automotive OEMs, fleet operators or other large external data providers

This is an inside IR35 contract role working via umbrella company so please only apply for this role if you are willing to work on this basis and are resident in the UK.