Data Engineer – Complex Data Pipelines (On-Premise)

MARSS Group; Nice; Sur place; EUR 80 000 - 110 000; Plein temps

Résumé du poste

MARSS Group in Nice, France, is seeking a Data Engineer to join our AI team and build our in-house data platform and pipelines on-premises. You will design and implement data pipelines from the ground up, handling complex data types including images, video and time-series data, with offline-first, air-gapped infrastructure.

You will own data architecture, models and pipelines end-to-end, collaborating with MLOps, DevOps and software engineers to ensure reliability, scalability and data quality

Qualifications

• Strong professional experience in a closely related data engineering role.
• Designing and implementing data pipelines from zero (not cloud-based).
• Proficiency in Python and SQL.
• Experience with multiple data types (images, video, sensor data, time-series).
• Offline-first architectures and edge/infrastructure handling.
• Experience with bare metal or self-managed servers.
• Knowledge of data ingestion, transformation, storage, validation and quality principles.
• Experience with large/complex datasets.
• Strong Linux skills and network troubleshooting.
• Good software engineering practices: Git, testing, code review, documentation.

Responsabilités

• Design and build data pipelines from scratch, from ingestion through processing, transformation, storage and consumption.
• Design ingestion for offline/disconnected nodes with local buffering, resumable transfer and late data reconciliation.
• Define data prioritization with ML team during bandwidth windows.
• Handle multiple data types: structured/tabular data, images, video and time-series.
• Work with sensor data including timestamp sequencing and clock drift management.
• Develop robust Python/SQL processing solutions.
• Design data models and storage with capacity planning for large image/video data on self-managed storage.
• Own workflow definitions in the orchestration layer, including retry, idempotency and backfill behaviour.
• Build processes for ingestion, transformation, validation, quality control and traceability.
• Collaborate with MLOps, DevOps and Software Engineers to meet data requirements.
• Ensure pipelines are reliable, maintainable and scalable as volumes grow.
• Identify data-quality issues and define monitoring with the MLOps Engineer.
• Define overall architecture and standards for the in-house data platform.
• Document pipeline architecture, data flows and technical solutions.

Connaissances

Python

SQL

Data pipelines

Linux

Git

Testing

Documentation

Offline-first

Edge computing

Description du poste

Job Title: Data Engineer – Complex Data Pipelines (On-Premise)

Reports to: VP AI

Location: France - Nice

Type: Full-time

The Role

We are looking for an experienced Data Engineer to join our AI team and play a key role in building our in-house data platform and pipelines.

We are looking for someone with strong professional experience who has already designed and built data pipelines from the ground up, rather than only maintaining or using existing pipelines.

This platform runs entirely on our own infrastructure, no AWS, GCP or Azure, and no managed services to fall back on. Deployed systems are genuinely air-gapped: they operate offline for extended periods and connect only briefly, on request, with limited bandwidth. Deciding what data is worth those windows, and designing pipelines that behave correctly when nodes reconnect with large backlogs, is central to the role. Your assumptions about cloud-native workflows will be tested regularly.

Our data environment is complex and varied. We work not only with traditional structured/tabular data, but also with images, video and temporal/time-series data, including data generated by sensors and tracking systems where the sequence and timing of information are critical.

You will design the architecture required to ingest, process, transform, store and make these different data sources available to our AI and software teams.

This role works alongside an MLOps Data Engineer who owns the infrastructure, environments and deployment layer. You own the data itself: architecture, models, pipelines, quality, and prioritisation. Both roles report directly to the VP AI.

Main Responsibilities

• Design and build data pipelines from scratch, from data ingestion through processing, transformation, storage and consumption.
• Design ingestion for distributed recording nodes that are offline most of the time, including local buffering, resumable transfer, and reconciliation of late-arriving or out-of-order data on reconnection.
• Define, together with the ML team, which data is prioritised during short and bandwidth-limited connection windows.
• Design pipelines capable of handling multiple data types, including structured/tabular data, images, video and temporal/time-series data.
• Work with sensor-generated data, including sequential data where timestamps and the order of observations are important, and handle clock drift across nodes to ensure timing remains trustworthy downstream.
• Develop robust and scalable data processing solutions using Python and SQL.
• Design appropriate data models and storage approaches according to the nature and use of the data, including capacity planning and retention for large volumes of image and video data on self-managed storage.
• Own the workflow definitions in our orchestration layer: what runs, in what order, and with what retry, idempotency and backfill behaviour.
• Build processes for data ingestion, transformation, validation, quality control and traceability.
• Develop tools to support the preparation and availability of data for machine learning and AI applications.
• Work closely with Machine Learning Engineers, DevOps and Software Engineers to understand data requirements and provide appropriate solutions.
• Ensure data pipelines are reliable, maintainable and scalable as data volumes and use cases increase.
• Identify data-quality issues, including gaps and duplicates caused by node outages and retries, and define the data-quality and pipeline-health monitoring that the MLOps Engineer will implement in the deployment system.
• Define the overall architecture and technical standards for our in-house data platform.
• Document pipeline architecture, data flows and technical solutions.

Requirements

• Strong professional experience in a closely related data engineering role.
• Demonstrated experience designing and implementing data pipelines from zero (not cloud-based), including architectural and technical decisions.
• Good programming skills in Python & SQL
• Professional experience working with several different types of data
• Experience building pipelines involving at least some of the following: images, video, sensor data, time-series or other sequential data.
• Experience with systems that must tolerate unreliable or absent network connectivity and recover gracefully, offline-first, store-and-forward, edge collection or similar architectures.
• Experience running data infrastructure on bare metal or self-managed servers, rather than exclusively on managed cloud services.
• Good understanding of data ingestion, transformation, storage, validation and data-quality principles.
• Experience working with large or complex datasets.
• Strong Linux skills, including comfort with filesystems, storage, services and network troubleshooting.
• Good software engineering practices, including Git, testing, code review and documentation.
• Ability to independently investigate technical problems and propose appropriate architecture and solutions.