Data Engineer, ML Data Pipelines & High-Throughput Video

Harmattan AI; Paris; On-site; EUR 90,000 - 150,000; Full time

Job summary

Harmattan AI, a Paris-based defense technology company, seeks a Data Engineer to build massive data pipelines for multimodal video data. You will optimize ingestion, synchronization, and loading to accelerate ML workflows and sustain high GPU utilization in multi-node clusters.

Based in Paris, you will work on terabyte- to petabyte-scale datasets, deploy scalable storage solutions, and collaborate with ML engineers to design robust, reproducible data architectures for mission-critical models.

Qualifications

• BS or MS in Computer Science, Software Engineering, or Distributed Systems preferred.
• Deep knowledge of operating systems, networking, and parallel computing is essential.
• 5–6+ years of experience building and maintaining terabyte-scale pipelines for unstructured data (video, images, or point clouds).

Responsibilities

• Build ETL/ELT pipelines to extract, decode, and store EO/IR video into formats like WebDataset, TFRecords, or Parquet.
• Develop algorithms to synchronise EO and IR frames temporally and spatially for model training.
• Architect storage-to-GPU pipelines to maintain >90% GPU utilisation in multi-node clusters.
• Write distributed data processing jobs using Spark, Ray, or Beam to process thousands of hours of tactical video logs.
• Implement automated quality checks to filter corrupted frames and ensure reproducible training via versioning and lineage tracking.
• Assess and implement storage solutions (MinIO, S3 tiering) to manage growing datasets cost-effectively.

Skills

Distributed systems

Data pipelines

Performance optimization

Ownership mindset

Terabyte pipelines

Education

BS or MS in Computer Science / Distributed Systems

Tools

Spark

Ray

Beam

DVC

Apache Iceberg

Pachyderm

MinIO

Parquet

Job description

Harmattan AI, a Paris-based defense technology company, seeks a Data Engineer to build massive data pipelines for multimodal video data. You will optimize ingestion, synchronization, and loading to accelerate ML workflows and sustain high GPU utilization in multi-node clusters.

Based in Paris, you will work on terabyte- to petabyte-scale datasets, deploy scalable storage solutions, and collaborate with ML engineers to design robust, reproducible data architectures for mission-critical models.