Data Engineer: High-Throughput ML Data Pipelines

Harmattan AI; Paris; On-site; EUR 70,000 - 110,000; Full time

Job summary

Harmattan AI in Paris seeks a Data Engineer on the Foundational team to build the data infra powering our autonomous defense models. You will handle terabytes of EO/IR video, implement robust ingestion pipelines and ensure data is ready for ML engineers.

You will collaborate across teams to optimise GPU utilisation, implement versioning and data lineage, and evaluate storage solutions like MinIO or S3. This role combines systems engineering with high-impact, mission-driven technology in a

Qualifications

• Educational background: BS/MS in Computer Science, Software Engineering, or Distributed Systems.
• 2,
• 5-6+ years building and maintaining terabyte-scale pipelines for unstructured data (video, images, or point clouds).
• Proven track record of maximising multi-node GPU utilisation and optimising data loaders for PyTorch or JAX.
• Expertise with distributed computing tools (Spark, Ray, Beam) and ML data versioning tools (DVC, Iceberg, Pachyderm).
• Ability to work in a fast-paced startup and automate messy data problems.

Responsibilities

• Multi-Modal Ingestion Pipeline: Build ETL/ELT pipelines to extract, decode, and store raw EO and IR video from field logs into highly optimised formats like WebDataset, TFRecords, or Parquet.

Skills

Terabyte-scale pipelines

Distributed systems

GPU utilisation

Data loading optimization

PyTorch/JAX experience

Spark/Ray/Beam

Data versioning/ lineage

Education

BS/MS in CS/SE/Distributed Systems

Tools

Spark

Ray

Beam

DVC

Iceberg

Pachyderm

MinIO

S3

WebDataset

TFRecords

Parquet

PyTorch

JAX

Job description

Harmattan AI in Paris seeks a Data Engineer on the Foundational team to build the data infra powering our autonomous defense models. You will handle terabytes of EO/IR video, implement robust ingestion pipelines and ensure data is ready for ML engineers.

You will collaborate across teams to optimise GPU utilisation, implement versioning and data lineage, and evaluate storage solutions like MinIO or S3. This role combines systems engineering with high-impact, mission-driven technology in a