Data Engineer - Foundational

Harmattan AI

Paris

On-site

EUR 90,000 - 140,000

Full time

47 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Harmattan AI is seeking a Data Engineer on the Foundational team in Paris to build massive data infrastructure for deep learning models. You will manage terabytes—soon petabytes—of EO and IR video data, ensuring data pipelines are robust and performant so ML engineers can focus on model design.

You will implement high-throughput loading, synchronization of EO/IR frames, and scalable processing using Spark, Ray, or Beam, while optimizing storage and data quality.

Qualifications

  • BS or MS in CS, SE, or distributed systems preferred.
  • 5–6+ years building terabyte-scale pipelines for unstructured data.
  • Proven track record in maximizing multi-node GPU utilisation.

Responsibilities

  • Build ETL/ELT pipelines to ingest EO/IR video into optimized formats.
  • Develop algorithms for EO/IR frame synchronization.
  • Architect storage-to-GPU pipelines for multi-node training clusters.
  • Write distributed data processing jobs with Spark, Ray, or Beam.
  • Implement automated data quality checks and reproducible training with versioning.
  • Evaluate storage solutions (MinIO, S3 tiering) for cost and latency.

Skills

Distributed systems
Parallel computing
GPU utilisation
Data engineering

Education

BS/MS in Computer Science or Distributed Systems

Tools

Spark
Ray
Beam
DVC
Iceberg
Pachyderm
MinIO
S3

Job description

About Us

Harmattan AI is a next-generation defense prime building autonomous and scalable defense systems. Following the close of a $200M Series B, valuing the company at $1.4 billion, we are expanding our teams and capabilities to deliver mission-critical systems to allied forces.



About Us

Harmattan AI is a next-generation defense prime building autonomous and scalable defense systems. Following the close of a $200M Series B, valuing the company at $1.4 billion, we are expanding our teams and capabilities to deliver mission-critical systems to allied forces. Our work is guided by clear values: building technologies with real-world impact, pursuing excellence in everything we do, setting ambitious goals, and taking on the hardest technical challenges. We operate in a demanding environment where rigor, ownership, and execution are expected.



About The Role

As a Data Engineer on the Foundational team, you will serve as the "plumber" for deep learning, building the massive, high-performance data infrastructure required to power our foundational models. Based in Paris, you will manage terabytes—and eventually petabytes—of raw, unstructured, and noisy video data (EO and IR). Your mission is to ensure our ML engineers spend their time designing architectures, not waiting for data loaders or wrangling corrupted files.



Responsibilities


  • Multi-Modal Ingestion Pipeline: Build ETL/ELT pipelines to extract, decode, and store raw Electro-Optical (EO) and Infrared (IR) video from field logs into highly optimised formats like WebDataset, TFRecords, or Parquet.

  • Sensor Synchronisation & Alignment: Develop algorithms to programmatically synchronise EO and IR frames temporally and spatially to provide paired inputs for model training.

  • High-Throughput Data Loading: Architect storage-to-GPU pipelines to ensure multi-node training clusters maintain >90% GPU utilisation without I/O bottlenecks.

  • Distributed Processing: Write and optimise distributed data processing jobs using tools like Apache Spark, Ray, or Apache Beam to process thousands of hours of tactical video logs.

  • Data Quality & Versioning: Implement automated quality checks to filter corrupted or blank frames and maintain 100% reproducible training runs through robust versioning and lineage tracking.

  • Infrastructure Evaluation: Assess and implement advanced storage solutions (e.g., MinIO, S3 tiering) to manage growing datasets while optimising for cost and latency.



Candidate Requirements


  • Educational Background: A BS or MS in Computer Science, Software Engineering, or Distributed Systems is highly preferred. Deep knowledge of operating systems, networking, and parallel computing is essential.

  • Technical Experience: 5-6+ years of experience building and maintaining terabyte-scale pipelines for unstructured data (video, images, or point clouds).

  • Performance Optimisation: Proven track record of maximising multi-node GPU utilisation and optimising data loaders for frameworks like PyTorch or JAX.

  • Tooling Expertise: Strong command of distributed computing tools (Spark, Ray, Beam) and ML data versioning tools (DVC, Apache Iceberg, or Pachyderm).

  • Adaptability & Ownership: A systems-thinker who thrives in a fast-paced startup environment and views messy data as an engineering problem to be solved via automation.

  • Commitment: 100% dedication to Harmattan AI’s mission of providing a defensive edge to allied nations through ethical, high-impact technology



We look forward to hearing how you can help shape the future of autonomous defense systems at Harmattan AI.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Foundational Data Engineer - High-Volume Video Pipelines
Foundational Data Engineer - High-Volume Video Pipelines

Harmattan AI • Paris

On-site
EUR 70,000 - 90,000
Data Engineer - Foundational
Data Engineer - Foundational

Harmattan AI • Paris

On-site
EUR 70,000 - 90,000
Data Engineer: Build Multimodal, Training-Ready Datasets
Data Engineer: Build Multimodal, Training-Ready Datasets

Harmattan AI • Paris

On-site
EUR 70,000 - 95,000
Data Engineer (Detect & Track Distillation)
Data Engineer (Detect & Track Distillation)

Signal Processing Engineer (RF Processing & Analytics) • Paris

On-site
EUR 60,000 - 90,000
Data Engineer (Detect & Track Distillation)
Data Engineer (Detect & Track Distillation)

Harmattan AI • Paris

On-site
EUR 70,000 - 95,000
Foundational ML Engineer for Multi-Modal Edge AI
Foundational ML Engineer for Multi-Modal Edge AI

Harmattan AI • Paris

On-site
EUR 70,000 - 90,000
Machine Learning Engineer - Foundational
Machine Learning Engineer - Foundational

Harmattan AI • Paris

On-site
EUR 70,000 - 90,000
Machine Learning Engineer (Detect & Track Distillation)
Machine Learning Engineer (Detect & Track Distillation)

Harmattan AI • Paris

On-site
EUR 90,000 - 140,000
Machine Learning Engineer - Foundational
Machine Learning Engineer - Foundational

Harmattan AI • Paris

On-site
EUR 80,000 - 120,000
Computer Vision Engineer
Computer Vision Engineer

Harmattan AI • France

On-site
EUR 75,000 - 110,000