Data Engineer, High-Performance ML Data Pipelines

Harmattan AI; Paris; Sur place; EUR 85 000 - 120 000; Plein temps

Résumé du poste

Harmattan AI seeks a Data Engineer for the Foundational team in Paris to build massive data infrastructure powering our autonomous defense models. You will manage terabytes to petabytes of EO/IR video, focusing on scalable ingestion, synchronization, and high-throughput loading to keep GPUs fed for training.

You’ll implement distributed processing with Spark/Ray/Beam, ensure data quality and reproducibility, and evaluate storage solutions like MinIO or S3.

Qualifications

• BS or MS in Computer Science, Software Engineering, or Distributed Systems.
• Deep knowledge of operating systems, networking, and parallel computing.
• 5–6+ years building and maintaining terabyte-scale pipelines for unstructured data.

Responsabilités

• Multi-Modal Ingestion Pipeline: build ETL/ELT pipelines to extract, decode, and store EO/IR video into WebDataset, TFRecords, or Parquet.
• Sensor Synchronisation: align EO and IR frames temporally and spatially for model training.
• High-Throughput Data Loading: storage-to-GPU pipelines to sustain >90% GPU utilisation.
• Distributed Processing: develop jobs with Spark, Ray, or Beam for thousands of hours of video.
• Data Quality & Versioning: implement automated quality checks and reproducibility via versioning.
• Infrastructure Evaluation: assess storage solutions (MinIO, S3 tiering) for cost and latency.

Connaissances

Terabyte pipelines

Distributed systems

PyTorch

JAX

Spark

Ray

Beam

DVC

Iceberg

Pachyderm

Formation

BS/MS in CS/SE/DS

OS knowledge

Networking fundamentals

Parallel computing

Outils

Spark

Ray

Beam

MinIO

S3

WebDataset

TFRecords

Parquet

PyTorch

JAX

Description du poste

Harmattan AI seeks a Data Engineer for the Foundational team in Paris to build massive data infrastructure powering our autonomous defense models. You will manage terabytes to petabytes of EO/IR video, focusing on scalable ingestion, synchronization, and high-throughput loading to keep GPUs fed for training.

You’ll implement distributed processing with Spark/Ray/Beam, ensure data quality and reproducibility, and evaluate storage solutions like MinIO or S3.