Foundational Data Engineer - High-Volume Video Pipelines

Harmattan AI; Paris; On-site; EUR 70,000 - 90,000; Full time

Job summary

A next-generation defense technology company based in Paris is seeking a Data Engineer to build and manage high-performance data infrastructures. The role involves creating pipelines for unstructured video data, optimizing GPU utilization, and ensuring data quality for ML applications. Ideal candidates should have 5-6+ years of experience with large datasets and strong skills in distributed computing. Join us to contribute to innovative solutions in autonomous defense systems.

Qualifications

• 5-6+ years of experience with unstructured data processing.
• Deep knowledge of operating systems, networking, and parallel computing.
• Strong command of ML data versioning tools.

Responsibilities

• Build ETL/ELT pipelines for raw video data.
• Develop algorithms to synchronize EO and IR frames.
• Architect storage-to-GPU pipelines to prevent I/O bottlenecks.
• Write distributed data processing jobs for tactical video logs.
• Implement automated quality checks and versioning.
• Evaluate storage solutions to manage growing datasets.

Skills

Building and maintaining terabyte-scale pipelines

Maximizing multi-node GPU utilization

Distributed computing tools (Spark, Ray, Beam)

Automating data quality checks

Education

BS or MS in Computer Science, Software Engineering, or Distributed Systems

Tools

Apache Spark

Ray

Apache Beam

PyTorch

JAX

DVC

Apache Iceberg

Pachyderm

Job description

A next-generation defense technology company based in Paris is seeking a Data Engineer to build and manage high-performance data infrastructures. The role involves creating pipelines for unstructured video data, optimizing GPU utilization, and ensuring data quality for ML applications. Ideal candidates should have 5-6+ years of experience with large datasets and strong skills in distributed computing. Join us to contribute to innovative solutions in autonomous defense systems.