Data Engineer, ML Data Pipelines & Versioned Datasets

Blackshark; Zürich; On-site; CHF 90,000 - 150,000; Full time

Job summary

Harmattan AI is seeking a Data Engineer to own the data layer for autonomous defense ML models, working from Paris, Lausanne, or Zurich. You will turn raw field logs and public datasets into clean, versioned datasets for modelers.

You’ll shape data processing for deep-learning pipelines and enable efficient data delivery to training workloads. Ideal candidates have a STEM background, strong Python and data-engineering skills, and experience with unstructured data pipelines.

Qualifications

• Degree in a STEM field or equivalent practical experience.
• Experience building pipelines for unstructured data at scale (video, images, or sensor data).
• Strong Python skills and data-loader optimization for training frameworks (e.g. PyTorch).
• Bonus: multimodal sensor data, dataset versioning tooling, and distributed data processing tooling.

Responsibilities

• Ingest, decode, and store raw, unstructured field data (video and other sensor streams) from field logs into efficient, controlled formats.
• Multimodal alignment to make paired data usable for training.
• Curation: convert ingested and public data into high-value datasets with parsing, filtering, de-duplication and revision.
• Define data gathering and labeling requirements and own the dataset-construction workflow and tooling.
• Build task-specific datasets for training and evaluation in collaboration with acquisition, annotation and product teams.
• Version datasets and maintain lineage for reproducible training runs.
• Store data in training-ready formats and manage storage tiering to balance cost and latency.
• Deliver clean, documented datasets and tools for loading to keep training from being I/O-bound.

Skills

Python

Data engineering

Pipelines at scale

PyTorch

Education

STEM degree or equivalent

Job description

Harmattan AI is seeking a Data Engineer to own the data layer for autonomous defense ML models, working from Paris, Lausanne, or Zurich. You will turn raw field logs and public datasets into clean, versioned datasets for modelers.

You’ll shape data processing for deep-learning pipelines and enable efficient data delivery to training workloads. Ideal candidates have a STEM background, strong Python and data-engineering skills, and experience with unstructured data pipelines.