Data platform engineer for AI research

European Tech Recruit; Berlin; On-site; EUR 90,000 - 130,000; Full time

Job summary

European Tech Recruit ищет старшего инженера по данным для разработки и эксплуатации распределённой инфраструктуры обработки больших данных и ML-нагрузок в Берлине. Ваша работа включает проектирование конвейеров, обеспечение воспроизводимости и устойчивости систем.

Требуется PhD и 3+ года опыта, знание Ray/Spark/Flink, Kubernetes/Slurm, Docker/Enroot и форматов Arrow/Parquet. Роль обеспечивает масштабируемость, контроль затрат и надёжность.

Qualifications

• PhD in Computer Science, Computer Engineering, Software Engineering or related discipline.
• At least 3 years of full-time experience building or operating distributed data-processing systems or ML data infrastructure.
• Strong experience with distributed processing frameworks (Ray, Spark, Flink) and workflow orchestration.

Responsibilities

• Проектирование, сборка и эксплуатация распределённой инфраструктуры обработки данных для петабайтовых нагрузок.
• Разработка и оптимизация конвейеров данных, оркестрации рабочих процессов и графов выполнения.
• Повышение производительности, пропускной способности и эффективности использования ресурсов, включая управление стоимостью.
• Создание надёжных конвейеров с мониторингом и обработкой сбоев.
• Внедрение версионирования наборов данных, трассировки и воспроизводимости.
• Интеграция инфраструктуры данных с крупномасшётными ML-обучающими пайплайнами.

Skills

Ray

Apache Spark

Apache Flink

Kubernetes

Slurm

Docker

Enroot

Arrow

Parquet

Education

PhD in Computer Science or related field

Tools

Ray

Apache Spark

Apache Flink

Job description

Описание

Join a highly technical AI research and engineering organisation building the data infrastructure behind next-generation AI models. As a Senior Data Platform Engineer, you’ll design and operate distributed data-processing systems capable of handling petabyte-scale workloads, ensuring research teams can access reliable, reproducible datasets and run complex pipelines efficiently. This role combines distributed systems engineering, workflow orchestration and performance optimisation, with a focus on scalability, fault tolerance and cost efficiency across large-scale data and machine learning infrastructure.

Задачи

• Design, build and operate distributed data-processing infrastructure for petabyte-scale workloads.
• Develop and optimise data pipelines, execution frameworks and workflow orchestration across diverse computational workloads, including GPU-accelerated stages.
• Improve system performance, throughput and resource utilisation while managing infrastructure costs.
• Build reliable, observable pipelines with robust failure recovery, restartability and monitoring.
• Implement dataset versioning, lineage and reproducibility to ensure research datasets can be reliably traced and recreated.
• Integrate data infrastructure with large-scale ML training workflows and establish engineering standards and best practices.

Требования

• PhD in Computer Science, Computer Engineering, Software Engineering or a related discipline.
• At least 3 years of relevant full-time experience building or operating distributed data-processing systems or ML data infrastructure.
• Strong experience with distributed processing frameworks such as Ray, Apache Spark or Flink, alongside workflow orchestration and large-scale pipeline execution.
• Solid understanding of distributed computing, performance profiling, throughput optimisation and fault tolerance.
• Experience with high-performance data formats such as Apache Arrow or Parquet, and dataset versioning, lineage or reproducibility.
• Familiarity with workload orchestration and scheduling tools such as Kubernetes or Slurm, and containerisation technologies such as Docker or Enroot.
• Strong software engineering skills and experience owning production systems where reliability, performance and cost matter.

Условия

Help build the infrastructure that enables cutting-edge AI research at scale. You’ll tackle challenging distributed-systems problems, influence core architectural decisions and work closely with researchers and engineers to make large-scale data and ML workflows more efficient, reliable and reproducible.