Senior Data Platform Engineer

European Tech Recruit; Berlin; On-site; EUR 110,000 - 140,000; Full time

Job summary

Unknown company is seeking a Senior Data Platform Engineer to architect, build, scale, and maintain data infrastructure powering frontier AI models in a Berlin setting. You will treat the data platform as an internal product, serving research teams as customers.

You will design scalable pipelines, ensure reproducibility, and collaborate with Research and Product teams to evaluate new technologies that improve performance, reliability and cost efficiency.

Qualifications

• 5+ years designing and managing large-scale distributed data processing systems.
• Experience with Ray, Spark, Arrow, and Parquet in real workloads.
• Proven ability to optimize throughput and manage cost under pressure.
• Experience with dataset versioning, lineage, and reproducibility tooling.
• Collaborates across Research and Engineering, documenting best practices.

Responsibilities

• Scale and automate petabyte-scale data processing infrastructure.
• Design execution layers for pipeline stages with diverse compute needs.
• Optimise compute resources, including GPU-accelerated tasks.
• Build reliable pipelines with observability and restartability.
• Diagnose performance bottlenecks and attribute costs to stages.
• Establish dataset versioning, lineage and reproducibility across lifecycle.
• Ensure outputs are reproducible for evolving research needs.
• Document data specifications and governance requirements.
• Partner with Research and Product teams to integrate with training pipelines.
• Evaluate new technologies to improve scalability and cost efficiency.

Skills

Ray

Apache Spark

Workflow orchestration

Apache Arrow

Parquet

Kubernetes

Docker

Enroot

GPU computing

Data governance

Education

Bachelor's degree in CS/engineering

Tools

Ray

Apache Spark

Slurm

Docker

Enroot

Parquet

Apache Arrow

Job description

Our client is a non-profit organization committed to advancing research and creating technical solutions that enable safe-by-design AI systems.

They are seeking a highly technical Senior Data Platform Engineer to architect, build, scale and maintain the data infrastructure powering next-generation frontier AI models.

In this high-impact role, you will bridge the gap between cutting-edge AI research and high-performance engineering, treating the data platform as an internal product with research teams as your primary customers.

You will be responsible for building automated, petabyte-scale data processing pipelines and ensuring they are efficient, reliable, reproducible and fully traceable.

Their technical environment will continue to evolve as research and engineering requirements scale. They are looking for someone who can help shape that environment — challenging existing approaches, evaluating emerging technologies and making sustainable technical decisions in close collaboration with Research and Product teams.

Key Responsibilities

• Scale and automate data processing infrastructure to handle petabytes of data while ensuring reliable day-to-day operation.
• Design execution layers for pipeline stages with different computational requirements, selecting appropriate processing engines and maintaining efficient end-to-end throughput.
• Optimise the use of compute resources across large-scale workloads, including GPU resources for compute-intensive data processing tasks.
• Build reliable pipelines with graceful failure recovery, restartability and comprehensive observability.
• Identify and diagnose performance bottlenecks, attributing compute usage and costs to individual pipeline stages.
• Establish robust systems for dataset versioning, lineage, reproducibility and traceability across all stages of the data processing lifecycle.
• Ensure intermediate outputs and final datasets can be reliably reproduced for specific and evolving research requirements.
• Develop and maintain appropriate documentation and data specifications in line with internal data governance requirements.
• Partner closely with Research and Engineering teams to ensure datasets integrate seamlessly with model training pipelines.
• Evaluate and introduce new technologies and approaches where they can improve scalability, reliability, performance or cost efficiency.

Skills & Qualifications

• A bachelor's degree in a relevant field (e.g., computer science, computer engineering, software engineering) is required.
• 5+ years of experience designing, implementing, and managing large-scale distributed data processing systems, or working within large-scale distributed ML data frameworks, with recent experience using e.g. Ray, Apache Spark, workflow orchestrators, Apache Arrow, and/or Parquet.
• Demonstrated ownership of a data processing system under real throughput, reliability, and cost pressure. The specific frameworks matter less to us than evidence that you have had to reason about where a large pipeline breaks and why.
• Experience profiling and optimizing throughput and cost across heterogeneous workloads, including GPU-accelerated stages.
• Experience with dataset versioning, lineage, and reproducibility tooling.
• Ability to collaborate effectively with cross-functional teams, document best practices, and stay updated with the latest advancements in large-scale data processing and software development.
• Experience with workload managers (e.g., Ray, Kubernetes, Slurm).
• Familiarity with containerization tools (e.g., Docker, Enroot).
• Familiarity with data infrastructures and platforms (e.g., vector databases).