Lead Data Engineer

Aether Biomedical; Deutschland; Vor Ort; EUR 90.000 - 130.000; Vollzeit

Benefits dieser Stelle

Vacation days

Illness days

Health insurance

Learning platforms

Language classes

Flexible workplace

Public speaking opportunities

Bench time

Zusammenfassung

Aether Biomedical is seeking a Lead Data Engineer to own end-to-end delivery of an AWS Lakehouse platform. The role is hands-on (60–70% engineering) with 30–40% technical leadership, defining architecture, standards, and build-vs-buy decisions while mentoring senior engineers.

You will drive data product decoupling, ensure reliability, and collaborate with product management to translate requirements into scalable data pipelines across batch and streaming workloads.

Qualifikationen

• 8+ years in data engineering with 2+ years in technical leadership.
• Hands-on with AWS data services: S3, Glue, Redshift, Athena, MSK Kafka, EMR Spark, Lake Formation, IAM.
• Terraform IaC and CI/CD pipelines experience.
• Python and SQL for building data pipelines and data models.
• Experience with Iceberg, Delta Lake, or Hudi.
• Experience designing CDC and real-time streaming pipelines with Kafka, Flink, Spark Structured Streaming.
• End-to-end data platform delivery or large-scale data migrations.
• Migration of on-premises data systems to cloud environments.

Aufgaben

• Design and lead the implementation of an AWS native lakehouse platform using Medallion architecture with S3 and Iceberg.
• Define and maintain Infrastructure as Code using Terraform, including environment setup and CI/CD pipelines.
• Design and implement Change Data Capture and streaming ingestion frameworks for migrating legacy data workloads.
• Establish engineering standards including code reviews, automated testing, data quality validation, and reusable pipeline design.
• Evaluate AWS services and align architecture with Well Architected principles.
• Collaborate with product management to break down data products and guide engineering execution.
• Mentor data engineers and support knowledge transfer to build internal capabilities.
• Ensure platform observability, monitoring, and reliability across the data ecosystem.

Kenntnisse

AWS

Python

SQL

Data modeling

Data pipelines

Mentoring

Communication

Kafka

Tools

Terraform

CI/CD

Iceberg

Delta Lake

Jobbeschreibung

Client

The client is a major player within the high-volume, regulated iGaming and betting sector, operating across multiple international markets. They focus on delivering data-driven solutions and high-performance digital entertainment experiences.

Project overview

This initiative involves a major cloud migration and modernization effort, shifting legacy on-premises data infrastructure (including Kafka, NiFi, ClickHouse, and PostgreSQL workloads) into a modern, native AWS Lakehouse environment. The project is executed under a structured program governance model using a distributed, multi-market team delivery approach.

Position overview

We are looking for a Lead Data Engineer to take ownership of end to end technical delivery of an AWS Lakehouse platform. This is a hands-on leadership role split approximately 60–70% engineering and 30–40% technical leadership. The successful candidate will define the platform's technical architecture, set engineering standards, make critical build-vs-buy decisions, and actively mentor senior and mid-level engineers within a shared backlog environment.

Responsibilities

• Design and lead the implementation of an AWS native lakehouse platform using Medallion architecture with S3 and Iceberg
• Define and maintain Infrastructure as Code using Terraform, including environment setup and CI CD pipelines
• Design and implement Change Data Capture and streaming ingestion frameworks for migrating legacy data workloads
• Establish engineering standards including code reviews, automated testing, data quality validation, and reusable pipeline design
• Evaluate AWS services and align architecture with Well Architected principles
• Collaborate with product management to break down data products and guide engineering execution
• Mentor data engineers and support knowledge transfer to build internal capabilities
• Ensure platform observability, monitoring, and reliability across the data ecosystem

Requirements

• 8+ years of experience in data engineering including at least 2 years in a technical leadership role
• Hands on experience with AWS data services including S3, Glue, Redshift, Athena, MSK Kafka, EMR Spark, Lake Formation, IAM
• Strong experience with Infrastructure as Code using Terraform and implementing CI CD pipelines
• Proficiency in Python and SQL for building data pipelines and data models across batch and streaming workloads
• Experience with open table formats such as Apache Iceberg, Delta Lake, or Apache Hudi
• Experience designing and implementing Change Data Capture and real time streaming pipelines using Kafka, Flink, or Spark Structured Streaming
• Experience delivering end to end data platforms or leading large scale data migrations
• Experience migrating on premises data systems to cloud environments
• Ability to communicate technical decisions clearly to both engineering and business stakeholders
• Experience mentoring engineers and supporting team capability development

Nice to have

• Experience migrating legacy platforms such as ClickHouse, Apache NiFi, or PostgreSQL
• AWS certifications such as Data Engineer or Solutions Architect
• Experience working in regulated industries with high volume real time data processing
• Exposure to data governance tools such as AWS Lake Formation, AWS Glue Data Catalog, or Amazon Macie
• Experience with business intelligence tools such as Tableau

What We Offer:

• Vacation days: Up to 26 business days per year.
• 10 illness/special days off per year (fully paid, no medical papers needed) for all contract types
• Health and life insurance (Luxmed)
• MyBenefit platform with Multisport option
• Internal psychological support service
• English language classes from the first working day
• Access to external learning platforms: O’Reilly, LinkedIn Learning, Udemy, and a wide catalog of diverse internal training
• Flexible workplace: work from the office, from home, or choose a hybrid option
• Opportunities to develop as a public speaker, mentor, or technical interviewer
• Fully paid idle (bench) when not involved in a project