Azure Databricks Data Engineer - Medallion Lakehouse

EPM Scientific; København; In loco; DKK 520.000 - 900.000; Tempo pieno

Descrizione del lavoro

EPM Scientific is seeking an experienced Data Engineer to design, build, and optimize data platforms in Copenhagen. You will work with Azure Databricks, PySpark, Spark SQL, and Delta Lake to deliver scalable data products using a Medallion architecture.

You will implement ingestion frameworks, manage ADLS Gen2 storage, and support CI/CD for data notebooks and pipelines, ensuring data quality, governance, and regulatory compliance across environments.

Competenze

• Strong hands-on experience with Azure Databricks, Python/PySpark, Spark SQL, and Delta Lake.
• Deep understanding of modern data lakehouse architectures, including Bronze-Silver-Gold (Medallion) design.
• Experience developing enterprise-scale ETL/ELT pipelines and data integration solutions.
• Strong knowledge of Azure storage technologies, particularly ADLS Gen2.
• Experience implementing data governance, lineage, and access control frameworks.
• Proven experience with data quality, validation, monitoring, and operational support.
• Experience in regulated environments with compliance and traceability requirements.

Mansioni

• Design, develop, and optimize data solutions using Azure Databricks, Python/PySpark, Spark SQL, and Delta Lake.
• Develop and maintain data pipelines following the Medallion Architecture to deliver scalable, trusted data products.
• Implement ingestion frameworks for structured and semi-structured data from APIs and files.
• Manage and optimize data storage solutions using ADLS Gen2 and related Azure services.
• Build, schedule, and support workloads using Databricks Workflows/Jobs with Delta Live Tables or Lakeflow as a plus.
• Develop and maintain data quality controls, validation frameworks, and automated testing.

Conoscenze

Azure Databricks

Python/PySpark

Spark SQL

Delta Lake

Medallion Architecture

Data governance

ETL/ELT pipelines

ADLS Gen2

DevOps CI/CD for data

Strumenti

Delta Lake

Unity Catalog

Delta Live Tables

Lakeflow

Descrizione del lavoro

EPM Scientific is seeking an experienced Data Engineer to design, build, and optimize data platforms in Copenhagen. You will work with Azure Databricks, PySpark, Spark SQL, and Delta Lake to deliver scalable data products using a Medallion architecture.

You will implement ingestion frameworks, manage ADLS Gen2 storage, and support CI/CD for data notebooks and pipelines, ensuring data quality, governance, and regulatory compliance across environments.