Senior Data Engineer - Azure Databricks

emagine; København; On-site; DKK 850,000 - 1,100,000; Full time

Job summary

emagine seeks a Senior Data Engineer to build and operate a data platform for regulated labeling and eLabel data in an Azure Databricks Lakehouse. You will develop scalable pipelines with a strong focus on quality, governance and traceability, handling Bronze→Silver→Gold architecture and metadata lineage.

Based in Greater Copenhagen, the role involves Azure storage, Databricks workflows, and CI/CD for notebooks, with emphasis on reliable data processing in a regulated environment. 37 hours/week.

Qualifications

• Hands-on Azure Databricks engineering experience.
• Strong data governance and quality focus.
• Experience in regulated environments.

Responsibilities

• Develop and maintain data pipelines using Azure Databricks, Python/PySpark, Spark SQL and Delta Lake.
• Build Bronze → Silver → Gold medallion architectures.
• Ingest data from APIs, JSON/files, Azure Storage and enterprise systems.
• Implement data quality, validation, reconciliation and automated testing.
• Handle schema enforcement/evolution, metadata and data lineage.
• Optimize pipeline performance, partitioning and scalability.
• Implement CI/CD, monitoring, error handling and reprocessing.
• Ensure traceability, auditability and reproducibility of regulated data pipelines.

Skills

Azure Databricks

Python/PySpark

Spark SQL

Delta Lake

Bronze → Silver → Gold

ADLS Gen2/Azure Storage

Databricks Workflows/Jobs

Unity Catalog & governance

API data ingestion

CI/CD for Databricks

Metadata & data lineage

Delta Live Tables

Lakeflow (advantage)

Tools

Delta Live Tables

Lakeflow

Job description

On behalf of a client in the pharmaceutical industry, we are looking for a Senior Data Engineer to support the development and operation of a data platform for regulated labeling and eLabel data.

The consultant will build scalable data pipelines and data products in an Azure Databricks Lakehouse environment, with a strong focus on data quality, governance, traceability and reliable data processing.

Main Responsibilities

• Develop and maintain data pipelines using Azure Databricks, Python/PySpark, Spark SQL and Delta Lake
• Build solutions following Bronze → Silver → Gold medallion architecture
• Ingest and process data from APIs, JSON/files, Azure Storage and enterprise systems
• Implement data quality, validation, reconciliation and automated testing
• Handle schema enforcement/evolution, metadata and data lineage
• Optimise pipeline performance, partitioning and scalability
• Implement CI/CD, monitoring, error handling and reprocessing
• Ensure traceability, auditability and reproducibility of regulated data pipelines

Key Requirements

• Strong hands-on Azure Databricks engineering experience
• Strong Python/PySpark, Spark SQL and Delta Lake
• Deep experience with Bronze → Silver → Gold medallion architecture
• ADLS Gen2 / Azure Storage
• Databricks Workflows / Jobs
• Unity Catalog and data governance
• API-based data ingestion and export
• Data quality, validation and automated testing
• CI/CD for Databricks notebooks/code
• Experience building data pipelines in regulated environments
• Experience with metadata and references to document/PDF assets
• Delta Live Tables / Lakeflow is an advantage

Other Details

• Location: Greater Copenhagen
• Duration: 1/10/2026 - 31/03/2026
• Allocation: 37 hours/week