Azure Databricks Data Engineer — Remote/Hybrid Pipelines

Andersen; United States; Hybrid; USD 78,000 - 123,000; Full time

Benefits offered by this job

Private health insurance

Sports compensation

Mentoring and adaptation program

Training portal access

Referral program

Job summary

Andersen is seeking a Data Engineer (Azure Databricks) for a EU-based project to enhance a cloud-native data platform for drug development. You will build scalable data pipelines, enforce data governance, and collaborate with scientists and business stakeholders across markets.

The role offers remote or hybrid work options within Europe, with exposure to MLOps, databricks, dbt, and advanced analytics in a regulated life sciences environment.

Qualifications

• Commercial experience in Data Engineering for 4+ years.
• Strong hands-on experience with Azure Databricks, including Spark and Delta Lake.
• Experience building and maintaining multi-layer data pipelines following the Landing → Raw → Enriched → Curated architecture.
• Experience with schema enforcement and schema-drift handling.
• Ability to implement data quality checks at ingestion.
• Experience orchestrating pipelines using Databricks Jobs / Workflows, including dependencies, retries, alerts, and failure handling.
• Experience with both batch and API-based data ingestion.
• Strong hands-on experience with dbt.
• Ability to monitor job failures, data freshness, and overall pipeline health.
• Experience delivering data in line with defined SLAs.

Responsibilities

• Building and maintaining multi-layer pipelines (Landing → Raw → Enriched → Curated) on Azure Databricks (Spark, Delta Lake).
• Orchestrating pipelines via Databricks Jobs / Workflows: dependencies, retries, alerting.
• Implementing batch and API-based data ingestion from multiple source systems.
• Applying schema enforcement and handle schema drift; run data quality checks at ingestion.
• Working hands-on with dbt for pipeline observability and data governance tasks.
• Monitoring job failures, data freshness, and pipeline health against SLA-based delivery targets.
• Setting up alerts, logging, and retry strategies for production pipelines.
• Working within a controlled Dev/Test/Prod environment: access control, audit trails, pipeline documentation, traceability to requirements.
• Contributing to CI/CD and MLOps/AI pipelines on Databricks where the project needs it.

Skills

Azure Databricks

Spark

Delta Lake

Data pipelines

Schema enforcement

Data quality checks

dbt

APIs ingestion

Databricks Jobs

Data governance

CI/CD

MLOps

Audit trails

Access control

Data lineage

GxP compliance

English proficiency

Tools

Databricks

SAP S/4HANA

Microsoft 365

Azure

SharePoint

Git

CI/CD pipelines

Job description

Andersen is seeking a Data Engineer (Azure Databricks) for a EU-based project to enhance a cloud-native data platform for drug development. You will build scalable data pipelines, enforce data governance, and collaborate with scientists and business stakeholders across markets.

The role offers remote or hybrid work options within Europe, with exposure to MLOps, databricks, dbt, and advanced analytics in a regulated life sciences environment.