Senior Databricks Data Engineer: Azure Lakehouse&Pipelines

Luxoft Poland; Poland; On-site; PLN 180,000 - 280,000; Full time

Benefits offered by this job

Internal Mobility program

Private medical & dental care & life保险

Job summary

Luxoft Poland is seeking a Senior Data Engineer to develop and operate Databricks/Spark pipelines and Delta Lake lakehouse layers on Azure. You will be accountable for pipeline reliability, data freshness and dataset quality against SLAs.

The role requires 7+ years in data engineering, with 3+ on Databricks and 2+ on Azure data services. Responsibilities include design of lakehouse layers, orchestration with ADF/Airflow, and ensuring data governance and lineage.

Qualifications

• Bachelor's degree or equivalent practical experience in computer science, engineering or related field.
• 7+ years in data engineering, with 3+ years on Databricks / Apache Spark and 2+ on Azure data services (ADF, ADLS, Delta Lake).
• Expert in Databricks (Workflows, Delta Live Tables, Unity Catalog) and Spark (PySpark, Spark SQL).
• Proficient in Azure data services (ADF, Data Lake Gen2, Key Vault, Synapse/SQL DB) and Azure DevOps CI/CD.
• Python and SQL at expert level; data modelling (dimensional, data vault) and ELT design.
• Orchestration (ADF, Airflow), data-quality frameworks (Great Expectations, DLT expectations) and monitoring.

Responsibilities

• Develop and operate Databricks / Spark pipelines and Delta Lake lakehouse layers on Azure.
• Design bronze–silver–gold lakehouse layers with partitioning and Unity Catalog governance.
• Build ETL/ELT jobs and orchestration with ADF and Airflow; manage dependencies and retries.
• Implement data-quality checks and validation; monitor data freshness and dataset quality against SLAs.
• Tune Spark performance and control costs (autoscaling, Photon, job clusters, spot).
• Manage schema evolution and change control; document lineage and transformations.

Skills

Databricks / Spark

Python

SQL

Azure

Delta Lake

Unity Catalog

Airflow

Azure Data Factory

Spark tuning

Data modelling

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Azure Data Factory

Azure Data Lake Storage Gen2

Delta Live Tables

Airflow

Unity Catalog

Azure DevOps CI/CD

Job description

Luxoft Poland is seeking a Senior Data Engineer to develop and operate Databricks/Spark pipelines and Delta Lake lakehouse layers on Azure. You will be accountable for pipeline reliability, data freshness and dataset quality against SLAs.

The role requires 7+ years in data engineering, with 3+ on Databricks and 2+ on Azure data services. Responsibilities include design of lakehouse layers, orchestration with ADF/Airflow, and ensuring data governance and lineage.