Senior Databricks Data Engineer on Azure: Lakehouse

Luxoft Poland; Poland; On-site; PLN 180,000 - 320,000; Full time

Job summary

Luxoft Poland is seeking an experienced Senior Data Engineer to develop and operate Databricks/Spark pipelines, design Delta Lake lakehouse layers, and orchestrate ETL/ELT jobs with Azure Data Factory or Airflow. You will implement data-quality checks, document data lineage, and optimize Spark performance while managing scaling and costs.

The role requires 7+ years in data engineering, with 3+ years on Databricks and 2+ on Azure data services.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems or a related field, or equivalent practical experience.
• 7+ years in data engineering, of which 3+ on Databricks / Apache Spark and 2+ on Azure data services (ADF, ADLS, Delta Lake).
• Databricks (Workflows, Delta Live Tables, Unity Catalog), Apache Spark (PySpark, Spark SQL), Delta Lake.
• Azure: Data Factory, Data Lake Storage Gen2, Key Vault, Event Hubs, Synapse or SQL DB; Azure DevOps CI/CD for notebooks and jobs.
• Python and SQL at expert level; data modelling (dimensional, data vault) and ELT design.
• Orchestration (ADF, Airflow), data-quality frameworks (Great Expectations, DLT expectations), monitoring and alerting.
• Performance and cost tuning of Spark workloads; Git-based development and testing of pipelines.

Responsibilities

• Pipeline reliability and data freshness against agreed SLAs.
• Accuracy and completeness of curated datasets.
• Schema change management and backward compatibility for downstream consumers.
• Documentation of data lineage and transformations (Unity Catalog, data catalogue).

Job description

Luxoft Poland is seeking an experienced Senior Data Engineer to develop and operate Databricks/Spark pipelines, design Delta Lake lakehouse layers, and orchestrate ETL/ELT jobs with Azure Data Factory or Airflow. You will implement data-quality checks, document data lineage, and optimize Spark performance while managing scaling and costs.

The role requires 7+ years in data engineering, with 3+ years on Databricks and 2+ on Azure data services.