Senior Data Engineer: Databricks Lakehouse Migration

Sigma Software; Województwo małopolskie; On-site; PLN 180,000 - 240,000; Full time

Job summary

Sigma Software’s Data Engineering Center of Excellence is seeking a Senior Data Engineer to build scalable, high-performance data platforms for enterprise retail analytics. You will drive the migration of a large-scale platform from BigQuery to Databricks, design Lakehouse architectures with Delta Lake, and collaborate with international teams to deliver reliable, well-governed data solutions.

The role emphasizes CI/CD, cross-functional collaboration, and modernization of analytics workloads

Qualifications

• 5+ years of professional experience as a Data Engineer.
• Strong programming skills in Python and advanced SQL.
• Hands-on commercial experience with Databricks.
• Strong knowledge of Apache Spark, primarily PySpark.
• Experience designing and building modern cloud-based data platforms.
• Experience developing ETL / ELT pipelines and large-scale data processing solutions.
• Hands-on experience with Delta Lake.
• Experience with Spark Declarative Pipelines.
• Experience with cluster monitoring, metrics analysis, and performance optimization.
• Strong understanding of distributed data processing architectures.
• Solid understanding of data warehousing concepts and dimensional modeling.
• Experience with Airflow or similar orchestration tools.
• Experience optimizing complex analytical SQL workloads.
• Experience implementing CI / CD practices for data engineering platforms.
• Strong troubleshooting and performance optimization skills.
• Ability to work collaboratively in cross-functional international teams.
• Upper-Intermediate or higher English level.

Responsibilities

• Participate in migration of large-scale analytics platform from BigQuery to Databricks.
• Design and implement scalable Lakehouse architectures using Databricks and Delta Lake.
• Analyze existing ETL/ELT workloads and define migration approaches.
• Develop and optimize data pipelines processing large volumes of retail and analytical data.
• Implement incremental processing strategies and scalable transformation frameworks.
• Build and maintain Spark-based data processing solutions using PySpark.
• Design and maintain medallion architecture layers Bronze, Silver, and Gold.
• Implement data governance and security best practices using Unity Catalog.
• Collaborate with Data Science, Analytics, Product, and Customer Engineering teams.
• Participate in architecture discussions and technical solution design.
• Develop reusable data platform components and engineering standards.
• Conduct code reviews and contribute to platform reliability and maintainability.
• Troubleshoot and optimize complex SQL and Spark workloads.
• Support production deployments and platform modernization activities.

Skills

Python

SQL

Databricks

PySpark

Apache Spark

Delta Lake

Airflow

CI/CD

Unity Catalog

Cloud Data Platform

BigQuery

Tools

Databricks

Apache Spark

PySpark

Delta Lake

Python

SQL

Airflow

GCP

CI/CD

Unity Catalog

Job description

Sigma Software’s Data Engineering Center of Excellence is seeking a Senior Data Engineer to build scalable, high-performance data platforms for enterprise retail analytics. You will drive the migration of a large-scale platform from BigQuery to Databricks, design Lakehouse architectures with Delta Lake, and collaborate with international teams to deliver reliable, well-governed data solutions.

The role emphasizes CI/CD, cross-functional collaboration, and modernization of analytics workloads