Senior Data Engineer: Databricks Lakehouse Migration Lead

Sigma Software; Warszawa; On-site; PLN 180,000 - 280,000; Full time

Job summary

Sigma Software's Data Engineering Center of Excellence in Warsaw is seeking a Senior Data Engineer to build scalable, high-performance data platforms and contribute to the migration of a large-scale analytics platform from BigQuery to Databricks. You will collaborate with international teams and help shape reliable data solutions.

You will design Spark-based processing, Delta Lake architecture, and medallion data layers while implementing governance and security practices.

Qualifications

• 5+ years of professional experience as a Data Engineer.
• Strong programming skills in Python and advanced SQL.
• Hands-on commercial experience with Databricks.
• Strong knowledge of Apache Spark, primarily PySpark.
• Experience designing and building modern cloud-based data platforms.
• Experience developing ETL / ELT pipelines and large-scale data processing solutions.
• Hands-on experience with Delta Lake.
• Experience with Spark Declarative Pipelines.
• Experience with cluster monitoring, metrics analysis, and performance optimization.
• Strong understanding of distributed data processing architectures.
• Solid understanding of data warehousing concepts and dimensional modeling.
• Experience with Airflow or similar orchestration tools.
• Experience optimizing complex analytical SQL workloads.
• Strong troubleshooting and performance optimization skills.
• Ability to work collaboratively in cross-functional international teams.
• Upper-Intermediate or higher English level.

Responsibilities

• Participate in the migration of a large-scale analytical platform from BigQuery to Databricks.
• Design and implement scalable Lakehouse architectures using Databricks and Delta Lake.
• Analyze existing ETL / ELT workloads and define migration approaches.
• Develop and optimize data pipelines processing large volumes of retail and analytical data.
• Implement incremental processing strategies and scalable transformation frameworks.
• Build and maintain Spark-based data processing solutions using PySpark.
• Design and maintain medallion architecture layers including Bronze, Silver, and Gold.
• Implement data governance and security best practices using Unity Catalog.
• Collaborate with Data Science, Analytics, Product, and Customer Engineering teams.
• Participate in architecture discussions and technical solution design.
• Develop reusable data platform components and engineering standards.
• Conduct code reviews and contribute to platform reliability and maintainability.
• Troubleshoot and optimize complex SQL and Spark workloads.
• Support production deployments and platform modernization activities.

Skills

Python programming

Advanced SQL

Team collaboration

Analytical thinking

Problem solving

Tools

Databricks

Apache Spark

Delta Lake

Airflow

CI/CD

Unity Catalog

GCP

BigQuery

Job description

Sigma Software's Data Engineering Center of Excellence in Warsaw is seeking a Senior Data Engineer to build scalable, high-performance data platforms and contribute to the migration of a large-scale analytics platform from BigQuery to Databricks. You will collaborate with international teams and help shape reliable data solutions.

You will design Spark-based processing, Delta Lake architecture, and medallion data layers while implementing governance and security practices.