Senior Big Data Engineer (Databricks + Azure)

SoftServe; Poland; On-site; PLN 180,000 - 320,000; Full time

Job summary

SoftServe is seeking an experienced Data Engineer to build scalable data solutions on the Databricks Lakehouse platform in Azure. You will work on batch and streaming ETL pipelines and collaborate with stakeholders to deliver production-ready solutions.

You will design and optimize data processing with PySpark, Python, and SQL, manage Databricks clusters, and implement governance with Unity Catalog. This role emphasizes collaboration and continuous improvement within the team.

Qualifications

• 5+ years of experience in Data Engineering.
• Strong PySpark, Python and SQL proficiency.
• Hands-on Databricks experience including clusters and SQL Warehouses.
• Knowledge of Unity Catalog governance and data sharing.
• CI/CD practices with Databricks Asset Bundles.
• Delta Lake and Lakehouse concepts understanding.
• Experience with Azure data services such as ADF, Synapse, ADLS (nice to have).

Responsibilities

• Design, develop, and maintain scalable batch and streaming ETL pipelines on the Databricks Lakehouse platform.
• Build data processing solutions using PySpark, Python, and SQL.
• Configure and maintain Databricks clusters and SQL Warehouses.
• Implement Unity Catalog governance, including access control, lineage, and data sharing.
• Develop and deploy data solutions using Databricks Asset Bundles and CI/CD processes.
• Design and maintain bronze-silver-gold Lakehouse architectures using Delta Lake.
• Expose and visualize data using Databricks SQL and BI tools such as Power BI.
• Collaborate with stakeholders to understand requirements and deliver effective data solutions.
• Participate in architecture discussions and continuous improvement initiatives.
• Support Databricks AI/BI capabilities such as dashboards and ML tooling.

Skills

PySpark

Python

SQL

Databricks

Delta Lake

Unity Catalog

Power BI

Tools

Databricks

CI/CD

Power BI

Azure DevOps

Delta Lake

Job description

About The Role

In this role, you will build scalable, efficient data solutions on the Databricks Lakehouse platform in Azure environments. You will work on batch and streaming data pipelines, support data platform optimization initiatives, and collaborate with technical and business stakeholders to deliver reliable, production-ready data solutions across different stages of the project lifecycle.

About The Role

In this role, you will build scalable, efficient data solutions on the Databricks Lakehouse platform in Azure environments. You will work on batch and streaming data pipelines, support data platform optimization initiatives, and collaborate with technical and business stakeholders to deliver reliable, production-ready data solutions across different stages of the project lifecycle.

Responsibilities

• Design, develop, and maintain scalable batch and streaming ETL pipelines on the Databricks Lakehouse platform
• Build and optimize data processing solutions using PySpark, Python, and SQL
• Configure and maintain Databricks clusters and SQL Warehouses
• Implement and support data governance practices using Unity Catalog, including access control, lineage, and cross-workspace sharing
• Develop and deploy data solutions using Databricks Asset Bundles and CI/CD processes
• Design and maintain bronze-silver-gold Lakehouse architectures using Delta Lake
• Expose and visualize data using Databricks SQL and BI tools such as Power BI
• Collaborate with technical and business stakeholders to understand requirements and deliver effective data solutions
• Participate in architecture discussions and contribute to continuous improvement initiatives within the team
• Support implementation and usage of modern Databricks AI/BI capabilities such as Dashboards, Metric Views, MLflow, Genie, and agents where applicable

Requirements

• 5+ years of experience in Data Engineering, including designing data models and scalable ETL pipelines
• Strong expertise in PySpark, Python, and SQL, including performance tuning and optimization
• Hands-on experience with Databricks, including clusters and SQL Warehouses
• Practical knowledge of Unity Catalog governance, including fine-grained access control, lineage, and data sharing
• Experience delivering end-to-end data solutions using Databricks Asset Bundles and CI/CD practices
• Strong understanding of Delta Lake and Lakehouse architecture concepts
• Experience working with Databricks SQL and BI tools such as Power BI
• Strong communication and collaboration skills with technical and business stakeholders
• Upper-intermediate or higher level of English
• Familiarity with Databricks AI/BI tools such as MLflow, Dashboards, Metric Views, Genie, or agents (is a plus)
• Experience with Azure data services such as ADF, Synapse, ADLS, Azure DevOps, or Microsoft Fabric (nice to have)

SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.