Senior Databricks Big Data Engineer (Azure, Delta Lake)

SoftServe; Poland; On-site; PLN 180,000 - 320,000; Full time

Job summary

SoftServe is seeking an experienced Data Engineer to build scalable data solutions on the Databricks Lakehouse platform in Azure. You will work on batch and streaming ETL pipelines and collaborate with stakeholders to deliver production-ready solutions.

You will design and optimize data processing with PySpark, Python, and SQL, manage Databricks clusters, and implement governance with Unity Catalog. This role emphasizes collaboration and continuous improvement within the team.

Qualifications

• 5+ years of experience in Data Engineering.
• Strong PySpark, Python and SQL proficiency.
• Hands-on Databricks experience including clusters and SQL Warehouses.
• Knowledge of Unity Catalog governance and data sharing.
• CI/CD practices with Databricks Asset Bundles.
• Delta Lake and Lakehouse concepts understanding.
• Experience with Azure data services such as ADF, Synapse, ADLS (nice to have).

Responsibilities

• Design, develop, and maintain scalable batch and streaming ETL pipelines on the Databricks Lakehouse platform.
• Build data processing solutions using PySpark, Python, and SQL.
• Configure and maintain Databricks clusters and SQL Warehouses.
• Implement Unity Catalog governance, including access control, lineage, and data sharing.
• Develop and deploy data solutions using Databricks Asset Bundles and CI/CD processes.
• Design and maintain bronze-silver-gold Lakehouse architectures using Delta Lake.
• Expose and visualize data using Databricks SQL and BI tools such as Power BI.
• Collaborate with stakeholders to understand requirements and deliver effective data solutions.
• Participate in architecture discussions and continuous improvement initiatives.
• Support Databricks AI/BI capabilities such as dashboards and ML tooling.

Skills

PySpark

Python

SQL

Databricks

Delta Lake

Unity Catalog

Power BI

Tools

Databricks

CI/CD

Power BI

Azure DevOps

Delta Lake

Job description

SoftServe is seeking an experienced Data Engineer to build scalable data solutions on the Databricks Lakehouse platform in Azure. You will work on batch and streaming ETL pipelines and collaborate with stakeholders to deliver production-ready solutions.

You will design and optimize data processing with PySpark, Python, and SQL, manage Databricks clusters, and implement governance with Unity Catalog. This role emphasizes collaboration and continuous improvement within the team.