Senior Data Platform Engineer: Azure Databricks & PySpark

EPAM Systems; Wrocław; Hybrid; PLN 180,000 - 280,000; Full time

Benefits offered by this job

Hybrid by design

Remote within Poland

relocation opportunities

Employee stock purchase plan

Health insurance

Well-being programs

Job summary

EPAM Systems in Wroclaw is seeking a Senior/Lead Data Software Engineer to drive a scalable, ML-ready platform, migrating to Azure-native Databricks and enhancing data governance for compliant data workflows.

You will optimize data processing, tune ML models in Azure, and develop robust PySpark ETL/ELT pipelines while collaborating across teams to incorporate GenAI capabilities and ensure data quality and security.

Qualifications

• 3+ years of data engineering experience with Python and Spark.
• Strong experience with Azure Databricks and PySpark.
• Experience migrating big data platforms to Azure-native services.
• Knowledge of data governance and regulatory compliance.

Responsibilities

• Migrate and optimize over 500 data jobs using Azure Databricks.
• Manage and process 12 TB of data across platforms.
• Tune ML models in Azure using Spark and Delta tables.
• Develop and maintain PySpark ETL/ELT pipelines.
• Collaborate to integrate GenAI into data workflows.
• Monitor data quality and implement improvements.
• Automate deployment with Terraform and GitLab CI/CD.
• Support data governance for regulatory compliance.
• Document processes and guide junior engineers.
• Implement code quality and data security best practices.
• Participate in code reviews and knowledge sharing.
• Optimize data storage and processing costs.

Skills

Python

Spark

ETL/ELT design

English (B2+)

Tools

Azure Databricks

PySpark

Delta tables

Terraform

Docker

Kubernetes

AKS

Job description

EPAM Systems in Wroclaw is seeking a Senior/Lead Data Software Engineer to drive a scalable, ML-ready platform, migrating to Azure-native Databricks and enhancing data governance for compliant data workflows.

You will optimize data processing, tune ML models in Azure, and develop robust PySpark ETL/ELT pipelines while collaborating across teams to incorporate GenAI capabilities and ensure data quality and security.