Senior Data Engineer: Azure Databricks & PySpark

EPAM Systems; México; Remote; MXN 900,000 - 1,500,000; Full time

Benefits offered by this job

Healthcare benefits

Paid time off

Upskilling and certification courses

LinkedIn Learning access

Job summary

EPAM Systems seeks a Senior Data Engineer to design and deliver scalable cloud-native data solutions on Azure and Databricks. You will craft ETL/ELT pipelines with PySpark, Spark SQL, and Delta Lake, while enforcing data governance and robust quality controls.

You will collaborate with stakeholders to capture requirements, optimize workloads, and drive best practices in CI/CD and infrastructure automation.

Qualifications

• 3+ years of experience in Data Engineering or Software Engineering.
• Expertise in Azure Databricks, Apache Spark, and PySpark.
• Proficiency in Spark SQL with cloud-based Azure solutions.
• Advanced Python programming and SQL optimization.
• Knowledge of Unity Catalog and data governance practices.
• Familiarity with Git-based workflows and CI/CD pipelines.
• Experience with Databricks Workflows and Azure Data Factory.
• Strong communication and stakeholder management skills.

Responsibilities

• Design, build, and sustain scalable data ingestion, transformation, and delivery pipelines.
• Deliver cloud-native data solutions using Azure and Databricks technologies.
• Develop ETL/ELT processes with PySpark, Spark SQL, and Delta Lake.
• Apply metadata-driven ingestion frameworks to boost scalability and maintainability.
• Build batch and incremental data processing pipelines and ensure data quality.
• Set up monitoring, logging, and error handling mechanisms; address production issues.
• Track and optimize Databricks workloads for efficiency and cost control.
• Guide team members and shape engineering standards and best practices.

Skills

Data engineering

Cloud data platforms

Analytical thinking

Problem solving

Stakeholder management

Communication

Python

Education

Bachelor's degree in a technical field

Tools

Azure Databricks

Apache Spark

PySpark

Spark SQL

Delta Lake

Unity Catalog

Azure Data Factory

Terraform / IaC

Databricks Workflows

Git workflows

CI/CD

Job description

EPAM Systems seeks a Senior Data Engineer to design and deliver scalable cloud-native data solutions on Azure and Databricks. You will craft ETL/ELT pipelines with PySpark, Spark SQL, and Delta Lake, while enforcing data governance and robust quality controls.

You will collaborate with stakeholders to capture requirements, optimize workloads, and drive best practices in CI/CD and infrastructure automation.