Senior Data Engineer - Azure Databricks & PySpark

EPAM Systems; Colombia; Remote; COP 100,000,000 - 170,000,000; Full time

Job summary

EPAM Systems in Colombia seeks a Senior Data Engineer to design and deliver scalable, cloud-native data solutions on Azure and Databricks. You will manage large-scale datasets, refine data pipelines, and enforce governance with Unity Catalog.

The role emphasizes hands-on engineering, CI/CD, and collaboration with architects and teams to push reliability and performance improvements across the data platform.

Qualifications

• 3+ years of experience in Data Engineering or Software Engineering.
• Expertise in Azure Databricks, Apache Spark, and PySpark.
• Proficiency in Spark SQL and Azure cloud platforms.
• Advanced Python programming with strong SQL development and optimization.
• Background in Unity Catalog and enterprise data governance practices.
• Familiarity with Git-based source control workflows.
• Experience designing and maintaining CI/CD pipelines.
• Knowledge of Databricks Workflows and Azure Data Factory (ADF).
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Responsibilities

• Design, build, and sustain scalable data ingestion, transformation, and delivery pipelines.
• Deliver cloud-native data solutions using Azure and Databricks technologies.
• Create and refine ETL/ELT processes with PySpark, Spark SQL, and Delta Lake.
• Apply metadata-driven ingestion frameworks to boost scalability and maintainability.
• Build batch and incremental data processing pipelines.
• Set up data quality controls, monitoring, and error handling mechanisms.
• Collaborate with stakeholders to capture requirements and deliver data solutions.
• Guide team members and shape engineering standards and best practices.
• Contribute to data governance, security, and access management initiatives.

Skills

Azure Databricks

PySpark

Spark SQL

Python

SQL

CI/CD

Git workflows

Data governance

Unity Catalog

Azure services

Data quality

Tools

Databricks

Azure Data Factory

Terraform

Job description

EPAM Systems in Colombia seeks a Senior Data Engineer to design and deliver scalable, cloud-native data solutions on Azure and Databricks. You will manage large-scale datasets, refine data pipelines, and enforce governance with Unity Catalog.

The role emphasizes hands-on engineering, CI/CD, and collaboration with architects and teams to push reliability and performance improvements across the data platform.