Senior Data Engineer

EPAM Systems; Colombia; Remote; COP 100,000,000 - 170,000,000; Full time

Job summary

EPAM Systems in Colombia seeks a Senior Data Engineer to design and deliver scalable, cloud-native data solutions on Azure and Databricks. You will manage large-scale datasets, refine data pipelines, and enforce governance with Unity Catalog.

The role emphasizes hands-on engineering, CI/CD, and collaboration with architects and teams to push reliability and performance improvements across the data platform.

Qualifications

• 3+ years of experience in Data Engineering or Software Engineering.
• Expertise in Azure Databricks, Apache Spark, and PySpark.
• Proficiency in Spark SQL and Azure cloud platforms.
• Advanced Python programming with strong SQL development and optimization.
• Background in Unity Catalog and enterprise data governance practices.
• Familiarity with Git-based source control workflows.
• Experience designing and maintaining CI/CD pipelines.
• Knowledge of Databricks Workflows and Azure Data Factory (ADF).
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Responsibilities

• Design, build, and sustain scalable data ingestion, transformation, and delivery pipelines.
• Deliver cloud-native data solutions using Azure and Databricks technologies.
• Create and refine ETL/ELT processes with PySpark, Spark SQL, and Delta Lake.
• Apply metadata-driven ingestion frameworks to boost scalability and maintainability.
• Build batch and incremental data processing pipelines.
• Set up data quality controls, monitoring, and error handling mechanisms.
• Collaborate with stakeholders to capture requirements and deliver data solutions.
• Guide team members and shape engineering standards and best practices.
• Contribute to data governance, security, and access management initiatives.

Skills

Azure Databricks

PySpark

Spark SQL

Python

SQL

CI/CD

Git workflows

Data governance

Unity Catalog

Azure services

Data quality

Tools

Databricks

Azure Data Factory

Terraform

Job description

We are seeking a Senior Data Engineer with proven expertise in Azure Data Platform technologies, Databricks, and PySpark to strengthen our growing data engineering organization. In this position, you will craft and deliver scalable, cloud-native data solutions that drive enterprise analytics, reporting, and business intelligence efforts. You will manage large-scale datasets, develop reusable ingestion frameworks, refine data pipelines, and play a key role in evolving modern data platforms built on Azure and Databricks. The right candidate pairs strong hands-on engineering capabilities with a background in designing robust, metadata-driven data solutions, along with a genuine drive to deliver high-quality, maintainable data products.ResponsibilitiesDesign, build, and sustain scalable data ingestion, transformation, and delivery pipelinesDeliver cloud-native data solutions using Azure and Databricks technologiesCreate and refine ETL/ELT processes with PySpark, Spark SQL, and Delta LakeApply metadata-driven and configuration-based ingestion frameworks to boost scalability and maintainabilityBuild and sustain batch and incremental data processing pipelinesSet up data quality controls, validation frameworks, monitoring, logging, and error handling mechanismsWork closely with business stakeholders, architects, and engineering teams to capture requirements and deliver data solutionsAddress production issues and push forward continuous improvements in platform reliability and performanceTrack and fine-tune Databricks workloads for efficiency, scalability, and cost managementUphold engineering best practices including code reviews, automated testing, CI/CD, and infrastructure automationContribute to data governance, security, and access management initiatives using Unity Catalog and Azure servicesGuide team members and shape engineering standards and best practicesRequirements3+ years of experience in Data Engineering or Software EngineeringExpertise in Azure Databricks, Apache Spark, and PySparkProficiency in Spark SQL and a track record of building solutions on Azure cloud platformsAdvanced Python programming skills paired with strong SQL development and optimization experienceBackground in Unity Catalog and enterprise data governance practicesFamiliarity with Git-based source control workflowsExperience designing and maintaining CI/CD pipelinesKnowledge of orchestration tools such as Databricks Workflows and Azure Data Factory (ADF)Strong analytical, troubleshooting, and problem-solving skillsExcellent communication and stakeholder management abilitiesEnglish proficiency at B2 level or higherNice to haveKnowledge of Terraform and Infrastructure as Code (IaC)Experience with Databricks Asset BundlesSkills in building reusable data engineering frameworks and acceleratorsExpertise in advanced Spark performance tuning and optimizationFamiliarity with enterprise-scale lakehouse architecturesExposure to data security, compliance, and governance initiativesExperience working within Agile delivery teamsWe offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedInEPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.