Lead Data Engineer

EPAM Systems; Colombia; On-site; COP 120,000,000 - 180,000,000; Full time

Benefits offered by this job

Healthcare benefits

Paid time off

Upskilling & certification courses

LinkedIn Learning access

Global career opportunities

Volunteer opportunities

Job summary

EPAM Systems in Colombia is seeking a Lead Data Engineer to build and evolve a next-generation analytics platform with Databricks and scalable data products. You will combine hands-on delivery with technical leadership across ingestion, modeling, governance, and AI-enabled engineering practices.

The role requires 5+ years in data engineering, 1+ year of technical leadership, strong Spark/PySpark, Databricks production experience, advanced SQL, data governance, and solid CI/CD skills, plus

Qualifications

• 5+ years of data engineering experience
• 1+ years of technical leadership experience guiding engineering standards and best practices
• Strong platform delivery experience building enterprise analytics solutions end to end
• Advanced Apache Spark and PySpark expertise for large-scale processing
• Strong Databricks production experience with performance optimization and operational excellence
• Advanced SQL skills for transformation, modeling, and analytics consumption
• Strong data governance knowledge including security, quality, and lineage practices
• Solid CI/CD experience and Data Engineering SDLC best practices
• Strong communication skills to lead technical discussions with cross-functional stakeholders
• Upper-Intermediate English proficiency (B2) for day-to-day collaboration

Responsibilities

• Design scalable data ingestion, transformation, and data product frameworks
• Define Databricks best practices, including Medallion Architecture, optimization, governance, and operational standards
• Build batch, near real-time, and streaming pipelines using Databricks capabilities
• Deliver end-to-end data products that support analytics use cases across domains
• Develop trusted analytical datasets, semantic models, and governed consumption layers for reporting
• Implement data quality, lineage, monitoring, and observability practices across pipelines
• Enable self-service analytics through reusable, business-ready datasets and platform features
• Lead proof-of-concepts and evaluate emerging capabilities across the Databricks ecosystem
• Define AI SDLC practices and AI-assisted development patterns to accelerate delivery
• Partner with engineering, analytics, product, and business stakeholders and mentor engineers on best practices

Skills

Databricks

Apache Spark

PySpark

SQL

Data governance

CI/CD

Leadership

English communication

Tools

Databricks DL

Job description

EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.

We are looking for a Lead Data Engineer to build and evolve a next-generation analytics platform with a strong focus on Databricks and scalable data products. You will combine hands-on delivery with technical leadership across ingestion, modeling, governance, and AI-enabled engineering practices.

Responsibilities

• Design scalable data ingestion, transformation, and data product frameworks
• Define Databricks best practices, including Medallion Architecture, optimization, governance, and operational standards
• Build batch, near real-time, and streaming pipelines using Databricks capabilities
• Deliver end-to-end data products that support analytics use cases across domains
• Develop trusted analytical datasets, semantic models, and governed consumption layers for reporting
• Implement data quality, lineage, monitoring, and observability practices across pipelines
• Enable self-service analytics through reusable, business-ready datasets and platform features
• Lead proof-of-concepts and evaluate emerging capabilities across the Databricks ecosystem
• Define AI SDLC practices and AI-assisted development patterns to accelerate delivery
• Partner with engineering, analytics, product, and business stakeholders and mentor engineers on best practices

Requirements

• 5+ years of data engineering experience
• 1+ years of technical leadership experience guiding engineering standards and best practices
• Strong platform delivery experience building enterprise analytics solutions end to end
• Advanced Apache Spark and PySpark expertise for large-scale processing
• Strong Databricks production experience with performance optimization and operational excellence
• Advanced SQL skills for transformation, modeling, and analytics consumption
• Strong data governance knowledge including security, quality, and lineage practices
• Solid CI/CD experience and Data Engineering SDLC best practices
• Strong communication skills to lead technical discussions with cross-functional stakeholders
• Upper-Intermediate English proficiency (B2) for day-to-day collaboration

Nice to have

• Data Architecture experience designing lakehouse patterns and enterprise data platforms
• Data Solution Architecture experience translating analytics needs into scalable designs
• Scala proficiency for Spark-based development and optimization

We offer

• International projects with top brands
• Work with global teams of highly skilled, diverse peers
• Healthcare benefits
• Employee financial programs
• Paid time off and sick leave
• Upskilling, reskilling and certification courses
• Unlimited access to the LinkedIn Learning library and 22,000+ courses
• Global career opportunities
• Volunteer and community involvement opportunities
• EPAM Employee Groups
• Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn

EPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.