Lead Data Engineer (Databricks & Spark)

EPAM Systems; Colombia; On-site; COP 120,000,000 - 180,000,000; Full time

Benefits offered by this job

Healthcare benefits

Employee learning programs

Paid time off

Job summary

EPAM Systems is seeking a Lead Data Engineer to shape scalable ingestion, transformation, and governed data products on Databricks. You will combine hands-on delivery with technical leadership to enable self-service analytics and AI-assisted engineering.

Responsibilities include designing data ingestion frameworks on Databricks, building batch and streaming pipelines with Spark and PySpark, and mentoring engineers across teams while enforcing best practices and data quality standards.

Qualifications

• 5+ years of data engineering on production-grade platforms.
• Strong Databricks experience in enterprise environments.
• Lead-level technical leadership and mentoring.
• End-to-end delivery across ingestion, transformation, semantic layers, and reporting.
• Advanced Apache Spark and PySpark expertise.
• Advanced SQL with performance tuning.
• Upper-Intermediate English (B2) for leading technical discussions.

Responsibilities

• Design scalable data ingestion and transformation on Databricks.
• Define Databricks best practices for medallion layers and operations.
• Build batch and streaming pipelines with Spark and PySpark.
• Deliver end-to-end data products from ingestion to consumption layers.
• Develop trusted analytical datasets and semantic models.
• Implement data quality, lineage, monitoring, and observability.
• Lead PoCs and evaluate new Databricks capabilities.
• Mentor engineers and promote reusable frameworks.

Skills

Databricks

Spark

PySpark

SQL

Leadership

Mentoring

Tools

Scala

ETL tools

Job description

EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.

We are building a next-generation analytics platform and need a Lead Data Engineer to shape scalable ingestion, transformation, and governed data products on Databricks. You will combine hands-on delivery with technical leadership and architecture to enable self-service analytics and AI-assisted engineering.

Responsibilities

• Design scalable data ingestion and transformation frameworks on Databricks
• Define and enforce Databricks best practices for medallion layers and operational excellence
• Build batch and streaming pipelines using Apache Spark and PySpark
• Deliver end-to-end data products from ingestion through governed consumption layers
• Develop trusted analytical datasets and semantic models for reporting and self-service analytics
• Implement data quality, lineage, monitoring, and observability standards
• Lead proof-of-concepts and evaluate emerging capabilities in the Databricks ecosystem
• Mentor engineers and drive reusable frameworks adopted across teams

Requirements

• 5+ years of data engineering experience with production-grade platforms
• Strong Databricks experience in enterprise production environments
• Lead-level technical leadership skills to drive standards and mentor engineers
• End-to-end delivery experience across ingestion, transformation, semantic layers, and reporting
• Advanced Apache Spark and PySpark expertise for scalable pipeline development
• Advanced SQL skills for transformation, modeling, and performance tuning
• Upper-Intermediate English proficiency (B2) for leading technical discussions

Nice to have

• Data architecture experience for enterprise lakehouse patterns
• Data solution architecture skills to define scalable platform designs
• Scala proficiency for Spark development and optimization

We offer

• International projects with top brands
• Work with global teams of highly skilled, diverse peers
• Healthcare benefits
• Employee financial programs
• Paid time off and sick leave
• Upskilling, reskilling and certification courses
• Unlimited access to the LinkedIn Learning library and 22,000+ courses
• Global career opportunities
• Volunteer and community involvement opportunities
• EPAM Employee Groups
• Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn

EPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.