Lead Data Engineer

EPAM Systems, Inc.; Colombia; A distancia; COP 120.000.000 - 190.000.000; Jornada completa

Descripción de la vacante

EPAM Systems, Inc. is seeking a Lead Data Engineer to build and evolve a next-generation analytics platform with Databricks focus and scalable data products.

You will blend hands-on delivery with technical leadership across ingestion, modeling, governance, and AI-enabled engineering practices. Responsibilities include designing scalable data ingestion and data product frameworks, implementing governance, quality, and observability, and delivering end-to-end analytics capabilities.

Formación

• 5+ years of data engineering experience.
• 1+ years of technical leadership experience guiding engineering standards and best practices.
• Strong platform delivery experience building enterprise analytics solutions end-to-end.
• Advanced SQL skills for transformation, modeling, and analytics consumption.
• Strong Databricks production experience with performance optimization and operational excellence.
• Solid data governance knowledge including security, quality, and lineage practices.

Responsabilidades

• Design scalable data ingestion, transformation, and data product frameworks.
• Define Databricks best practices, including Medallion Architecture, optimization, governance, and operational standards.
• Build batch, near real-time, and streaming pipelines using Databricks capabilities.
• Deliver end-to-end data products that support analytics use cases across domains.
• Develop trusted analytical datasets, semantic models, and governed consumption layers for reporting.
• Implement data quality, lineage, monitoring, and observability practices across pipelines.
• Enable self-service analytics through reusable, business-ready datasets and platform features.
• Lead proof-of-concepts and evaluate emerging capabilities across the Databricks ecosystem.
• Define AI SDLC practices and AI-assisted development patterns to accelerate delivery.
• Partner with engineering, analytics, product, and business stakeholders and mentor engineers on best practices.

Conocimientos

Spark

PySpark

SQL

Data governance

CI/CD

Leadership

Communication

English (B2)

Herramientas

Databricks

Apache Spark

PySpark

CI/CD pipelines

Descripción del empleo

We are looking for a Lead Data Engineer to build and evolve a next-generation analytics platform with a strong focus on Databricks and scalable data products. You will combine hands-on delivery with technical leadership across ingestion, modeling, governance, and AI-enabled engineering practices.ResponsibilitiesDesign scalable data ingestion, transformation, and data product frameworksDefine Databricks best practices, including Medallion Architecture, optimization, governance, and operational standardsBuild batch, near real-time, and streaming pipelines using Databricks capabilitiesDeliver end-to-end data products that support analytics use cases across domainsDevelop trusted analytical datasets, semantic models, and governed consumption layers for reportingImplement data quality, lineage, monitoring, and observability practices across pipelinesEnable self-service analytics through reusable, business-ready datasets and platform featuresLead proof-of-concepts and evaluate emerging capabilities across the Databricks ecosystemDefine AI SDLC practices and AI-assisted development patterns to accelerate deliveryPartner with engineering, analytics, product, and business stakeholders and mentor engineers on best practicesRequirements5+ years of data engineering experience1+ years of technical leadership experience guiding engineering standards and best practicesStrong platform delivery experience building enterprise analytics solutions end to endAdvanced Apache Spark and PySpark expertise for large-scale processingStrong Databricks production experience with performance optimization and operational excellenceAdvanced SQL skills for transformation, modeling, and analytics consumptionStrong data governance knowledge including security, quality, and lineage practicesSolid CI/CD experience and Data Engineering SDLC best practicesStrong communication skills to lead technical discussions with cross-functional stakeholdersUpper-Intermediate English proficiency (B2) for day-to-day collaborationNice to haveData Architecture experience designing lakehouse patterns and enterprise data platformsData Solution Architecture experience translating analytics needs into scalable designsScala proficiency for Spark-based development and optimizationWe offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedInEPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.