Lead Data Engineer: Databricks & AI-Driven Data Platforms

EPAM Systems, Inc.; Colombia; A distancia; COP 120.000.000 - 190.000.000; Jornada completa

Descripción de la vacante

EPAM Systems, Inc. is seeking a Lead Data Engineer to build and evolve a next-generation analytics platform with Databricks focus and scalable data products.

You will blend hands-on delivery with technical leadership across ingestion, modeling, governance, and AI-enabled engineering practices. Responsibilities include designing scalable data ingestion and data product frameworks, implementing governance, quality, and observability, and delivering end-to-end analytics capabilities.

Formación

• 5+ years of data engineering experience.
• 1+ years of technical leadership experience guiding engineering standards and best practices.
• Strong platform delivery experience building enterprise analytics solutions end-to-end.
• Advanced SQL skills for transformation, modeling, and analytics consumption.
• Strong Databricks production experience with performance optimization and operational excellence.
• Solid data governance knowledge including security, quality, and lineage practices.

Responsabilidades

• Design scalable data ingestion, transformation, and data product frameworks.
• Define Databricks best practices, including Medallion Architecture, optimization, governance, and operational standards.
• Build batch, near real-time, and streaming pipelines using Databricks capabilities.
• Deliver end-to-end data products that support analytics use cases across domains.
• Develop trusted analytical datasets, semantic models, and governed consumption layers for reporting.
• Implement data quality, lineage, monitoring, and observability practices across pipelines.
• Enable self-service analytics through reusable, business-ready datasets and platform features.
• Lead proof-of-concepts and evaluate emerging capabilities across the Databricks ecosystem.
• Define AI SDLC practices and AI-assisted development patterns to accelerate delivery.
• Partner with engineering, analytics, product, and business stakeholders and mentor engineers on best practices.

Conocimientos

Spark

PySpark

SQL

Data governance

CI/CD

Leadership

Communication

English (B2)

Herramientas

Databricks

Apache Spark

PySpark

CI/CD pipelines

Descripción del empleo

EPAM Systems, Inc. is seeking a Lead Data Engineer to build and evolve a next-generation analytics platform with Databricks focus and scalable data products.

You will blend hands-on delivery with technical leadership across ingestion, modeling, governance, and AI-enabled engineering practices. Responsibilities include designing scalable data ingestion and data product frameworks, implementing governance, quality, and observability, and delivering end-to-end analytics capabilities.