Lead Data Engineer: Databricks & Spark Platform Architect

EPAM Systems, Inc.; Colombia; A distancia; COP 100.000.000 - 180.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Healthcare benefits

Paid time off and sick leave

Upskilling, reskilling and certificate

LinkedIn Learning access

Global career opportunities

Volunteer and community involvement

Descripción de la vacante

EPAM Systems, Inc. is seeking a Lead Data Engineer in Colombia to shape scalable ingestion, transformation, and governed data products on Databricks.

You will combine hands-on delivery with technical leadership to enable self-service analytics and AI-assisted engineering. You will design ingestion frameworks, build batch and streaming pipelines with Spark and PySpark, and deliver end-to-end data products with strong data governance and observability.

Formación

• 5+ years of data engineering on production-grade platforms.
• Strong Databricks experience in enterprise environments.
• End-to-end delivery across ingestion, transformation, semantic layers, and reporting.
• Advanced Spark and PySpark expertise for scalable pipeline development.
• Upper-Intermediate English (B2) for leading technical discussions.

Responsabilidades

• Design scalable data ingestion and transformation frameworks on Databricks.
• Define and enforce Databricks best practices for medallion layers and operational excellence.
• Build batch and streaming pipelines using Apache Spark and PySpark.
• Deliver end-to-end data products from ingestion through governed consumption layers.
• Develop trusted analytical datasets and semantic models for reporting and self-service analytics.
• Implement data quality, lineage, monitoring, and observability standards.
• Lead proof-of-concepts and evaluate emerging capabilities in the Databricks ecosystem.
• Mentor engineers and drive reusable frameworks adopted across teams.

Conocimientos

Databricks

Spark

PySpark

SQL

Scala

Data ingestion

Data transformation

Data governance

Mentoring

Leadership

Descripción del empleo

EPAM Systems, Inc. is seeking a Lead Data Engineer in Colombia to shape scalable ingestion, transformation, and governed data products on Databricks.

You will combine hands-on delivery with technical leadership to enable self-service analytics and AI-assisted engineering. You will design ingestion frameworks, build batch and streaming pipelines with Spark and PySpark, and deliver end-to-end data products with strong data governance and observability.