Senior Data Platform Engineer - Databricks AI Pipelines

EPAM Systems, Inc.; Colombia; A distancia; COP 295.741.000 - 394.322.000; Jornada completa

Descripción de la vacante

EPAM Systems, Inc. is seeking a Lead Data Platform Engineer to design and optimize scalable Databricks data platforms powering analytics and AI initiatives.

You will own modern data pipelines, governance, and high-quality data products in the cloud. Responsibilities include building end-to-end pipelines with Lakeflow DLT, Auto Loader, and Medallion Architecture; implementing Unity Catalog governance; and optimizing Spark workloads for performance and cost.

Formación

• Bachelor-level or higher data/analytics education is not specified.
• Lead platform initiatives with end-to-end ownership and governance.
• Proven ability to drive architecture decisions and mentor teams.

Responsabilidades

• Design scalable data platform architectures on Databricks for analytics and AI.
• Build end-to-end pipelines using Lakeflow DLT, Auto Loader, and Medallion Architecture.
• Implement data governance with Unity Catalog including lineage and access controls.
• Optimize Spark workloads for performance and cost efficiency.
• Develop dimensional models and curated datasets for reporting and downstream consumption.
• Deliver AI/ML-ready datasets and enable GenAI/RAG data pipelines.
• Automate deployments with CI/CD using GitHub Actions and Databricks Asset Bundles.
• Integrate AWS services (S3, IAM, KMS, DMS, Aurora PostgreSQL) into data solutions.
• Establish data quality checks, monitoring, and runbooks for production pipelines.
• Collaborate with stakeholders to translate requirements into data products and SLAs.
• Review code and set engineering standards to improve maintainability.

Conocimientos

Leadership

Cross-team collaboration

Strong communication

English proficiency

Herramientas

Databricks

Delta Lake

Unity Catalog

Lakeflow DLT

Auto Loader

Medallion Architecture

Apache Spark

PySpark

SQL

GitHub Actions

Databricks Asset Bundles

Amazon S3

IAM

KMS

DMS

Aurora PostgreSQL

Descripción del empleo

EPAM Systems, Inc. is seeking a Lead Data Platform Engineer to design and optimize scalable Databricks data platforms powering analytics and AI initiatives.

You will own modern data pipelines, governance, and high-quality data products in the cloud. Responsibilities include building end-to-end pipelines with Lakeflow DLT, Auto Loader, and Medallion Architecture; implementing Unity Catalog governance; and optimizing Spark workloads for performance and cost.