Lead Data Platform Engineer (Databricks)

EPAM Systems, Inc.; Colombia; A distancia; COP 295.741.000 - 394.322.000; Jornada completa

Descripción de la vacante

EPAM Systems, Inc. is seeking a Lead Data Platform Engineer to design and optimize scalable Databricks data platforms powering analytics and AI initiatives.

You will own modern data pipelines, governance, and high-quality data products in the cloud. Responsibilities include building end-to-end pipelines with Lakeflow DLT, Auto Loader, and Medallion Architecture; implementing Unity Catalog governance; and optimizing Spark workloads for performance and cost.

Formación

• Bachelor-level or higher data/analytics education is not specified.
• Lead platform initiatives with end-to-end ownership and governance.
• Proven ability to drive architecture decisions and mentor teams.

Responsabilidades

• Design scalable data platform architectures on Databricks for analytics and AI.
• Build end-to-end pipelines using Lakeflow DLT, Auto Loader, and Medallion Architecture.
• Implement data governance with Unity Catalog including lineage and access controls.
• Optimize Spark workloads for performance and cost efficiency.
• Develop dimensional models and curated datasets for reporting and downstream consumption.
• Deliver AI/ML-ready datasets and enable GenAI/RAG data pipelines.
• Automate deployments with CI/CD using GitHub Actions and Databricks Asset Bundles.
• Integrate AWS services (S3, IAM, KMS, DMS, Aurora PostgreSQL) into data solutions.
• Establish data quality checks, monitoring, and runbooks for production pipelines.
• Collaborate with stakeholders to translate requirements into data products and SLAs.
• Review code and set engineering standards to improve maintainability.

Conocimientos

Leadership

Cross-team collaboration

Strong communication

English proficiency

Herramientas

Databricks

Delta Lake

Unity Catalog

Lakeflow DLT

Auto Loader

Medallion Architecture

Apache Spark

PySpark

SQL

GitHub Actions

Databricks Asset Bundles

Amazon S3

IAM

KMS

DMS

Aurora PostgreSQL

Descripción del empleo

We are seeking a Lead Data Platform Engineer to design, build, and optimize scalable Databricks data platforms that power analytics, reporting, and AI initiatives. You will own modern pipelines, governance, and high-quality data products in the cloud.ResponsibilitiesDesign scalable data platform architectures on Databricks to support analytics and AI use casesBuild end-to-end data pipelines using Lakeflow DLT, Auto Loader, and Medallion ArchitectureImplement data governance with Unity Catalog, including access controls, lineage, and metadata standardsOptimize Apache Spark workloads for performance, reliability, and cost efficiencyDevelop dimensional data models and curated datasets for reporting and downstream consumptionDeliver AI/ML-ready datasets and enable feature engineering or GenAI/RAG data pipelinesAutomate deployments with CI/CD using GitHub Actions and Databricks Asset BundlesIntegrate AWS services such as Amazon S3, IAM, KMS, DMS, and Aurora PostgreSQL into data solutionsEstablish data quality checks, monitoring, and operational runbooks for production pipelinesCollaborate with stakeholders to translate requirements into well-defined data products and SLAsReview code and set engineering standards to improve maintainability and delivery speedRequirements5+ years of data engineering or analytics engineering experience1+ years of leadership experience driving platform standards and technical decisionsStrong project delivery skills owning end-to-end data platform initiativesAdvanced Databricks skills with Unity Catalog, Delta Lake, and Databricks for Data AnalyticsStrong Apache Spark skills using PySpark and SQL for large-scale processingStrong data pipeline skills with Lakeflow DLT, Auto Loader, and Medallion ArchitectureAdvanced data modeling skills with FACT/DIM, Star Schema, and SCD Type 2Strong data governance skills including metadata management, lineage, and access controlsStrong AWS skills with Amazon S3, IAM, KMS, DMS, and Aurora PostgreSQLStrong CI/CD skills with Git, GitHub Actions, and Databricks Asset Bundles (DABs)Strong communication skills for cross-team collaboration and clear technical writingEnglish proficiency: B2 Upper-IntermediateNice to haveDatabricks Mosaic AI, Vector Search, Lakebase, or Model Serving experienceAI/BI (Genie), semantic layers, or natural language analytics experienceHealthcare or Life Sciences domain knowledge including HIPAA/PHI considerationsChange Data Capture (CDC) implementation experienceServerless data platform experienceWe offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedInEPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.