Lead Databricks & Spark Data Engineer - Platform Architect

EPAM Systems; Colombia; On-site; COP 120,000,000 - 180,000,000; Full time

Benefits offered by this job

Healthcare benefits

Employee learning programs

Paid time off

Job summary

EPAM Systems is seeking a Lead Data Engineer to shape scalable ingestion, transformation, and governed data products on Databricks. You will combine hands-on delivery with technical leadership to enable self-service analytics and AI-assisted engineering.

Responsibilities include designing data ingestion frameworks on Databricks, building batch and streaming pipelines with Spark and PySpark, and mentoring engineers across teams while enforcing best practices and data quality standards.

Qualifications

• 5+ years of data engineering on production-grade platforms.
• Strong Databricks experience in enterprise environments.
• Lead-level technical leadership and mentoring.
• End-to-end delivery across ingestion, transformation, semantic layers, and reporting.
• Advanced Apache Spark and PySpark expertise.
• Advanced SQL with performance tuning.
• Upper-Intermediate English (B2) for leading technical discussions.

Responsibilities

• Design scalable data ingestion and transformation on Databricks.
• Define Databricks best practices for medallion layers and operations.
• Build batch and streaming pipelines with Spark and PySpark.
• Deliver end-to-end data products from ingestion to consumption layers.
• Develop trusted analytical datasets and semantic models.
• Implement data quality, lineage, monitoring, and observability.
• Lead PoCs and evaluate new Databricks capabilities.
• Mentor engineers and promote reusable frameworks.

Skills

Databricks

Spark

PySpark

SQL

Leadership

Mentoring

Tools

Scala

ETL tools

Job description

EPAM Systems is seeking a Lead Data Engineer to shape scalable ingestion, transformation, and governed data products on Databricks. You will combine hands-on delivery with technical leadership to enable self-service analytics and AI-assisted engineering.

Responsibilities include designing data ingestion frameworks on Databricks, building batch and streaming pipelines with Spark and PySpark, and mentoring engineers across teams while enforcing best practices and data quality standards.