Lead Data Engineer (Databricks & Spark)

EPAM Systems Inc; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

EPAM Systems seeks a Lead Data Engineer to shape scalable ingestion, transformation, and governed data products on Databricks. You will blend hands-on delivery with technical leadership to enable self-service analytics and AI-assisted engineering.

You will design and implement data pipelines using Spark/PySpark, enforce medallion architectural best practices, and mentor engineers across teams while delivering end-to-end data solutions.

Qualifications

• Proven track record as a data engineering leader.
• Experience delivering end-to-end data solutions.
• Strong capability to mentor engineers and drive standards.
• Experience with data governance, quality, and observability.

Responsibilities

• Design scalable data ingestion and transformation frameworks on Databricks.
• Define and enforce Databricks best practices for medallion layers and ops excellence.
• Build batch and streaming pipelines using Spark and PySpark.
• Deliver end-to-end data products from ingestion through governed layers.
• Develop trusted datasets and semantic models for reporting and self-service analytics.

Skills

Data engineering leadership

Mentoring engineers

End-to-end delivery

Data governance & quality

Tools

Databricks

Apache Spark

PySpark

SQL

Job description

We are building a next-generation analytics platform and need a Lead Data Engineer to shape scalable ingestion, transformation, and governed data products on Databricks. You will combine hands-on delivery with technical leadership and architecture to enable self-service analytics and AI-assisted engineering.

Responsibilities

• Design scalable data ingestion and transformation frameworks on Databricks
• Define and enforce Databricks best practices for medallion layers and operational excellence
• Build batch and streaming pipelines using Apache Spark and PySpark
• Deliver end-to-end data products from ingestion through governed consumption layers
• Develop trusted analytical datasets and semantic models for reporting and self-service analytics
• Implement data quality, lineage, monitoring, and observability standards
• Lead proof-of-concepts and evaluate emerging capabilities in the Databricks ecosystem
• Mentor engineers and drive reusable frameworks adopted across teams

Requirements

• 5+ years of data engineering experience with production-grade platforms
• Strong Databricks experience in enterprise production environments
• Lead-level technical leadership skills to drive standards and mentor engineers
• End-to-end delivery experience across ingestion, transformation, semantic layers, and reporting
• Advanced Apache Spark and PySpark expertise for scalable pipeline development
• Advanced SQL skills for transformation, modeling, and performance tuning
• Upper-Intermediate English proficiency (B2) for leading technical discussions

Nice to have

• Data architecture experience for enterprise lakehouse patterns
• Data solution architecture skills to define scalable platform designs
• Scala proficiency for Spark development and optimization