Senior Data Engineer: Azure Spark & AI Data Pipelines Lead

EPAM Systems Inc; United States; Remote; USD 140,000 - 180,000; Full time

Job summary

EPAM Systems is seeking an experienced Senior Data Software Engineer to lead Azure-based Spark data pipelines, driving scalable transformation workloads and data quality.

You will mentor engineers, contribute to architecture decisions, collaborate with data scientists, and implement governance across enterprise data platforms. Candidates should have 3+ years in data engineering and strong Azure Spark experience.

Qualifications

• 3+ years of data engineering experience in enterprise or financial services environments.
• Expertise in Azure data services including Data Factory, Databricks, and Synapse.
• Strong Spark ecosystem skills: PySpark, Spark SQL, Scala Spark.
• Ability to design end-to-end data pipelines for ingestion, transformation, storage, and analytics.
• Knowledge of data quality, security, and governance across large platforms.
• Ability to contribute to architecture decisions and mentor engineers.
• English proficiency at B2 or higher.

Responsibilities

• Lead design, development, and optimization of scalable data solutions on Azure using Spark-based processing.
• Own end-to-end data transformation pipelines from ingestion to analytics.
• Collaborate with Data Factory, Databricks, and Synapse services.
• Support high-performance data access patterns using Cosmos DB.
• Mentor junior engineers and establish coding standards.
• Ensure data quality, security, and governance across platforms.
• Contribute to technical decision-making and solution architecture discussions.

Skills

Azure Cloud

Spark Ecosystem

Data pipelines

Mentoring

Tools

Azure Data Factory

Databricks

Synapse

Cosmos DB

Spark SQL

PySpark

Scala Spark

Job description

EPAM Systems is seeking an experienced Senior Data Software Engineer to lead Azure-based Spark data pipelines, driving scalable transformation workloads and data quality.

You will mentor engineers, contribute to architecture decisions, collaborate with data scientists, and implement governance across enterprise data platforms. Candidates should have 3+ years in data engineering and strong Azure Spark experience.