Senior Data Software Engineer/ AI Agents, Azure, Spark

EPAM Systems Inc; United States; Remote; USD 140,000 - 180,000; Full time

Job summary

EPAM Systems is seeking an experienced Senior Data Software Engineer to lead Azure-based Spark data pipelines, driving scalable transformation workloads and data quality.

You will mentor engineers, contribute to architecture decisions, collaborate with data scientists, and implement governance across enterprise data platforms. Candidates should have 3+ years in data engineering and strong Azure Spark experience.

Qualifications

• 3+ years of data engineering experience in enterprise or financial services environments.
• Expertise in Azure data services including Data Factory, Databricks, and Synapse.
• Strong Spark ecosystem skills: PySpark, Spark SQL, Scala Spark.
• Ability to design end-to-end data pipelines for ingestion, transformation, storage, and analytics.
• Knowledge of data quality, security, and governance across large platforms.
• Ability to contribute to architecture decisions and mentor engineers.
• English proficiency at B2 or higher.

Responsibilities

• Lead design, development, and optimization of scalable data solutions on Azure using Spark-based processing.
• Own end-to-end data transformation pipelines from ingestion to analytics.
• Collaborate with Data Factory, Databricks, and Synapse services.
• Support high-performance data access patterns using Cosmos DB.
• Mentor junior engineers and establish coding standards.
• Ensure data quality, security, and governance across platforms.
• Contribute to technical decision-making and solution architecture discussions.

Skills

Azure Cloud

Spark Ecosystem

Data pipelines

Mentoring

Tools

Azure Data Factory

Databricks

Synapse

Cosmos DB

Spark SQL

PySpark

Scala Spark

Job description

We are looking for an experienced Senior Data Software Engineer with strong hands-on expertise in Azure and the Spark ecosystem , primarily focused on building and maintaining data transformation pipelines. The candidate should be comfortable with any Spark-adjacent technology (PySpark, Spark SQL, Scala Spark, Databricks, Synapse, etc.) rather than being locked into one specific flavor. Experience with Microsoft Fabric is a strong plus but not a requirement. The ideal candidate combines great technical skills with leadership capability, contributing to architecture, design, development, and mentoring of engineering teams — preferably in complex enterprise or financial services environments.

Responsibilities

• Lead the design, development, and optimization of scalable data engineering solutions on Azure, using Spark-based processing (PySpark, Spark SQL, Scala, or equivalent)
• Own end-to-end data transformation pipelines, including ingestion, transformation, storage, and analytics
• Work with Azure-native data services such as Data Factory, Databricks, and Synapse
• Support high-performance data access patterns using Cosmos DB (NoSQL API) where applicable
• Collaborate with data scientists, AI engineers, and product stakeholders to enable data-driven analytics and insights
• Mentor and guide junior engineers, setting coding standards and best practices
• Ensure data quality, security, governance, and performance across platforms
• Contribute to technical decision-making and solution architecture discussions

Requirements

• 3+ years of experience in data engineering roles, preferably within complex enterprise or financial services environments
• Expertise in Azure cloud data services, including Data Factory, Databricks, and Synapse
• Proficiency in the Spark ecosystem, including PySpark, Spark SQL, and Scala Spark
• Background in designing and maintaining end-to-end data transformation pipelines covering ingestion, transformation, storage, and analytics
• Skills in ensuring data quality, security, and governance across large-scale platforms
• Capability to contribute to solution architecture and technical decision-making
• Competency in mentoring engineering teams and setting coding standards and best practices
• Understanding of high-performance data access patterns using Cosmos DB (NoSQL API)
• English proficiency at an Upper-Intermediate level (B2) or higher

Nice to have

• Hands-on experience with Microsoft Fabric and OneLake (Delta / OpenLake)
• Familiarity with financial instruments and financial services data
• Exposure to AI-assisted development tools such as GitHub Copilot and awareness of industry-standard LLMs
• Knowledge of Data Science fundamentals and collaboration experience with DS teams