Lead Data Engineer

Mastercard; Maharashtra; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

Mastercard is seeking a Lead Data Engineer in Maharashtra to design, build, and scale cloud-enabled data pipelines powering enterprise data platforms. You will lead end-to-end delivery, mentor engineers, and collaborate with product owners to deliver high-impact analytics capabilities.

You will work across on-premises and cloud environments, driving data quality, observability, and performance improvements while advancing Mastercard's data ecosystem strategy.

Qualifications

• 10-12 years of experience delivering enterprise-scale Data Warehouse, Data Lake, or Data Lakehouse solutions.
• Proven experience implementing multiple end-to-end data engineering projects within large-scale distributed computing environments.
• Hands-on experience migrating ETL and analytics workloads from on-premises platforms to cloud-native environments.

Responsibilities

• Design, develop, test, and deploy data pipelines using Spark/Scala/Java across Hadoop and cloud storage.
• Build batch and near real-time data processing frameworks for petabyte-scale workloads.
• Develop reusable components and frameworks to accelerate data product delivery while upholding standards.
• Design a build-once, run-anywhere architecture for on-prem and cloud deployments.
• Implement data lineage, metadata, data quality, and observability across the data ecosystem.
• Collaborate with architects to establish scalable design patterns and best practices.
• Migrate legacy ETL and data warehouse workloads to cloud-native architectures.
• Leverage AWS services (S3, EMR, Glue) to improve scalability and reliability.
• Lead end-to-end development activities and mentor junior engineers.
• Troubleshoot incidents and ensure governance and security standards.

Skills

Apache Spark

Scala

Java

Hadoop

Object Storage

Apache Airflow

Apache NiFi

Oracle

SQL Server

Cassandra

DynamoDB

AWS

Amazon S3

AWS Glue

EMR

Tools

Hadoop ecosystem tools

Job description

Lead Data Engineer

Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.

Overview

Mastercard powers the global economy by enabling secure, seamless, and intelligent payments across the world. Behind every transaction is a sophisticated technology ecosystem that processes billions of payment events with speed, resilience, and precision. As a Lead Data Engineer within the Data Collection & Engineering (DC&E) organisation, you will play a critical role in designing, building, and scaling Mastercard's next-generation data platforms and analytical ecosystems. You will develop high-performance, cloud-enabled data pipelines that power Mastercard's enterprise data warehouse and lakehouse environments, enabling advanced analytics, business intelligence, regulatory reporting, and data-driven decision making across the organisation. This role offers a unique opportunity to solve large-scale data engineering challenges, work with cutting-edge big data technologies, and contribute to a modern cloud transformation programme supporting global payment processing platforms.

Role

Data Engineering & Development

• Design, develop, test, and deploy high-quality, secure, scalable, and resilient data pipelines using Apache Spark, Java/Scala across Hadoop and cloud-native object storage platforms.
• Build and maintain batch and near real-time data processing frameworks capable of supporting petabyte-scale workloads.
• Develop reusable engineering components and frameworks that accelerate data product delivery while maintaining enterprise standards.

Architecture & Platform Engineering

• Design and implement a "build once, run anywhere" architecture supporting seamless deployment across on-premises and public cloud environments without code changes.
• Implement data lineage, metadata management, data cataloguing, data quality controls, and observability capabilities across the data ecosystem.
• Collaborate with architects and platform teams to establish scalable design patterns and engineering best practices.

Cloud Modernisation

• Contribute to migration initiatives moving legacy ETL and data warehouse workloads from on-premises environments to cloud-native architectures.
• Leverage cloud services such as Amazon S3, EMR, Glue, and related data services to improve scalability, reliability, and operational efficiency.
• Drive adoption of modern lakehouse and distributed compute architectures.

Delivery & Technical Leadership

• Lead end-to-end development activities including requirement analysis, solution design, coding, testing, deployment, and production support.
• Mentor and guide junior engineers through code reviews, technical coaching, and engineering best practices.
• Partner with product owners, analysts, architects, and business stakeholders to deliver high-quality solutions within committed timelines.

Operational Excellence

• Troubleshoot complex production incidents and perform root cause analysis to identify and implement long-term remediation strategies.
• Ensure compliance with Mastercard's engineering, security, quality assurance, and operational governance standards.
• Continuously identify opportunities to improve performance, automation, monitoring, and process efficiency.

Innovation

• Evaluate emerging data technologies and conduct proof-of-concept (POC) initiatives to determine their applicability within Mastercard's data ecosystem.
• Contribute to engineering innovation and continuous improvement initiatives across the organisation.

All About You

Required Experience

• 10-12 years of experience delivering enterprise-scale Data Warehouse, Data Lake, or Data Lakehouse solutions.
• Proven experience implementing multiple end-to-end data engineering projects within large-scale distributed computing environments.
• Hands-on experience migrating ETL and analytics workloads from on-premises platforms to cloud-native environments.

Technical Expertise

• Strong development experience using:Apache Spark, Scala or Java, Hadoop ecosystem technologies, Object Storage platforms
• Experience building orchestration and workflow solutions using: Apache Airflow/ Apache NiFi and Similar enterprise scheduling frameworks
• Strong SQL expertise and experience with Relational and NoSQL database technologies: Oracle/ SQL Server, Cassandra, Dynamo DB etc.
• Working knowledge of cloud platforms, preferably AWS: Amazon S3, EMR, AWS Glue, Cloud-native data services

Professional Skills

• Strong analytical and problem-solving capabilities.
• Experience operating within Agile delivery environments.
• Excellent written and verbal communication skills.
• Proven ability to collaborate within geographically distributed and matrix-based teams.
• Self-starter with strong ownership, accountability, and execution focus.
• Ability to learn emerging technologies quickly and apply them effectively to business challenges.

Corporate Security Responsibility

All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:

• Abide by Mastercard’s security policies and practices;
• Ensure the confidentiality and integrity of the information being accessed;
• Report any suspected information security violation or breach, and
• Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.