Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow)

Mastercard; Maharashtra; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Mastercard Foundry R&D seeks a Principal Data Engineer to lead architecture and engineering of scalable data platforms, enabling analytics, ML, and AI workloads. You will design pipelines, lakehouse architectures, and governance while mentoring teams across cross-functional groups.

The role requires deep experience in Python, PySpark, SQL, and modern data architectures, with strong leadership and collaboration skills in a fast-paced R&D environment.

Qualifications

• 12–18 years of experience in enterprise data engineering.
• Experience designing and operating large-scale production data platforms.
• Strong programming in Python and PySpark with CI/CD practices.
• Advanced SQL, data modeling, and large-scale analytics.

Responsibilities

• Lead architecture and engineering of scalable data platforms (batch/streaming).
• Build data pipelines for ingestion, transformation, and consumption.
• Develop lakehouse architectures using cloud-native tech and distributed processing.
• Operate platforms supporting analytics, ML, and AI workloads.
• Create reusable data products, APIs, and self-service capabilities.

Skills

Python

PySpark

SQL

Data modeling

Distributed processing

Cloud platforms

Education

Bachelor's degree in CS/Engineering

Tools

Apache Spark

Airflow

Databricks

Terraform

Docker

Kubernetes

Job description

Our Purpose

Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.

Title And Summary

Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow)

Overview

Mastercard Foundry R&D develops emerging technology solutions and transforms promising concepts into scalable products and platforms.

What You’ll Do

• Lead the architecture, design, and engineering of scalable batch, streaming, and event-driven data platforms.
• Design and build reliable data ingestion, transformation, enrichment, processing and consumption pipelines.
• Develop governed lakehouse and modern data platform architectures using cloud-native technologies and distributed data processing frameworks.
• Build and operate platforms that support analytics, machine learning, generative AI, and agentic AI workloads.
• Create reusable data products, services, APIs, and self-service capabilities that accelerate experimentation and production delivery.
• Enable the full AI and machine learning lifecycle through robust feature engineering, training, evaluation, deployment, and inference pipelines.
• Optimize data workloads for performance, scalability, reliability, resiliency, and cost efficiency.
• Define engineering standards for data modeling, software development, testing, CI/CD, observability, and operational excellence.
• Implement data governance capabilities including quality controls, lineage, metadata management, cataloging, access control, retention, and compliance.
• Apply security and privacy-by-design principles for sensitive and regulated data environments.
• Evaluate emerging data and AI technologies through prototypes, proof-of-concepts, and technical assessments.
• Translate loosely defined research, innovation or product requirements into practical architectures and incremental delivery plans.
• Provide hands‑on technical leadership through architecture reviews, code reviews, troubleshooting, and engineering mentorship.
• Partner with cross‑functional teams to mature successful R&D initiatives into enterprise‑grade production solutions.
• Communicate technical decisions, trade‑offs, risks, dependencies, and roadmap recommendations to both technical and business stakeholders.
• Drive adoption of modern data engineering practices, platform automation, and platform reliability disciplines.

All About You

Required Qualifications

• Typically 12‑18 years of overall career relevant experience, including significant ownership of complex enterprise data engineering solutions.
• Extensive experience designing, building, and operating large‑scale production data platforms.
• Expert programming skills in Python, PySpark, and modern software engineering practices.
• Advanced SQL expertise, including data modeling, performance tuning, query optimization, and large‑scale analytical processing.
• Strong hands‑on experience with distributed data processing technologies such as Apache Spark and modern lakehouse platforms.
• Experience building and operating cloud‑native data platforms in AWS, or Azure, or other enterprise cloud environments.
• Strong knowledge of modern data architecture patterns, including data lakes, lakehouses, data meshes, data products, and event‑driven architectures.
• Experience with cloud storage, data integration, streaming, serverless, observability, and security services across public cloud platforms.
• Experience building scalable batch and real‑time data pipelines.
• Experience with workflow orchestration platforms such as Apache Airflow, Databricks Workflows, AWS Step Functions, Azure Data Factory, or similar technologies.
• Experience implementing CI/CD, automated testing, version control, Infrastructure as Code, and platform automation.
• Strong understanding of data governance, metadata management, lineage, quality frameworks, privacy controls, and access management.
• Experience diagnosing and resolving complex performance, reliability, scalability, and operational challenges.
• Ability to make sound architectural decisions while balancing delivery speed, innovation, maintainability, security, and cost.
• Proven ability to thrive in R&D and innovation‑focused environments where priorities and requirements may evolve through experimentation.
• Strong communication, collaboration, technical leadership, and mentoring skills.
• Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related technical discipline, or equivalent practical experience.

Preferred Qualifications

• Experience supporting machine learning, generative AI, or agentic AI systems.
• Experience with MLflow or comparable ML lifecycle tooling.
• Experience with Azure Machine Learning, Azure AI Foundry, or Microsoft Fabric.
• Experience with Unity Catalog, Microsoft Purview, or comparable catalog and governance platforms.
• Experience with Kafka or other event‑streaming technologies.
• Experience with containerized and cloud‑native platforms, including Docker and Kubernetes.
• Experience with Terraform or another Infrastructure as Code framework.
• Experience integrating structured, semi‑structured, unstructured, and streaming data.
• Knowledge of responsible AI, model evaluation, and AI platform observability.
• Experience in payments, financial services, or another regulated industry.

Success in This Role

Success Requires Someone Who

• Remains strongly hands‑on while providing technical direction.
• Can build production‑quality systems without introducing unnecessary platform complexity.
• Converts experimentation into reusable engineering capabilities.
• Designs for security, governance, reliability, and observability from the outset.
• Challenges assumptions and validates architectural choices through evidence and prototypes.
• Enables AI and product teams rather than taking ownership of data science or model research.
• Influences across teams without depending on formal people‑management authority.

Corporate Security Responsibility

• Abide by Mastercard’s security policies and practices;
• Ensure the confidentiality and integrity of the information being accessed;
• Report any suspected information security violation or breach, and
• Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.