Lead Data Engineer

Mastercard Inc.; Pune District; Presencial; INR 2.400.000 - 3.800.000; Jornada completa

Descripción de la vacante

Mastercard is seeking a Lead Data Engineer to join the AI & DPE Data Engineering platform team. You will build and evolve large-scale backend data systems, real-time and batch pipelines, and AI-enabled services powering analytics and automation.

The role focuses on Scala/Python/Java, distributed platforms like Cloudera and Spark, and cloud architectures with emphasis on AI agent creation. You will handle massive datasets and modern cloud tools to operationalize data at global scale.

Formación

• BS/BA in Computer Science, Engineering, Information Systems, or related field.
• Strong hands-on experience with Scala, Python, and Java in backend or data-intensive systems.
• Experience with CDP and Spark, and familiarity with Cloudera Manager.
• Understanding of data modeling, distributed systems, and large-scale data processing.
• Ability to work independently in complex environments.

Responsabilidades

• Design, develop, and maintain backend services and data pipelines using Scala, Python, and Java.
• Build and optimize batch and streaming workloads on CDP using Spark.
• Work with Cloudera Manager for configuration, monitoring, and performance tuning.
• Design datamarts and curated datasets ensuring data integrity and performance.
• Lead AI agent concepts and automation within backend systems.

Conocimientos

Scala

Python

Java

Distributed systems

Analytical thinking

Educación

BS/BA in Computer Science or related field

Herramientas

Cloudera Data Platform (CDP)

Spark

Databricks

AWS

Kafka

Hadoop/Hive/Impala

ETL Informatica

Descripción del empleo

Our Purpose

Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we're helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.

Title And Summary

Lead Data Engineer

Position Overview

Have you ever wanted to be part of something BIG? Now is the time to make an immediate impact at a leading global technology company, Mastercard.

This role is part of the AI&DPE Data Engineering platform team, responsible for building and evolving large scale backend data systems, real time and batch pipelines, and AI enabled services that power analytics, decisioning, and automation across the organization.

This is a backend engineering role, focused on Scala/Python/Java, distributed data platforms (Cloudera/Spark), and cloud based architectures, with a strong emphasis on AI agent creation and intelligent automation. Visualization tools (e.g., Qlik) are consumers of the platform, not the core focus of this role.

You will work with massive transactional datasets, modern big data and cloud platforms, and AI driven workflows to transform how Mastercard processes, enriches, and operationalizes data at global scale.

Primary Responsibilities

Backend & Data Platform Engineering

• Design, develop, and maintain backend services and data pipelines using Scala, Python and Java
• Build and optimize batch and streaming workloads on Cloudera Data Platform (CDP) using Spark
• Work with Cloudera Manager to support platform configuration, monitoring, performance tuning, and operational stability
• Design and implement high quality datamarts and curated datasets with strong emphasis on data integrity, performance, and reliability

AI Agents & Intelligent Automation

• Anomaly detection and operational intelligence
• workflow automation
• Apply generative AI driven, or rule based agents to reduce manual effort and improve scalability across backend systems

Cloud & Modern Data Architecture

• Build and support data and compute workloads in AWS environments
• Leverage Databricks for large scale data processing, advanced analytics
• Contribute to cloud native and hybrid architectures integrating on prem and cloud platforms

Integration & Downstream Enablement

• Enable downstream consumers (analytics, visualization, reporting tools such as Qlik) through well designed, reliable backend data interfaces
• Partner with analytics, fraud, and business teams to ensure backend systems meet evolving needs without compromising platform stability

Documentation & Collaboration

• Create clear technical documentation, including architecture diagrams, data flows, and design specifications
• Participate in Agile/Scrum ceremonies and cross functional design reviews
• Mentor and upskill team members in backend engineering, big data, and AI agent concepts

Required

KNOWLEDGE AND SKILL REQUIREMENTS

• BS/BA degree in Computer Science, Engineering, Information Systems, or related field
• Strong handson experience with Scala,Python/Java in backend or data intensive systems
• Experience working with Cloudera Data Platform (CDP) and Spark
• Familiarity with Cloudera Manager for cluster administration, monitoring, or troubleshooting
• Strong understanding of data modeling concepts, distributed systems, and large scale data processing
• Excellent problem solving skills and ability to work independently in complex environments

Good To Have / Strongly Preferred

• Hands on experience building, integrating, or supporting AI driven agents or intelligent automation solutions
• Experience with Databricks for data engineering or ML workloads
• Experience working in AWS (e.g., S3, EC2, EMR, Glue, Lambda, IAM, or equivalent services)

Knowledge Of Streaming And Big-data Technologies

• Kafka
• Hadoop ecosystem
• Hive/Impala
• Exposure to model monitoring, or AI platform enablement
• Experience with ETL tools such as Informatica
• Experience working in Agile / Scrum teams within large enterprises

Corporate Security Responsibility

• Abide by Mastercard's security policies and practices;
• Ensure the confidentiality and integrity of the information being accessed;
• Report any suspected information security violation or breach, and
• Complete all periodic mandatory security trainings in accordance with Mastercard's guidelines.