Lead Cloud Engineer

Mastercard Inc.; Lisboa; Presencial; EUR 90 000 - 120 000; Tempo integral

Resumo da oferta

Mastercard is seeking a Lead Cloud Data Engineer to design and deliver cloud-native data platforms at scale, enabling AI-driven insights for global brands. You will lead data pipelines, orchestrate batch and streaming workloads, and guide modernization from Hadoop to cloud-native environments.

A strong background in Spark, Databricks, and Lakehouse concepts is essential. You will collaborate with architects, data scientists, and engineers to build secure, scalable, and cost-efficient data

Qualificações

• Hands-on experience with Big Data technologies including Hadoop, HDFS, Hive, and Spark.
• Proficiency in Python, Java, or Scala for large-scale data processing.
• Experience designing scalable data solutions and modern Lakehouse architectures.

Responsabilidades

• Design, develop, and support large-scale cloud-native data solutions on AWS and Databricks.
• Build batch, streaming, and real-time data pipelines for analytics, reporting, ML, and AI workloads.
• Engineer scalable distributed data processing systems using Apache Spark and related Big Data technologies.
• Design and implement modern Lakehouse architectures leveraging Databricks, Delta Lake, and Apache Iceberg.
• Lead modernization initiatives migrating data workloads from Hadoop-based platforms to cloud-native architectures.
• Develop reusable frameworks and engineering patterns to improve scalability, reliability, and operational efficiency.
• Collaborate with Architects, Product Managers, Data Scientists, and Software Engineers to define and implement cloud data solutions.
• Evaluate technologies and approaches for data ingestion, transformation, storage, governance, and consumption.
• Implement best practices around security, governance, performance optimization, observability, and cost management.
• Drive automation using Infrastructure-as-Code, CI/CD pipelines, and modern software engineering practices.

Conhecimentos

Big Data

Hadoop

HDFS

Hive

Spark

Python

Java

Scala

Data modeling

Schema design

Partitioning

Query optimization

Performance tuning

Security

Governance

Agile/Scrum

Communication

Formação académica

BS/MS degree in Computer Science, Software Engineering, Information Systems, or a related field

Ferramentas

Databricks

Spark

Delta Lake

Apache Iceberg

Hadoop

HDFS

Hive

Kafka

Flink

Terraform

Descrição da oferta de emprego

Our Purpose Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.

Lead Cloud Engineer Overview

Interested in building the next generation cloud data ecosystem that processes petabytes of data and powers AI-driven insights for some of the world's largest brands? The Marketing Services Technology team is transforming its analytics and data platform from traditional on-premises Hadoop environments to a modern cloud-native architecture powered by AWS, Databricks, Apache Spark, Iceberg, and AI technologies. We are looking for a Lead Cloud Data Engineer who is passionate about cloud engineering, Big Data, distributed systems, and modern data platforms. This role works at the intersection of Cloud, Data Engineering, Analytics, and AI. You will design and deliver scalable, secure, and high-performing data solutions that support advanced analytics, machine learning, and business growth at global scale. The ideal candidate has hands-on experience with modern cloud technologies as well as large-scale Big Data systems, including Hadoop and Spark, and is equally comfortable architecting cloud-native solutions, developing data pipelines, optimizing distributed workloads, and driving the modernization of enterprise data platforms.

Role

• Design, develop, and support large-scale cloud-native data solutions on AWS and Databricks.
• Build and maintain batch, streaming, and real-time data pipelines supporting analytics, reporting, machine learning, and AI workloads.
• Engineer scalable distributed data processing systems using Apache Spark and related Big Data technologies.
• Design and implement modern Lakehouse architectures leveraging Databricks, Delta Lake, and Apache Iceberg.
• Lead modernization initiatives migrating data workloads from Hadoop-based platforms to cloud-native architectures.
• Develop reusable frameworks and engineering patterns that improve scalability, reliability, and operational efficiency.
• Partner with Architects, Product Managers, Data Scientists, and Software Engineers to define and implement cloud data solutions.
• Evaluate technologies and approaches for data ingestion, transformation, storage, governance, and consumption.
• Implement best practices around security, governance, performance optimization, observability, and cost management.
• Drive automation using Infrastructure-as-Code, CI/CD pipelines, and modern software engineering practices.
• Contribute to the evolution of Marketing Services' cloud and data strategy while enabling future AI and machine learning capabilities.

All About You

• Strong hands-on experience with Big Data technologies including Hadoop, HDFS, Hive, Spark, and related distributed computing frameworks.
• Deep expertise in cloud data engineering on AWS, including storage, compute, networking, security, and scalable data architectures.
• Experience designing and developing modern data solutions using Databricks, Spark, Delta Lake, and cloud-native technologies.
• Proficiency in Python, Java, or Scala, building large-scale distributed data processing applications.
• Experience building robust batch and streaming pipelines supporting high-volume and high-velocity workloads.
• Understanding of modern Lakehouse architecture principles and open-table formats such as Apache Iceberg and Delta Lake.
• Skilled in data modelling, schema design, partitioning strategies, query optimization, and performance tuning.
• Familiarity with enterprise-scale data governance, security, privacy, and compliance, working with structured and unstructured datasets at scale.
• Experience with Infrastructure-as-Code (e.g., Terraform), modern CI/CD, and cloud-native monitoring and observability practices.
• Strong analytical and problem-solving skills, a passion for automation, and the ability to assess emerging technologies and recommend scalable, cost-effective solutions.
• Excellent communication and collaboration across technical and business teams; comfortable in Agile/Scrum environments managing multiple priorities.
• BS/MS degree in Computer Science, Software Engineering, Information Systems, or a related field.

Preferred Qualifications

• Experience with Databricks on AWS in enterprise production environments.
• Experience migrating workloads from Hadoop or Cloudera ecosystems to modern cloud architectures.
• Hands-on experience with Apache Kafka, Flink, or other event-streaming technologies.
• Experience supporting Machine Learning, MLOps, or AI-driven data platforms.
• AWS and/or Databricks certifications.
• Familiarity with FinOps and cloud cost optimization practices.
• Experience working with petabyte-scale data environments.

Corporate Security Responsibility

All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must: Abide by Mastercard’s security policies and practices; Ensure the confidentiality and integrity of the information being accessed; Report any suspected information security violation or breach, and Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.

Everyone wants easier ways to pay; we invent them. Checkout lines are slow; we speed them along. Merchants want more sales; we give them data and insights. People need financial access; we connect them. Corporate purchasing is complicated; we make it simple. Commuters are busy; we speed them on their way. Governments need greater efficiencies; we help create them. Small businesses are virtual; we give them access to a world of buyers. Retailers want to fight fraud; we provide the tools.