Big Data Engineer II

MetLif; Pune District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

The Data Engineer role at MetLife in Pune District, India focuses on designing, building, and maintaining scalable data platforms and pipelines across cloud and on-premises environments. It emphasizes production-grade data solutions, governance, security, and data quality to enable analytics and business insights.

You will work with Azure, Databricks, Apache Spark, and real-time processing while collaborating with cross-functional teams such as Business, Operations, and BI.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or equivalent qualification.
• 5-8+ years of experience in Data Engineering, Big Data, Data Integration, or related disciplines.
• Hands-on expertise in SQL and Python and/or Scala for data engineering and application development.
• Strong experience in ETL/ELT pipeline development and large-scale data processing.
• Experience with Big Data technologies including Apache Spark, Hadoop, and Hive.
• Proficiency with Azure Data Engineering services such as Azure Data Factory, Azure Databricks, Synapse Analytics, Event Hub, and Azure Functions.
• Knowledge of distributed and NoSQL databases such as Cosmos DB and HBase.
• Experience designing and supporting Data Lakes, Data Warehouses, Data Marts, and Medallion Architecture.
• Strong understanding of performance tuning, optimization techniques, and data quality validation.
• Experience in both batch and real-time streaming data processing.
• Excellent communication, analytical thinking, stakeholder management, and structured problem-solving skills.
• Ability to work independently in an individual contributor role while collaborating effectively with diverse teams.

Responsibilities

• Design, build, and maintain robust ETL/ELT pipelines on Azure cloud and on-premises platforms for batch and real-time data processing.
• Develop scalable data solutions to collect, ingest, transform, and store large volumes of structured and unstructured data.
• Monitor, optimize, and troubleshoot data pipelines to ensure high availability, reliability, scalability, and performance.
• Ensure adherence to data governance, security, privacy, compliance, and quality standards across all data engineering solutions.
• Develop and implement data architectures including data lakes, data warehouses, data marts, and Medallion architecture frameworks.
• Build real-time and batch data processing solutions using Apache Spark and streaming technologies.
• Collaborate with stakeholders across Business, Operations, Technology, Data Governance, Data Quality, Data Architecture, Data Science, DevOps, and BI teams.
• Support data platform modernization initiatives and continuous improvement of engineering practices.

Skills

SQL

Python/Scala

ETL/ELT

Big Data

Apache Spark

Hadoop/Hive

Azure Data Factory

Azure Databricks

Synapse Analytics

Cosmos DB

HBase

Data Lakes

Data Warehouses

Medallion Architecture

Streaming

Education

Bachelor's degree in CS/IT/Engineering

Tools

Azure

Databricks

Apache Spark

Hadoop

CI/CD (Azure DevOps)

Job description

The Data Engineer is responsible for designing, building, and maintaining scalable data engineering solutions that enable reliable, secure, and high-quality data for analytics, reporting, and business decision-making. This role plays a critical part in the data and analytics lifecycle by developing production-grade data platforms and pipelines across cloud and on-premises environments.

Guided by a data-driven and innovation-focused approach, this role contributes to transforming raw data into valuable business insights through modern data engineering practices, cloud technologies, and big data frameworks. The position offers an exciting opportunity to work with cutting-edge technologies such as Azure, Databricks, Apache Spark, and real-time data processing solutions while collaborating with cross-functional teams across Business, Technology, Operations, and Data & Analytics functions.

Key Responsibilities

• Design, build, and maintain robust ETL/ELT pipelines on Azure cloud and on-premises platforms for batch and real-time data processing.
• Develop scalable data solutions to collect, ingest, transform, and store large volumes of structured and unstructured data.
• Monitor, optimize, and troubleshoot data pipelines to ensure high availability, reliability, scalability, and performance.
• Ensure adherence to data governance, security, privacy, compliance, and quality standards across all data engineering solutions.
• Develop and implement data architectures including data lakes, data warehouses, data marts, and Medallion architecture frameworks.
• Build real-time and batch data processing solutions using Apache Spark and streaming technologies.
• Collaborate with stakeholders across Business, Operations, Technology, Data Governance, Data Quality, Data Architecture, Data Science, DevOps, and BI teams.
• Support data platform modernization initiatives and continuous improvement of engineering practices.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or an equivalent educational qualification.
• 5-8+ years of experience in Data Engineering, Big Data, Data Integration, or related disciplines.
• Hands-on expertise in SQL and Python and/or Scala for data engineering and application development.
• Strong experience in ETL/ELT pipeline development and large-scale data processing.
• Experience with Big Data technologies including Apache Spark, Hadoop, and Hive.
• Proficiency with Azure Data Engineering services such as Azure Data Factory, Azure Databricks, Synapse Analytics, Event Hub, and Azure Functions.
• Knowledge of distributed and NoSQL databases such as Cosmos DB and HBase.
• Experience designing and supporting Data Lakes, Data Warehouses, Data Marts, and Medallion Architecture.
• Strong understanding of performance tuning, optimization techniques, and data quality validation.
• Experience in both batch and real-time streaming data processing.
• Excellent communication, analytical thinking, stakeholder management, and structured problem-solving skills.
• Ability to work independently in an individual contributor role while collaborating effectively with diverse teams.

Preferred Qualifications

• Experience with DevOps practices including Git, Azure DevOps, and CI/CD pipeline implementation.
• Knowledge of Unix Shell Scripting, MongoDB, and Apache NiFi.
• Exposure to Generative AI technologies, tools, and AI-driven data solutions.
• Experience in cloud migration and modern data platform implementation projects.
• Strong business acumen and ability to translate business requirements into scalable data solutions.
• Relevant cloud or data engineering certifications on Azure will be an advantage.

Experience

5-8+ Years of Relevant Experience

Education

Bachelor's Degree in Computer Science, Information Technology, Engineering, or Equivalent Qualification.

About MetLife Recognized on Fortune magazine's list of the "World's Most Admired Companies" and Fortune World’s 25 Best Workplaces™, MetLife, through its subsidiaries and affiliates, is one of the world’s leading financial services companies; providing insurance, annuities, employee benefits and asset management to individual and institutional customers. With operations in more than 40 markets, we hold leading positions in the United States, Latin America, Asia, Europe, and the Middle East. As part of our New Frontier strategy, MetLife is building an AI-enabled, people-centered future. We’re looking for people who bring curiosity, adaptability, and a growth mindset as we use AI to enhance how we serve customers, support communities, and evolve the way work gets done. At MetLife, AI is a responsible partner that supports human judgment, creativity, and continuous improvement while helping us build trust, inclusion, and long-term value. Our purpose is simple - to help our colleagues, customers, communities, and the world at large create a more confident future. United by purpose and guided by our core values - Win Together, Do the Right Thing, Deliver Impact Over Activity, and Think Ahead - we’re inspired to transform the next century in financial services. At MetLife, it’s #AllTogetherPossible