Big Data Engineer II

MetLife; Pune District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

MetLife is seeking a Data Engineer to design, build, and maintain scalable data pipelines across Azure cloud and on-prem environments. You will enable secure, high-quality data for analytics, reporting, and decision-making, leveraging Azure Data Factory, Databricks, and Spark in both batch and real-time modes.

The role requires 5–8+ years in data engineering, strong SQL/Python/Scala, and experience with data lakes, warehouses, and medallion architectures.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or an equivalent qualification.
• 5-8+ years of experience in Data Engineering, Big Data, Data Integration, or related disciplines.
• Hands-on expertise in SQL and Python and/or Scala for data engineering and application development.
• Strong experience in ETL/ELT pipeline development and large-scale data processing.
• Experience with Big Data technologies including Apache Spark, Hadoop, and Hive.
• Proficiency with Azure Data Engineering services such as Azure Data Factory, Azure Databricks, Synapse Analytics, Event Hub, and Azure Functions.
• Knowledge of distributed and NoSQL databases such as Cosmos DB and HBase.
• Experience designing and supporting Data Lakes, Data Warehouses, Data Marts, and Medallion Architecture.
• Strong understanding of performance tuning, optimisation techniques, and data quality validation.
• Experience in both batch and real-time streaming data processing.
• Excellent communication, analytical thinking, stakeholder management, and structured problem-solving skills.
• Ability to work independently in an individual contributor role while collaborating effectively with diverse teams.

Responsibilities

• Design, build, and maintain robust ETL/ELT pipelines on Azure cloud and on-premises platforms for batch and real-time data processing.
• Develop scalable data solutions to collect, ingest, transform, and store large volumes of structured and unstructured data.
• Monitor, optimise, and troubleshoot data pipelines to ensure high availability, reliability, scalability, and performance.
• Ensure adherence to data governance, security, privacy, compliance, and quality standards across all data engineering solutions.
• Develop and implementing data architectures including data lakes, data warehouses, data marts, and Medallion architecture frameworks.
• Build real-time and batch data processing solutions using Apache Spark and streaming technologies.
• Collaborate with stakeholders across Business, Operations, Technology, Data Governance, Data Quality, Data Architecture, Data Science, DevOps, and BI teams.
• Support data platform modernization initiatives and continuous improvement of engineering practices.

Skills

SQL

Python/Scala

ETL/ELT pipelines

Big Data processing

Azure Data Factory/Databricks/Synapse

NoSQL (Cosmos DB/HBase)

Batch & real-time streaming

DevOps concepts (Git/Azure DevOps/CICD

Education

Bachelor's degree in CS/IT/Engineering or equivalent

Tools

Azure Data Factory

Azure Databricks

Synapse Analytics

Event Hub/Azure Functions

Apache Spark

Apache NiFi

MongoDB

Unix Shell Scripting

Job description

Data Engineer (GG10.2)

The Opportunity

The Data Engineer is responsible for designing, building, and maintaining scalable data engineering solutions that enable reliable, secure, and high-quality data for analytics, reporting, and business decision-making. This role plays a critical part in the data and analytics lifecycle by developing production-grade data platforms and pipelines across cloud and on-premises environments. Guided by a data-driven and innovation-focused approach, this role contributes to transforming raw data into valuable business insights through modern data engineering practices, cloud technologies, and big data frameworks. The position offers an exciting opportunity to work with cutting-edge technologies such as Azure, Databricks, Apache Spark, and real-time data processing solutions while collaborating with cross-functional teams across Business, Technology, Operations, and Data & Analytics functions.

Key Responsibilities

• Design, build, and maintain robust ETL/ELT pipelines on Azure cloud and on-premises platforms for batch and real-time data processing.
• Develop scalable data solutions to collect, ingest, transform, and store large volumes of structured and unstructured data.
• Monitor, optimise, and troubleshoot data pipelines to ensure high availability, reliability, scalability, and performance.
• Ensure adherence to data governance, security, privacy, compliance, and quality standards across all data engineering solutions.
• Develop and implementing data architectures including data lakes, data warehouses, data marts, and Medallion architecture frameworks.
• Build real-time and batch data processing solutions using Apache Spark and streaming technologies.
• Collaborate with stakeholders across Business, Operations, Technology, Data Governance, Data Quality, Data Architecture, Data Science, DevOps, and BI teams.
• Support data platform modernization initiatives and continuous improvement of engineering practices.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or an equivalent educational qualification.
• 5-8+ years of experience in Data Engineering, Big Data, Data Integration, or related disciplines.
• Hands-on expertise in SQL and Python and/or Scala for data engineering and application development.
• Strong experience in ETL/ELT pipeline development and large-scale data processing.
• Experience with Big Data technologies including Apache Spark, Hadoop, and Hive.
• Proficiency with Azure Data Engineering services such as Azure Data Factory, Azure Databricks, Synapse Analytics, Event Hub, and Azure Functions.
• Knowledge of distributed and NoSQL databases such as Cosmos DB and HBase.
• Experience designing and supporting Data Lakes, Data Warehouses, Data Marts, and Medallion Architecture.
• Strong understanding of performance tuning, optimisation techniques, and data quality validation.
• Experience in both batch and real-time streaming data processing.
• Excellent communication, analytical thinking, stakeholder management, and structured problem-solving skills.
• Ability to work independently in an individual contributor role while collaborating effectively with diverse teams.

Preferred Qualifications

• Experience with DevOps practices including Git, Azure DevOps, and CI/CD pipeline implementation.
• Knowledge of Unix Shell Scripting, MongoDB, and Apache NiFi.
• Exposure to Generative AI technologies, tools, and AI-driven data solutions.
• Experience in cloud migration and modern data platform implementation projects.
• Strong business acumen and ability to translate business requirements into scalable data solutions.
• Relevant cloud or data engineering certifications on Azure will be an advantage.

Experience

5-8+ Years of Relevant Experience

Education

Bachelor's Degree in Computer Science, Information Technology, Engineering, or Equivalent Qualification.