Senior Azure Data Engineer

MetLife; Hyderabad; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

MetLife is seeking a Senior Azure Data Engineer / Data Engineering Lead to design, build, and lead delivery of enterprise-scale cloud data platforms. The ideal candidate brings hands-on depth across Azure, Databricks, Spark, Data Lakehouse, and modern data pipelines, while guiding engineers and collaborating with stakeholders in an Agile and DevOps environment.

You will lead a team of data engineers, drive architectural decisions, ensure data quality, governance, and observability, and deliver

Qualifications

• Bachelor’s degree in Computer Science, Engineering or equivalent practical experience.
• 8+ years of experience in data engineering and cloud data platform development.
• 4+ years of hands-on Azure Databricks, Spark, PySpark and Lakehouse architecture.
• Leadership experience guiding engineers and delivering high-quality solutions.
• Strong knowledge of Azure data services and security practices.

Responsibilities

• Architect and deliver scalable data lakehouse solutions using Azure Data Lake Storage, Azure Databricks, Delta Lake and Unity Catalog.
• Design curated data products, data marts, and analytics-ready datasets for analysts and data scientists.
• Build batch and near-real-time pipelines with Azure Databricks, Spark, PySpark, SQL, and Azure data services.
• Lead and mentor a team of data engineers, set standards, and guide implementation decisions.
• Implement CI/CD for data workloads and promote governance, security, and observability.

Skills

Python

PySpark

SQL

Spark SQL

Data modeling

ETL/ELT design

orchestration

data quality

metadata management

performance tuning

Education

Bachelor’s degree in Computer Science, Engineering, IT, Data Engineering, or equivalent

Tools

Azure Databricks

Azure Data Factory

Azure Synapse Analytics

Azure Data Lake Storage

Azure SQL

Cosmos DB

Delta Lake

Unity Catalog

GitHub Actions

Azure Monitor

Job description

Job Description:

We’re Hiring: Senior Azure Data Engineer / Data Engineering Lead

Role: Senior Azure Data Engineer / Data Engineering Lead | Experience: 8+ Years | Level: Senior Individual Contributor

About The Role

We are looking for a senior Azure Data Engineering professional who can design, build, and lead delivery of enterprise-scale cloud data platforms. The ideal candidate will bring strong hands‑on engineering depth across Azure, Databricks, Spark, Data Lakehouse, and modern data pipelines, while also guiding engineers, collaborating with business stakeholders, and driving high-quality delivery in an Agile and DevOps environment.

What You’ll Lead And Deliver

• Modern Data Platform & Lakehouse Engineering:
• Architect and deliver scalable data lakehouse solutions using Azure Data Lake Storage, Azure Databricks, Delta Lake, Unity Catalog, and Medallion architecture.
• Design curated data products, data marts, and analytics‑ready datasets for business analysts, data scientists, and enterprise reporting teams.
• Apply strong data modeling practices, partitioning strategies, schema evolution, performance tuning, and cost optimization.
• Data Pipelines, ETL/ELT & Streaming:
• Build reliable batch and near‑real‑time pipelines using Azure Databricks, Apache Spark, PySpark, SQL, Azure Data Factory, Synapse, and related Azure services.
• Implement ingestion frameworks for structured, semi‑structured, and unstructured data from multiple enterprise sources.
• Ensure data quality, lineage, validation, observability, error handling, and reusable pipeline patterns.
• Engineering Leadership:
• Lead and mentor a team of data engineers, set engineering standards, review designs, and guide implementation decisions.
• Collaborate with architects, product owners, business stakeholders, data scientists, security, platform, and operations teams.
• Drive Agile delivery, estimation, sprint planning, technical roadmaps, and continuous improvement across the team.
• DevOps, Security & Governance:
• Implement CI/CD for data workloads using Azure DevOps or GitHub Actions, including automated testing, deployment, and release governance.
• Apply secure engineering practices using Microsoft Entra ID, role‑based access control, secrets management, and data governance controls.
• Promote monitoring, alerting, operational readiness, and SRE‑aligned practices for production data platforms.

Candidate Profile

• Education: Bachelor’s degree in Computer Science, Engineering, Information Technology, Data Engineering, or equivalent practical experience.
• Experience:
• 8+ years of experience in data engineering, data platform engineering, ETL/ELT, BI, analytics, or cloud data application development.
• 4+ years of hands‑on experience designing and delivering cloud‑based data platforms on Microsoft Azure.
• 2+ years of strong experience with Azure Databricks, Apache Spark, PySpark, Spark SQL, Delta Lake, and Lakehouse architecture.
• Proven experience leading engineers, reviewing architecture/design, owning delivery outcomes, and driving technical excellence.
• Core Azure Skills:
• Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, Azure SQL, Dedicated SQL Pool, Cosmos DB, Logic Apps, Azure Monitor, and Application Insights.
• Experience with Microsoft Entra ID, access control, secure connectivity, key management, and enterprise data governance.
• Azure Databricks Skills:
• Hands‑on expertise in Databricks workspaces, notebooks, jobs/workflows, clusters/serverless compute, Delta tables, Auto Loader, Delta Live Tables or Lakeflow‑style declarative pipelines.
• Strong understanding of Unity Catalog, catalog/schema/table governance, data lineage, access controls, data quality, and lakehouse security practices.
• Experience optimizing Spark jobs, Delta Lake performance, partitioning, Z‑ordering/clustering strategies, workload monitoring, and cost‑efficient compute usage.

Skills & Competencies

• Technical Skills:
• Python, PySpark, SQL, Spark SQL, data modeling, ETL/ELT design, orchestration, data quality, metadata management, and performance tuning.
• Modern lakehouse patterns including Bronze/Silver/Gold layers, Delta Lake, schema evolution, slowly changing dimensions, and reusable ingestion frameworks.
• CI/CD, Git, Azure DevOps, automated testing, environment configuration, release pipelines, SonarQube, and secure coding practices.
• Monitoring and observability using Azure Monitor, Application Insights, logs, alerts, pipeline health checks, and operational dashboards.
• Leadership & Collaboration:
• Ability to lead engineers, mentor junior team members, influence design decisions, and promote engineering best practices.
• Strong communication skills with the ability to work across global, multi-cultural teams and translate business needs into scalable technical solutions.
• Comfortable working in a fast‑paced Agile environment with ownership mindset, delivery focus, and continuous improvement culture.
• Certifications Preferred / Add‑ons:
• Microsoft Certified: Azure Data Engineer Associate.
• Microsoft Certified: Azure Databricks Data Engineer Associate.
• Databricks Certified Data Engineer Associate or Professional.
• Azure Solutions Architect, Azure Developer, or relevant cloud/data engineering certifications are a plus.

Why Join Us?

Be part of MetLife’s technology transformation journey and help build modern, secure, scalable, and governed data platforms that create meaningful business impact. If you are passionate about Azure, Databricks, data engineering, and leading teams to deliver high‑quality solutions, we would love to connect.