Senior Azure Data Engineer

MetLife; Hyderabad; On-site; INR 2,800,000 - 4,200,000; Full time

Job summary

MetLife in Hyderabad seeks a Senior Azure Data Engineer Lead to design, build, and lead delivery of enterprise-scale cloud data platforms on the Azure stack. You will guide a team of engineers, shape data lakehouse architectures, and collaborate with stakeholders to ensure high-quality, scalable solutions.

You will drive Databricks-based pipelines, Spark-based processing, and DevOps practices in an Agile environment, delivering secure, governed data platforms for enterprise reporting and

Qualifications

• Bachelor’s degree in CS, Engineering, IT or equivalent practical experience.
• 8+ years in data engineering, data platform engineering, ETL/ELT, BI, analytics or cloud data app development.
• 4+ years designing/delivering cloud data platforms on Microsoft Azure.
• 2+ years with Azure Databricks, Spark, PySpark, Spark SQL, Delta Lake, Lakehouse architecture.

Responsibilities

• Design, build, and lead delivery of enterprise-scale cloud data platforms on Azure.
• Architect scalable data lakehouse solutions using Azure Data Lake Storage, Databricks, Delta Lake, Unity Catalog, and Medallion architecture.
• Develop data products, data marts, and analytics-ready datasets for analysts, data scientists, and reporting teams.
• Lead and mentor a team of data engineers; set standards; review designs; guide implementation decisions.
• Implement CI/CD for data workloads using Azure DevOps or GitHub Actions; ensure governance and security.

Skills

Python

PySpark

SQL

Spark SQL

Data modeling

ETL/ELT design

Orchestration

Data quality

Metadata management

Performance tuning

Education

Bachelor’s degree in Computer Science, Engineering, Information Technology, Data Engineering, or equivalent practical experience

Tools

Azure Databricks

Azure Data Factory

Azure Synapse Analytics

Azure Data Lake Storage

Delta Lake

Unity Catalog

Cosmos DB

Job description

Role: Senior Azure Data Engineer / Data Engineering Lead | Experience: 8+ Years | Level: Senior Individual Contributor

About The Role

We are looking for a senior Azure Data Engineering professional who can design, build, and lead delivery of enterprise-scale cloud data platforms. The ideal candidate will bring strong hands-on engineering depth across Azure, Databricks, Spark, Data Lakehouse, and modern data pipelines, while also guiding engineers, collaborating with business stakeholders, and driving high-quality delivery in an Agile and DevOps environment.

What You’ll Lead And Deliver

• Modern Data Platform & Lakehouse Engineering:
• Architect and deliver scalable data lakehouse solutions using Azure Data Lake Storage, Azure Databricks, Delta Lake, Unity Catalog, and Medallion architecture.
• Design curated data products, data marts, and analytics-ready datasets for business analysts, data scientists, and enterprise reporting teams.
• Apply strong data modeling practices, partitioning strategies, schema evolution, performance tuning, and cost optimization.
• Data Pipelines, ETL/ELT & Streaming:
• Build reliable batch and near-real-time pipelines using Azure Databricks, Apache Spark, PySpark, SQL, Azure Data Factory, Synapse, and related Azure services.
• Implement ingestion frameworks for structured, semi-structured, and unstructured data from multiple enterprise sources.
• Ensure data quality, lineage, validation, observability, error handling, and reusable pipeline patterns.
• Engineering Leadership:
• Lead and mentor a team of data engineers, set engineering standards, review designs, and guide implementation decisions.
• Collaborate with architects, product owners, business stakeholders, data scientists, security, platform, and operations teams.
• Drive Agile delivery, estimation, sprint planning, technical roadmaps, and continuous improvement across the team.
• DevOps, Security & Governance:
• Implement CI/CD for data workloads using Azure DevOps or GitHub Actions, including automated testing, deployment, and release governance.
• Apply secure engineering practices using Microsoft Entra ID, role-based access control, secrets management, and data governance controls.
• Promote monitoring, alerting, operational readiness, and SRE-aligned practices for production data platforms.

Candidate Profile

• Education: Bachelor’s degree in Computer Science, Engineering, Information Technology, Data Engineering, or equivalent practical experience.
• Experience:
• 8+ years of experience in data engineering, data platform engineering, ETL/ELT, BI, analytics, or cloud data application development.
• 4+ years of hands-on experience designing and delivering cloud-based data platforms on Microsoft Azure.
• 2+ years of strong experience with Azure Databricks, Apache Spark, PySpark, Spark SQL, Delta Lake, and Lakehouse architecture.
• Proven experience leading engineers, reviewing architecture/design, owning delivery outcomes, and driving technical excellence.
• Core Azure Skills:
• Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, Azure SQL, Dedicated SQL Pool, Cosmos DB, Logic Apps, Azure Monitor, and Application Insights.
• Experience with Microsoft Entra ID, access control, secure connectivity, key management, and enterprise data governance.
• Azure Databricks Skills:
• Hands-on expertise in Databricks workspaces, notebooks, jobs/workflows, clusters/serverless compute, Delta tables, Auto Loader, Delta Live Tables or Lakeflow-style declarative pipelines.
• Strong understanding of Unity Catalog, catalog/schema/table governance, data lineage, access controls, data quality, and lakehouse security practices.
• Experience optimizing Spark jobs, Delta Lake performance, partitioning, Z-ordering/clustering strategies, workload monitoring, and cost-efficient compute usage.

Skills & Competencies

• Technical Skills:
• Python, PySpark, SQL, Spark SQL, data modeling, ETL/ELT design, orchestration, data quality, metadata management, and performance tuning.
• Modern lakehouse patterns including Bronze/Silver/Gold layers, Delta Lake, schema evolution, slowly changing dimensions, and reusable ingestion frameworks.
• CI/CD, Git, Azure DevOps, automated testing, environment configuration, release pipelines, SonarQube, and secure coding practices.
• Monitoring and observability using Azure Monitor, Application Insights, logs, alerts, pipeline health checks, and operational dashboards.
• Leadership & Collaboration:
• Ability to lead engineers, mentor junior team members, influence design decisions, and promote engineering best practices.
• Strong communication skills with the ability to work across global, multi-cultural teams and translate business needs into scalable technical solutions.
• Comfortable working in a fast-paced Agile environment with ownership mindset, delivery focus, and continuous improvement culture.
• Certifications Preferred / Add-ons:
• Microsoft Certified: Azure Data Engineer Associate.
• Microsoft Certified: Azure Databricks Data Engineer Associate.
• Databricks Certified Data Engineer Associate or Professional.
• Azure Solutions Architect, Azure Developer, or relevant cloud/data engineering certifications are a plus.

Why Join Us?

Be part of MetLife’s technology transformation journey and help build modern, secure, scalable, and governed data platforms that create meaningful business impact. If you are passionate about Azure, Databricks, data engineering, and leading teams to deliver high-quality solutions, we would love to connect.