Lead Data Engineer

MetLife; Cary (NC); On-site; USD 130,000 - 180,000; Full time

Job summary

MetLife’s Data and Analytics organization in the U.S. operates within the USB segment to drive data, analytics, and AI initiatives. The Lead Data Engineer will architect end-to-end data solutions, build scalable ETL pipelines on Azure Databricks, and mentor a team of engineers across multiple business lines.

You will partner with IT, BI, and data science teams to deliver reusable data products, improve data quality, and advance CI/CD for data solutions in cloud environments.

Qualifications

• 8+ years in data solutions development and delivery, with 4+ years in Azure/Databricks.
• Proficient in SQL, Spark, Python/Scala, and performance tuning.
• Hands-on Big Data experience with Hive, HBase, and cloud tools like Azure Databricks.
• Strong track record building data warehouses for analytics on cloud platforms.
• Experience building pipelines with Azure Data Factory, Spark, Databricks, and Delta Lake.
• Scripting skills in shell or PowerShell.
• Knowledge of data governance and data quality practices.
• Ability to draft functional/technical requirements and design data flows.
• Excellent written and verbal communication skills.

Responsibilities

• Design end-to-end data architecture for data hubs and products.
• Oversee data solutions to ensure proper storage, processing, and quality.
• Own reusable data pipelines using Big Data and Azure Databricks for eligibility data products.
• Ingest large data volumes from multiple platforms for analytics, writing robust ETL code.
• Lead and mentor a team of data engineers with defined processes and tools.
• Develop reusable frameworks to reduce development effort and cost.
• Enhance CI/CD practices to automate data solution delivery.
• Collaborate with IT, business, BI, and data science teams on data products.

Skills

SQL

Spark

Python

Scala

Azure Databricks

Big Data

Data Warehouses

CI/CD

Shell scripting

Data governance

Communication

Tools

Azure Data Factory

Delta Lake

Cosmos DB

Hive

HBase

Job description

Role Value Proposition

The position sits within the newly consolidated Data and Analytics (D&A) organization supporting the U.S. Business of MetLife. U.S. D&A assists all business lines of MetLife’s U.S. business (about 2/3 of MetLife Global by earnings) with everything related to data, analytics, and data science, from data infrastructure, data governance, data engineering, data modeling, data analysis to business intelligence, data science, and AI.

The Lead Data Engineer is crucial to D&A USB’s Eligibility team for creating and optimizing data architecture, solutions, and operations, and for ensuring alignment with data management and governance framework. The Lead Data Engineer serves as a big data development expert within the D&A Eligibility organization. This position is responsible for leading, architecting, and building ETL, data warehousing, and reusable components using cutting‑edge big data and cloud technologies, collaborating with architects, business systems analysts, technical leads, project managers, and business/operations teams across different lines of business and use cases.

Key Responsibilities

• Design and solution end-to-end data architecture for data hubs/data products, all the way from source systems to consumption.
• Oversee the design and management of data solutions to ensure data is stored, processed, curated, and utilized effectively.
• Own and build a reusable data pipeline utilizing Big Data and Azure/Databricks for Eligibility Program data products.
• Ingest large volumes of data from various platforms for analytics needs, writing high-performance, reliable, and maintainable ETL code.
• Lead and mentor a team of data engineers, ensuring the efficient flow of data within the organization with defined processes and tools.
• Collect, store, process, and analyze large datasets to build and implement ETL processes.
• Develop reusable frameworks to reduce development effort and ensure cost savings.
• Utilize and enhance CI/CD practices to automate delivery of data solutions, ensuring reliability and scalability.
• Utilize cloud technologies (Azure Databricks) to enable data product solutions.
• Develop quality code through performance optimizations at the development stage.
• Show appetite to learn new technologies and work on cutting‑edge cloud technologies.
• Partner with tech, business, BI, and data science teams to create reusable data products.
• Work with a global team to deliver projects and recommend development and performance improvements.
• Track and report on KPIs for solution delivery and data quality.
• Communicate and present use cases, solutions, and impact to business stakeholders and mid/senior management.
• Optimize reusable frameworks and Spark jobs for performance and cost efficiency in large-scale environments.
• Interact with business stakeholders to gather requirements and implement solutions.
• Analyze data in depth using SQL and other exploratory tools against platforms such as Bigdata, Oracle, SQL Server, Databricks, and others.
• Work with IT, business, and architects to develop and design requirements to formulate technical design.

Required Skills

• 8+ years of data solutions development and delivery experience, with 4+ years in Azure/Databricks environments.
• Proficiency and extensive experience with SQL, Spark, and/or Scala/Python, and performance tuning.
• Hands‑on expertise in Big Data (e.g., Hive, HBase), Azure Databricks, Azure Functions, Cosmos DB and/or Data Factory.
• Strong experience building and designing data warehouses and data stores for analytics consumption on cloud (real‑time and batch use cases).
• Design, build, and deploy robust data ingestion and curation pipelines using Azure Data Factory, Apache Spark (Scala or Python), Azure Databricks, and Delta Lake.
• Good scripting experience, primarily on shell/bash or PowerShell.
• Strong SQL knowledge and data analysis skills for data anomaly detection and data quality assurance.
• Experience implementing data governance and data quality using enterprise toolsets.
• Skilled in drafting functional and technical requirements, creating high‑level design documents, and data‑flow diagrams.
• Expertise in writing validation scripts to validate data, data integrations, and ETL transformations.
• Excellent problem‑solving and communication skills – both written and verbal.

Preferred Skills

• Expertise in Python and experience writing Azure functions using Python or Node.js.
• Databricks certifications and/or Microsoft Azure certifications.
• Experience using Event Hub for data integrations.
• Eagerness to learn new technologies and ship to production.
• Hive database management and performance tuning – partitioning and bucketing.
• Ability to interact with senior leadership teams in IT and business.