Cloud Data Architect & Lead Engineer (Azure Databricks)

MetLife; Cary (NC); On-site; USD 130,000 - 180,000; Full time

Job summary

MetLife’s Data and Analytics organization in the U.S. operates within the USB segment to drive data, analytics, and AI initiatives. The Lead Data Engineer will architect end-to-end data solutions, build scalable ETL pipelines on Azure Databricks, and mentor a team of engineers across multiple business lines.

You will partner with IT, BI, and data science teams to deliver reusable data products, improve data quality, and advance CI/CD for data solutions in cloud environments.

Qualifications

• 8+ years in data solutions development and delivery, with 4+ years in Azure/Databricks.
• Proficient in SQL, Spark, Python/Scala, and performance tuning.
• Hands-on Big Data experience with Hive, HBase, and cloud tools like Azure Databricks.
• Strong track record building data warehouses for analytics on cloud platforms.
• Experience building pipelines with Azure Data Factory, Spark, Databricks, and Delta Lake.
• Scripting skills in shell or PowerShell.
• Knowledge of data governance and data quality practices.
• Ability to draft functional/technical requirements and design data flows.
• Excellent written and verbal communication skills.

Responsibilities

• Design end-to-end data architecture for data hubs and products.
• Oversee data solutions to ensure proper storage, processing, and quality.
• Own reusable data pipelines using Big Data and Azure Databricks for eligibility data products.
• Ingest large data volumes from multiple platforms for analytics, writing robust ETL code.
• Lead and mentor a team of data engineers with defined processes and tools.
• Develop reusable frameworks to reduce development effort and cost.
• Enhance CI/CD practices to automate data solution delivery.
• Collaborate with IT, business, BI, and data science teams on data products.

Skills

SQL

Spark

Python

Scala

Azure Databricks

Big Data

Data Warehouses

CI/CD

Shell scripting

Data governance

Communication

Tools

Azure Data Factory

Delta Lake

Cosmos DB

Hive

HBase

Job description

MetLife’s Data and Analytics organization in the U.S. operates within the USB segment to drive data, analytics, and AI initiatives. The Lead Data Engineer will architect end-to-end data solutions, build scalable ETL pipelines on Azure Databricks, and mentor a team of engineers across multiple business lines.

You will partner with IT, BI, and data science teams to deliver reusable data products, improve data quality, and advance CI/CD for data solutions in cloud environments.