Data Engineer II: Quality & Pipelines in Azure/Databricks

MetLif; Cary (NC); Hybrid; USD 85,000 - 115,000; Full time

Job summary

MetLife is hiring a Data Engineer II to build, test, and support data pipelines in Azure Databricks and Delta Lake environments. You will focus on validation, profiling, reconciliation, and data quality to enable analytics-ready data assets.

You will collaborate with engineering, analytics, and operations teams in a hybrid, in-office role in Cary, NC, contributing to scalable data solutions and governance across the D/A organization.

Qualifications

• Bachelor's or master's degree in computer science, engineering or related quantitative field.
• 3-5 years of data engineering or analytics engineering experience.
• Strong hands-on experience with Python, PySpark, Spark SQL and SQL.

Responsibilities

• Perform data validation activities including source-to-target validation and data profiling.
• Develop, test, and operate ETL/ELT pipelines with Azure Data Factory, Databricks and Delta Lake.
• Troubleshoot data issues and collaborate with cross-functional teams to improve data reliability.
• Design and optimize scalable data processing and validation solutions for reporting and analytics.
• Support and evolve Delta Lake/Lakehouse architectures for reliability and performance.
• Ensure data processing complies with security and quality standards.
• Contribute to reusable validation frameworks and engineering best practices.

Skills

Python

PySpark

Spark SQL

SQL

Azure Data Factory

Databricks

Data validation

Data quality

Delta Lake

Lakehouse

CI/CD

Git

Azure DevOps

Education

Bachelor's or master's degree in CS/Engineering/IS/Math/Stats or related field

Tools

Azure Data Factory

Databricks

Delta Lake

Spark

Python

Job description

MetLife is hiring a Data Engineer II to build, test, and support data pipelines in Azure Databricks and Delta Lake environments. You will focus on validation, profiling, reconciliation, and data quality to enable analytics-ready data assets.

You will collaborate with engineering, analytics, and operations teams in a hybrid, in-office role in Cary, NC, contributing to scalable data solutions and governance across the D/A organization.