Lead Data Engineer: Azure Data Bricks, Data Lake & Pipelines

MetLife; New York (NY); On-site; USD 120,000 - 150,000; Full time

Benefits offered by this job

Flexible working hours

Comprehensive health benefits

Professional development opportunities

Job summary

A leading global insurance firm is seeking a Lead Data Engineer to implement scalable data solutions and drive analytics initiatives. This role involves designing large-scale data migrations, optimizing data pipelines, and leveraging cloud technologies, especially Microsoft Azure and Databricks. The ideal candidate should have over 10 years of experience in data engineering, strong analytic skills, and a proven record in developing solutions for complex data architectures. This is an exciting opportunity to work on cutting-edge technologies and make a significant impact.

Qualifications

• 10+ years of solutions development and delivery experience with 6+ years of recent experience in data engineering.
• Strong experience in Azure data bricks, including Spark, Delta Lake, data stores for analytics consumption.
• Proficiency and extensive experience with Spark/Scala/Python and performance tuning.

Responsibilities

• Design and execute a large-scale data migration initiative focused on architecture and quality.
• Ingest huge volumes of data for analytics needs and build reliable pipelines.
• Develop and support an API catalog for data services, ensuring standardization.

Skills

Analytic skills

Ability to interact with business analysts

Experience with APIs

Problem solving

Excellent communication skills

Education

Bachelor’s/ master’s degree in information technology/computer science or a relevant domain

Tools

Microsoft Azure

Databricks

Azure Data Factory

Spark

Scala

Python

Job description

A leading global insurance firm is seeking a Lead Data Engineer to implement scalable data solutions and drive analytics initiatives. This role involves designing large-scale data migrations, optimizing data pipelines, and leveraging cloud technologies, especially Microsoft Azure and Databricks. The ideal candidate should have over 10 years of experience in data engineering, strong analytic skills, and a proven record in developing solutions for complex data architectures. This is an exciting opportunity to work on cutting-edge technologies and make a significant impact.