Lead Data Engineer: Azure Databricks & Data Architecture

MetLife; Bridgewater (MA); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Health plan

Employer 401(k) with company match

Life insurance

Job summary

MetLife is seeking a Lead Data Engineer to join the Data & Analytics USB, focusing on building end-to-end data architecture and reusable data pipelines using Big Data and Azure Databricks for Eligibility data products.

You will lead a multinational team, ingest large data volumes, optimize performance, and partner with IT, BI, and data science teams to deliver scalable data solutions. This hybrid role requires 3 days in the office per week and offers extensive benefits.

Qualifications

• 8+ years of Data Solutions, development, and delivery experience with 4+ years of recent experience in Azure/Databricks environments.
• Proficiency and extensive experience with SQL, Spark &/or Scala/Python and performance tuning.
• Hands-on expertise in Big Data (e.g. Hive and HBase), Azure Databricks, Azure Functions, Cosmos DB and/or Data Factory experience is a MUST.

Responsibilities

• Design and solution end-to-end data architecture for data hubs/data products, all the way from source systems to consumption.
• Oversee the design and management of data solutions to ensure data is stored, processed, curated, and utilized effectively.
• Own and build a reusable data pipeline utilizing Big Data and Azure/Databricks for Eligibility Program data products.
• Ingesting huge volumes of data from various platforms for Analytics needs and writing high-performance, reliable, and maintainable ETL code.
• Leadership: Lead and mentor a team of data engineers, ensuring the efficient flow of data within the organization with the defined processes and tools.
• Collect, store, process, and analyze large datasets to build and implement ETL processes.
• Develop reusable frameworks to reduce development effort involved, thereby ensuring cost savings for the projects.
• Utilizing CI/CD Pipelines to automate the delivery of data solutions, ensuring reliability and scalability.
• Utilize Cloud technologies (Azure Databricks) to enable data product solutions.
• Develop quality code through performance optimizations in place right at the development stage.
• Partner with Tech, Business, BI, and Data Science teams to create reusable data products.
• Work with a team spread across the globe in driving the delivery of projects and recommend development and performance improvements.
• Track and report on KPIs for solution delivery and data quality.
• Communicate and present use cases, solutions, and impact to business stakeholders and mid/senior management.
• Optimize reusable frameworks, Spark jobs for performance and cost efficiency in large-scale environments.
• Ability to interact with business stakeholders in getting the requirements and implementing solutions.
• Analyze the data in depth, using SQLs and other exploratory tools against various platforms such as Bigdata, Oracle, SQL Server, Databricks and others.
• Work with IT, business, and architects to develop and design requirements to formulate technical design.

Skills

SQL

Spark

Python

Scala

Data warehousing design

ETL development

Data pipelines

Big Data concepts

Cloud computing

Tools

Azure Databricks

Azure Data Factory

Cosmos DB

Hive

HBase

Delta Lake

Shell scripting

Job description

MetLife is seeking a Lead Data Engineer to join the Data & Analytics USB, focusing on building end-to-end data architecture and reusable data pipelines using Big Data and Azure Databricks for Eligibility data products.

You will lead a multinational team, ingest large data volumes, optimize performance, and partner with IT, BI, and data science teams to deliver scalable data solutions. This hybrid role requires 3 days in the office per week and offers extensive benefits.