Data Engineering Lead: Azure Databricks Lakehouse

Builders; Atlanta (GA); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Bonus Structure

Profit Sharing

Employer Paid Short Term Disability

Employer Paid Long Term Disability

Employer Paid Life Insurance

Voluntary Life Insurance

401K with Company Match

PTO

Job summary

Builders is seeking a Data Engineering Lead to own the Azure Databricks lakehouse for its property and casualty insurance data. You will drive design, development, and modernization, aligning governance and security with Unity Catalog and partnering with governance leads.

A team of engineers and contractors will report to you. You’ll lead migrations from Guidewire and legacy systems, optimize performance and costs, and set engineering standards across pipelines, testing and documentation, while

Qualifications

• Associate degree or equivalent combination of education and experience.
• Ten+ years in data engineering, including six+ years with Databricks.
• Databricks Data Engineer Professional certification; Azure data/solutions architect preferred.
• Knowledge of Unity Catalog governance, permissions, row/column security, lineage.
• Experience leading large data migrations for insurance data.Understanding of policy/claims lifecycle and KPI data.

Responsibilities

• Design Databricks lakehouse architecture for policy, rating, claims and billing data.
• Lead phased data migration from Guidewire and legacy systems into the lakehouse.
• Model insurance data: accounts, policies, insured risks and exposures, premiums and claims.
• Provide data for actuarial/finance: premiums, loss triangles, IBNR, Schedule P, NAIC reporting.
• Define Unity Catalog design, security, and lineage; implement governance with stakeholders.
• Manage pipelines with Databricks Workflows; ensure performance, cost and reliability.
• Mentor data engineers and contractors; establish testing and documentation standards.

Skills

Databricks

PySpark

Delta Lake

Unity Catalog

Azure Databricks

Data governance

SQL

Python

Git

Terraform

Education

Associate degree or equivalent

Tools

Databricks

Guidewire data

Azure Data Factory

Azure DevOps

GitHub

Terraform

Job description

Builders is seeking a Data Engineering Lead to own the Azure Databricks lakehouse for its property and casualty insurance data. You will drive design, development, and modernization, aligning governance and security with Unity Catalog and partnering with governance leads.

A team of engineers and contractors will report to you. You’ll lead migrations from Guidewire and legacy systems, optimize performance and costs, and set engineering standards across pipelines, testing and documentation, while