Data Platform Lead - Azure Databricks Lakehouse(Onsite ATL)

Builders; Atlanta (GA); On-site; USD 140,000 - 180,000; Full time

Benefits offered by this job

Competitive Salary

Bonus Structure

Medical, Dental, Vision Insurance

401K with Company Match

PTO

Job summary

Builders is seeking an experienced Data Platform Lead to own and advance our Azure Databricks lakehouse for property and casualty data. You will architect bronze/silver/gold pipelines, drive modernization, governance alignment, and lead engineering teams to deliver scalable, secure data foundations.

You will orchestrate large data migrations from Guidewire and legacy systems, model P&C entities, and enable actuarial and finance use cases with robust data quality and lineage.

Qualifications

• Associate degree or an equivalent combination of education and experience.
• 10+ years of data engineering experience, including 6+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake).
• Databricks Data Engineer Professional certification; Azure data or solutions architect certification preferred.
• Strong knowledge of Unity Catalog: governance model, permissions, row/column-level security, lineage, and system tables.
• Experience leading large data migrations from core insurance and legacy systems into a lakehouse or data warehouse, with P&C domain knowledge.

Responsibilities

• Design the Databricks lakehouse architecture using bronze/silver/gold layers on Delta Lake for policy, rating, claims, billing, reinsurance, and customer data.
• Plan and lead phased migration from Guidewire and legacy systems into Azure Databricks, including historical loads, incremental/CDC loads, reconciliation, and decommissioning support.
• Model core property and casualty insurance entities: accounts, policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, recoveries.
• Provide data for Actuarial and Finance use cases including written/earned/unearned premium, loss triangles, IBNR, loss ratios, Schedule P and NAIC reporting.
• Integrate third-party sources such as geospatial and weather data.
• Define and operationalize Unity Catalog design: metastore, catalogs, schemas, environments, naming conventions, permissions, security, and lineage.
• Manage pipeline orchestration and operations in Databricks, including Databricks Workflows, scheduling, dependencies, retries, alerting and SLA monitoring, and runbooks.
• Own performance and cost work: Spark tuning, partitioning, liquid clustering, Photon, and cluster/SQL warehouse sizing.
• Partner with IT and Platform Engineer on storage, networking, security, identity, IaC, and CI/CD.
• Set standards for pipelines, testing, code review, and documentation; lead mentoring for Data Engineers and contractors.
• Collaborate with data governance to implement access controls, PII masking, lineage, and data quality rules.

Skills

Databricks

PySpark

Spark SQL

Delta Lake

SQL

Python

Data Modeling

Leadership

Education

Associate degree

Tools

Unity Catalog

Databricks Workflows

Terraform

Azure Data Factory

Git

Job description

Builders is seeking an experienced Data Platform Lead to own and advance our Azure Databricks lakehouse for property and casualty data. You will architect bronze/silver/gold pipelines, drive modernization, governance alignment, and lead engineering teams to deliver scalable, secure data foundations.

You will orchestrate large data migrations from Guidewire and legacy systems, model P&C entities, and enable actuarial and finance use cases with robust data quality and lineage.