Data Engineer Lead

Builders; Atlanta (GA); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Bonus Structure

Profit Sharing

Employer Paid Short Term Disability

Employer Paid Long Term Disability

Employer Paid Life Insurance

Voluntary Life Insurance

401K with Company Match

PTO

Job summary

Builders is seeking a Data Engineering Lead to own the Azure Databricks lakehouse for its property and casualty insurance data. You will drive design, development, and modernization, aligning governance and security with Unity Catalog and partnering with governance leads.

A team of engineers and contractors will report to you. You’ll lead migrations from Guidewire and legacy systems, optimize performance and costs, and set engineering standards across pipelines, testing and documentation, while

Qualifications

• Associate degree or equivalent combination of education and experience.
• Ten+ years in data engineering, including six+ years with Databricks.
• Databricks Data Engineer Professional certification; Azure data/solutions architect preferred.
• Knowledge of Unity Catalog governance, permissions, row/column security, lineage.
• Experience leading large data migrations for insurance data.Understanding of policy/claims lifecycle and KPI data.

Responsibilities

• Design Databricks lakehouse architecture for policy, rating, claims and billing data.
• Lead phased data migration from Guidewire and legacy systems into the lakehouse.
• Model insurance data: accounts, policies, insured risks and exposures, premiums and claims.
• Provide data for actuarial/finance: premiums, loss triangles, IBNR, Schedule P, NAIC reporting.
• Define Unity Catalog design, security, and lineage; implement governance with stakeholders.
• Manage pipelines with Databricks Workflows; ensure performance, cost and reliability.
• Mentor data engineers and contractors; establish testing and documentation standards.

Skills

Databricks

PySpark

Delta Lake

Unity Catalog

Azure Databricks

Data governance

SQL

Python

Git

Terraform

Education

Associate degree or equivalent

Tools

Databricks

Guidewire data

Azure Data Factory

Azure DevOps

GitHub

Terraform

Job description

• Location 2859 Paces Ferry Road Southeast,Atlanta, GA, 30339,United States
• Employee Type Full Time Exempt

Description

Integrity. Care. Trust. Compassion. Expertise.
Do these words resonate with you?
These values of Builders culture create success in all we do. We strive to provide deeply supportive partnerships to our customers, agents, and each other.

Builders is proud to be named among the Great Places to Work. Our award-winning culture has earned top marks in Company Direction, Employee Appreciation, Work-Life Balance, Leadership, and Compensation and Benefits. Our strong culture keeps us Built Strong in a forever-changing world, and our AM Best A Rating is evidence of our financial strength.

Position Summary

The Data Engineering Lead serves as the technical owner of the company’s Azure Databricks lakehouse platform supporting its property and casualty insurance business. This role drives the design, development, and evolution of the platform, leveraging Databricks technologies including Delta Lake, Unity Catalog, Lakeflow pipelines and Databricks SQL. The Data Engineering Lead establishes and promotes engineering standards, best practices and architectural principles to ensure a scalable, secure, reliable and high-performing data platform.

A key responsibility is to lead the phased migration and modernization of data from Guidewire and legacy policy, claims and billing systems into the Azure Databricks lakehouse. The role champions effective data governance, security and lineage through Unity Catalog, partnering closely with the Data Governance Lead to translate governance policies into practical, consistent engineering practices. In addition the Data Engineering Lead provides technical direction, mentorship and leadership to a team of engineers and contractors, fostering collaboration, engineering excellence and the successful delivery of strategic data initiatives.

Responsibilities

• Design the Databricks lakehouse architecture (bronze/silver/gold medallion on Delta Lake) for property and casualty policy, rating, claims, billing, reinsurance and customer data.
• Plan and lead the phased data migration from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems into the lakehouse on Azure Databricks, including historical loads, incremental/CDC (change data capture) loads, reconciliation and decommissioning support.
• Model property and casualty insurance data: accounts, policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments and recoveries.
• Deliver data for Actuarial and Finance: written/earned/unearned premium, loss triangles, IBNR (incurred but not reported) inputs, loss ratios, Schedule P and NAIC (National Association of Insurance Commissioners) statutory reporting.
• Integrate third-party data sources such as geospatial and weather data.
• Define the Unity Catalog design: metastore, catalogs, schemas, environments (development, test and production), naming conventions, permissions, row/column-level security and lineage.
• Manage pipeline orchestration and operations in Databricks: design job scheduling and dependencies with Databricks Workflows (Lakeflow Jobs), set up retries, alerting and SLA (Service Level Agreement) monitoring and establish failure handling and runbooks for production pipelines.
• Manage performance and cost: Spark tuning, partitioning and liquid clustering, Photon, cluster and SQL warehouse sizing.
• Shape the data infrastructure with IT (Information Technology) and the Platform Engineer: storage, networking, security, identity, IaC (Infrastructure as Code) and CI/CD (Continuous Integration/Continuous Deployment).
• Set engineering standards for pipelines, testing, code review and documentation; lead, mentor and review the work of Data Engineers and junior contractors.
• Partner with the data governance lead to implement access controls, PII (personally identifiable information) masking, lineage and data quality rules.
• Perform other duties as assigned.

Qualifications

• Associate degree or an equivalent combination of education and experience
• Ten or more years of experience in data engineering, including six or more years of hands-on experience with Databricks (PySpark, Spark SQL, Delta Lake); experience leading or mentoring engineering teams
• Certification as a Databricks Data Engineer Professional; certification as an Azure data or solutions architect is preferred
• Knowledge of Unity Catalog: governance model, permissions, row/column-level security, lineage and system tables
• Experience leading large data migrations from core insurance and legacy systems into a lakehouse or data warehouse; experience with property and casualty insurance data; understanding of the policy and claims lifecycle and core property and casualty KPIs (key performance indicators)
• Knowledge of data modeling: dimensional modeling, data vault, SCD (slowly changing dimension) patterns and insurance data models
• Skill in technical aspects : Spark internals and performance tuning, streaming (structured streaming, auto loader), CDC patterns, MERGE, Lakeflow/Delta Live Tables
• Skill at the expert level in SQL (structured query language) and Python; software engineering practice ability (testing, code review, modular design)
• Knowledge of data infrastructure: ADLS (Azure data lake storage) Gen2, Azure Data Factory, Event Hubs, Key Vault, Entra ID, VNets (Virtual Networks) and Private Link
• Experience scheduling and monitoring production pipelines with Databricks Workflows and/or Azure Data Factory, including job dependencies, parameterization, alerting and monitoring with Databricks system tables
• Experience with Git-based version control (GitHub or Bitbucket), including branching strategies, pull request reviews and Databricks Git folders
• Infrastructure-as-code and CI/CD for data (Terraform, Databricks Asset Bundles, Azure DevOps, GitHub Actions or Bitbucket Pipelines)
• Experience with Guidewire data (Guidewire DataHub/InfoCenter, Cloud Data Access); supporting actuarial reserving and pricing for property and casualty lines of business; API (application programming interface) and integration tools such as MuleSoft is preferred
• Knowledge of insurance data standards such as ACORD (Association for Cooperative Operations Research and Development) is preferred

Let’s talk benefits!

• Bonus Structure
• Profit Sharing
• Employer Paid Short Term Disability
• Employer Paid Long Term Disability
• Employer Paid Life Insurance
• Voluntary Life Insurance
• 401K with Company Match
• PTO

About Builders

Builders is a mid-sized mutual with remarkable strengths. Rated A by AM Best, Builders has forged rock-solid financial strength and a reputation for reliability and fairness in fulfilling our promises to customers. Kind, collaborative and customer-centric, our experienced and passionate teams foster a rewarding atmosphere of excellence trust and mutual respect, meriting the "Culture Excellence" honors from Top Workplaces. Flexible and highly personal, our experts leverage deeply supportive partnerships with knowledgeable independent agencies to drive better services and protection for policyholders.

Our financial excellence amazing people and powerful partnerships build outstanding outcomes and peace of mind for our agents and their clients. This is what we mean by Insurance Built Strong.

Builders Insurance Group is an Equal Opportunity Employer.We welcome applicants from all walks of life and don't discriminate based on any protected status. Join us in creating a diverse and inclusive workplace!