Azure Databricks Data Engineer – Insurance Lakehouse

Builders; Atlanta (GA); On-site; USD 120,000 - 160,000; Full time

Benefits offered by this job

Competitive salary

Bonus

Profit sharing

Medical insurance

Dental insurance

Vision insurance

Short term disability

Long term disability

Life insurance

Voluntary life

401K with company match

PTO

Job summary

Builders is seeking an experienced Data Engineer to design, build, and govern Azure Databricks lakehouse pipelines for property and casualty insurance data migrations. You will implement historical loads, CDC patterns, and medallion-architecture transformations using PySpark and Spark SQL, while integrating third-party data and ensuring quality and governance.

The role involves scheduling pipelines with Databricks Workflows or Azure Data Factory, applying CI/CD practices, and collaborating with

Qualifications

• Associate degree or equivalent combination of education and experience
• Databricks engineering certification
• 3+ years in data engineering, including 1+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake)

Responsibilities

• Design and build pipelines for phased migration from Guidewire and legacy systems into the Azure Databricks lakehouse.
• Implement historical loads, incremental/CDC loads, and reconciliation.
• Create bronze and silver layer transformations in PySpark and Spark SQL following medallion architecture on Delta Lake.
• Support data modeling for property and casualty insurance including policies, covers, and claims.
• Integrate third-party data sources such as geospatial and weather data.
• Schedule and run production pipelines using Databricks Workflows or Azure Data Factory.
• Investigate and resolve pipeline failures.
• Apply Unity Catalog controls including permissions, masking, and lineage.
• Implement data quality checks, error handling, and monitoring.
• Tune pipeline performance and cost via partitioning and right-sized compute.
• Use Git version control and CI/CD with Databricks Asset Bundles to deploy code across environments.
• Write documentation and runbooks; participate in code reviews; gradually own contractor-built pipelines.

Skills

Python/PySpark

SQL

Data engineering

Education

Associate degree

Tools

Databricks

PySpark

Spark SQL

Delta Lake

Azure Data Factory

Git/GitHub/Bitbucket

CI/CD

MuleSoft

Job description

Builders is seeking an experienced Data Engineer to design, build, and govern Azure Databricks lakehouse pipelines for property and casualty insurance data migrations. You will implement historical loads, CDC patterns, and medallion-architecture transformations using PySpark and Spark SQL, while integrating third-party data and ensuring quality and governance.

The role involves scheduling pipelines with Databricks Workflows or Azure Data Factory, applying CI/CD practices, and collaborating with