Data Engineer

Builders; Atlanta (GA); Presencial; USD 120 000 - 160 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Competitive salary

Bonus

Profit sharing

Medical insurance

Dental insurance

Vision insurance

Short term disability

Long term disability

Life insurance

Voluntary life

401K with company match

PTO

Resumo da oferta

Builders is seeking an experienced Data Engineer to design, build, and govern Azure Databricks lakehouse pipelines for property and casualty insurance data migrations. You will implement historical loads, CDC patterns, and medallion-architecture transformations using PySpark and Spark SQL, while integrating third-party data and ensuring quality and governance.

The role involves scheduling pipelines with Databricks Workflows or Azure Data Factory, applying CI/CD practices, and collaborating with

Qualificações

• Associate degree or equivalent combination of education and experience
• Databricks engineering certification
• 3+ years in data engineering, including 1+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake)

Responsabilidades

• Design and build pipelines for phased migration from Guidewire and legacy systems into the Azure Databricks lakehouse.
• Implement historical loads, incremental/CDC loads, and reconciliation.
• Create bronze and silver layer transformations in PySpark and Spark SQL following medallion architecture on Delta Lake.
• Support data modeling for property and casualty insurance including policies, covers, and claims.
• Integrate third-party data sources such as geospatial and weather data.
• Schedule and run production pipelines using Databricks Workflows or Azure Data Factory.
• Investigate and resolve pipeline failures.
• Apply Unity Catalog controls including permissions, masking, and lineage.
• Implement data quality checks, error handling, and monitoring.
• Tune pipeline performance and cost via partitioning and right-sized compute.
• Use Git version control and CI/CD with Databricks Asset Bundles to deploy code across environments.
• Write documentation and runbooks; participate in code reviews; gradually own contractor-built pipelines.

Conhecimentos

Python/PySpark

SQL

Data engineering

Formação académica

Associate degree

Ferramentas

Databricks

PySpark

Spark SQL

Delta Lake

Azure Data Factory

Git/GitHub/Bitbucket

CI/CD

MuleSoft

Descrição da oferta de emprego

Build, govern, and optimize Azure Databricks lakehouse pipelines supporting property and casualty insurance data migrations.

Responsibilities

• Design and build pipelines for the phased migration from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems into the Azure Databricks lakehouse
• Implement historical loads, incremental/CDC (Change Data Capture) loads, and reconciliation
• Create bronze and silver layer transformations in PySpark and Spark SQL following the medallion architecture on Delta Lake
• Support data modeling for property and casualty insurance, including policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries
• Integrate third-party data sources such as geospatial and weather data
• Schedule and run production pipelines using Databricks Workflows (Lakeflow Jobs) or Azure Data Factory, including dependencies, parameters, retries, and alerting
• Investigate and resolve pipeline failures
• Apply Unity Catalog controls defined by the Data Engineering Lead and Data Governance lead, including permissions, PII (Personally Identifiable Information) masking, tags, and lineage
• Implement data quality checks, error handling, and monitoring
• Tune pipeline performance and cost using partitioning, liquid clustering, and right-sized job compute
• Use Git version control (GitHub or Bitbucket) and CI/CD with Databricks Asset Bundles to version, review, and deploy code across development, test, and production
• Write documentation and runbooks, participate in code reviews, and gradually take ownership of contractor-built pipelines
• Perform other duties as assigned

Requirements

• Associate degree or equivalent combination of education and experience
• Databricks engineering certification
• 3+ years in data engineering, including 1+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake)
• Experience with property and casualty insurance data, ideally Guidewire data, and experience migrating from legacy source systems
• Knowledge of Python/PySpark and SQL (Structured Query Language)
• Knowledge of Unity Catalog: permissions, row/column-level security, and lineage
• Experience with incremental loading patterns such as CDC, MERGE, SCD (Slowly Changing Dimension), and Auto Loader
• Knowledge of data modeling, including dimensional modeling
• Experience scheduling and monitoring pipelines with Databricks Workflows and/or Azure Data Factory
• Experience with Git-based version control (GitHub or Bitbucket), including branching, pull request reviews, and Databricks Git folders
• Experience with Azure data services including ADLS (Azure Data Lake Storage) Gen2, Azure Data Factory, and Key Vault
• Knowledge of CI/CD basics (Databricks Asset Bundles, Azure DevOps, GitHub Actions, or Bitbucket Pipelines)
• Experience with streaming (Structured Streaming, Event Hubs)
• Experience with API and integration tools such as MuleSoft (added advantage)

Technologies

• Azure Databricks
• Delta Lake
• Unity Catalog
• Lakeflow
• Databricks SQL
• PySpark
• Spark SQL
• Databricks Workflows
• Azure Data Factory
• Git, GitHub, Bitbucket
• Databricks Asset Bundles
• Python
• SQL (Structured Query Language)
• ADLS (Azure Data Lake Storage) Gen2
• Azure DevOps
• GitHub Actions
• Bitbucket Pipelines
• Key Vault
• Structured Streaming
• Event Hubs
• Auto Loader
• MERGE
• SCD (Slowly Changing Dimension)
• CDC (Change Data Capture)
• MuleSoft
• API (Application Programming Interface)

Benefits

• Competitive Salary
• Bonus Structure
• Profit Sharing
• Medical, Dental, Vision Insurance
• Employer Paid Short Term Disability
• Employer Paid Long Term Disability
• Employer Paid Life Insurance
• Voluntary Life Insurance
• 401K with Company Match
• PTO

Location and Experience

• Atlanta, GA (onsite)
• Minimum experience: 3 years
• Education: Associate degree

Equal Opportunity Employer

• Builders welcomes applicants from all walks of life and does not discriminate based on protected status.