Lead Data Engineer with Databricks and Insurance domain

ALIS Software LLC; Charlotte (NC); Hybrid; USD 150,000 - 190,000; Full time

Job summary

ALIS Software LLC is hiring a Lead Data Engineer with Databricks expertise in the Charlotte, NC area. The role requires 13+ years of experience, strong Python and PySpark skills, and a solid cloud platform background (AWS/GCP/Azure). Insurance (property and casualty) domain experience is essential.

You will design secure, observable data pipelines and governance using Unity Catalog and Delta Sharing. The position is a long-term contract with a hybrid work arrangement in Charlotte, NC, offering

Qualifications

• Requires at least 13+ years of data engineering experience.
• Proficient in Python and PySpark for data transformation and validation.
• Experience with cloud platforms (AWS, GCP, or Azure).
• Insurance (P&C) domain experience is required.

Responsibilities

• Implement ingestion patterns using Auto Loader with checkpointing and schema evolution for structured and semi-structured data.
• Build secure and observable DLT pipelines with DLT Expectations and Bronze/Silver/Gold medallion layering.
• Configure Unity Catalog: catalogs, schemas, access controls, audit logging, and PII masking.
• Enable cross-domain secure data access via Unity Catalog External Locations, Volumes, and lineage tracking.
• Access Databricks Marketplace data assets for enrichment, model training, or benchmarking.
• Collaborate to implement Delta Sharing with internal and external stakeholders.

Skills

Python programming

PySpark

Insurance domain knowledge

Tools

Auto Loader

Unity Catalog

Delta Sharing

Job description

Role: Lead Data Engineer with Databricks and Insurance domain

Location: Hybrid – Charlotte, NC

Must have 13+ years experience

Type: Long Term Contract

Only local candidates currently residing in North Carolina (NC).

• Excellent programming and debugging skills in Python
• Strong hands-on experience with PySpark to build efficient data transformation and validation logic
• Must be proficient in at least one cloud platform: AWS, GCP, or Azure
• Need someone with Insurance domain experience - property and casualty
• Implement ingestion patterns using Auto Loader with checkpointing and schema evolution for structured and semi-structured data
• Build secure and observable DLT pipelines with DLT Expectations, supporting Bronze/Silver/Gold medallion layering
• Configure Unity Catalog: set up catalogs, schemas, user/group access, enable audit logging, & define masking for PII fields
• Enable secure data access across domains & workspaces via Unity Catalog External Locations, Volumes, & lineage tracking
• Access and utilize data assets from the Databricks Marketplace to support enrichment, model training, or benchmarking
• Collaborate with data sharing stakeholders to implement Delta Sharing — both internally and externally