Lead Data Engineer AI-Driven Lakehouse on Azure Databricks

Accrescent Group; Cary (NC); On-site; USD 140,000 - 180,000; Full time

Job summary

Accrescent Group is seeking a senior hands-on Lead Data Engineer to own the end-to-end technical design of data and AI layers for a centralized, AI-first Data Hub built on Azure Databricks. The role leads lakehouse architecture, metadata-driven ingestion, data quality, governance, and production engineering while mentoring the team.

You will push for Delta Lake design, Unity Catalog, and scalable streaming pipelines using Spark Structured Streaming, with CI/CD in Azure DevOps and Terraform.

Qualifications

• Expert-level Python/Scala/PySpark production code.
• Strong SQL and data modelling capabilities.
• Deep Databricks experience with Delta Lake and Unity Catalog.
• Extensive Azure data stack experience (ADLS Gen2, ADF, Event Hubs).
• 12–18 years in data engineering and lakehouse delivery.

Responsibilities

• Own end-to-end design for data and AI layers on a central Data Hub.
• Lead lakehouse architecture and governance across the stack.
• Mentor the engineering team and enforce CI/CD practices.
• Implement AI-augmented data ingestion and quality controls.

Skills

Python/Scala/PySpark

SQL modelling

Databricks

Azure

CI/CD

LangChain/LLM tooling

Data governance

Tools

Delta Lake

Unity Catalog

Azure Data Factory

Azure Event Hubs

Terraform

Databricks Jobs/Workflows

Job description

Accrescent Group is seeking a senior hands-on Lead Data Engineer to own the end-to-end technical design of data and AI layers for a centralized, AI-first Data Hub built on Azure Databricks. The role leads lakehouse architecture, metadata-driven ingestion, data quality, governance, and production engineering while mentoring the team.

You will push for Delta Lake design, Unity Catalog, and scalable streaming pipelines using Spark Structured Streaming, with CI/CD in Azure DevOps and Terraform.