Lead Data Engineer - AI-Powered Lakehouse Platform

Staffingine LLC; Cary (NC); On-site; USD 140,000 - 190,000; Full time

Job summary

Staffingine LLC in Cary, NC seeks a hands-on Lead Data Engineer to own lakehouse architecture across Bronze to Gold layers, optimize costs, and design scalable ingestion pipelines. You will implement Databricks Delta Lake and Unity Catalog, tune Spark workloads, and build CI/CD for Databricks and Azure pipelines.

The role requires expertise in Python/Scala/PySpark, strong data modeling, and experience with Azure Data Factory and Event Hubs.

Qualifications

• Expert level proficiency in Python, Scala, and PySpark with production-grade pipelines.
• Strong SQL and dimensional data modeling skills.
• Databricks Delta Lake, Unity Catalog, Jobs & Workflows experience.

Responsibilities

• Own the end-to-end lakehouse architecture, including Bronze/Silver/Gold layers and Delta Lake table design.
• Design and build metadata-driven ingestion frameworks for new data sources.
• Write and review canonical PySpark/Scala Spark transformations and set coding standards.

Skills

Python

Scala

PySpark

SQL

Data modeling

Tools

Databricks

Azure Data Factory

ADLS Gen2

Azure Event Hubs

Terraform

Job description

Staffingine LLC in Cary, NC seeks a hands-on Lead Data Engineer to own lakehouse architecture across Bronze to Gold layers, optimize costs, and design scalable ingestion pipelines. You will implement Databricks Delta Lake and Unity Catalog, tune Spark workloads, and build CI/CD for Databricks and Azure pipelines.

The role requires expertise in Python/Scala/PySpark, strong data modeling, and experience with Azure Data Factory and Event Hubs.