Lead Data Engineer

Innovative Information Technologies, Inc.; Cary (NC); On-site; USD 150,000 - 210,000; Full time

Job summary

Innovative Information Technologies, Inc. in Cary, NC seeks a senior data and AI lead to own end-to-end design of the data and AI layers on a modern Azure Databricks platform. The role focuses on building robust data pipelines and AI capabilities with enterprise-grade quality.

You will ship production-grade Python, Scala, and PySpark code weekly, mentor engineers, and drive reference implementations to accelerate delivery across the program.

Qualifications

• Senior hands-on leadership role
• Own end-to-end design of data and AI layers
• Ship production-grade Python, Scala, and PySpark code weekly

Responsibilities

• Own end-to-end data and AI layer design for the platform
• Mentor and lead the engineering team across data and AI initiatives
• Deliver weekly production-grade code in Python, Scala and PySpark

Skills

Leadership

Python

Scala

PySpark

Azure Databricks

Tools

Azure Databricks

Job description

About The Engagement

A centralized, AI-first enterprise Data Hub for a global insurance and financial services client on Azure Databricks. The platform ingests 150+ inbound data feeds, distributes to 35+ downstream systems, and is organized as a medallion architecture (Bronze / Silver / Gold). AI is embedded in ingestion, canonical mapping, data quality, reconciliation, and business user access from day one.

Job Description

Job Description

About The Engagement

A centralized, AI-first enterprise Data Hub for a global insurance and financial services client on Azure Databricks. The platform ingests 150+ inbound data feeds, distributes to 35+ downstream systems, and is organized as a medallion architecture (Bronze / Silver / Gold). AI is embedded in ingestion, canonical mapping, data quality, reconciliation, and business user access from day one.

This is a senior hands-on leadership role. The candidate will own the end-to-end technical design of the data and AI layers, build reference implementations for the engineering team, and ship production-grade Python, Scala, and PySpark code every week. Candidates who have not written or reviewed production code in the past year are not a fit.