Lead Data Engineer — Databricks Lakehouse & Real-Time

Humana; Nashville (TN); Hybrid; USD 142,000 - 196,000; Full time

Job summary

Humana is seeking a Lead Data Engineer to own the NBA platform's data foundation on the Databricks Lakehouse. You will guide architecture, delivery, governance, and reliability of data products powering decision intelligence, machine learning, reinforcement learning, and real-time insights across the platform.

You will lead a small team of engineers and contractors, stay hands-on, and evolve pipelines, feature layers, streaming architectures, and data quality controls to turn raw healthcare data

Qualifications

• Bachelor's degree in computer science or related field
• 7+ years of data engineering experience with at least 1–2 years in a lead engineer or technical leadership capacity.
• Expert-level SQL and Python skills with significant experience developing and operating Spark-based data platforms.
• Deep experience with Databricks, Delta Lake, and modern Lakehouse architectures.
• Experience designing and operating Medallion Architecture (Bronze/Silver/Gold) implementations at enterprise scale.
• Experience building and supporting both batch and streaming data pipelines in production environments.
• Strong understanding of data quality engineering, validation frameworks, lineage, monitoring, and operational support models.
• Experience designing secure and compliant data platforms in regulated environments.
• Demonstrated ability to lead a small engineering team while remaining an active hands-on contributor.
• Strong communication skills and the ability to explain technical tradeoffs and architecture decisions to engineering, business, and compliance stakeholders.

Responsibilities

• Own end-to-end delivery for the data engineering pod, including planning, execution, quality, and operational readiness.
• Own the design and evolution of the NBA Databricks Lakehouse, including Bronze, Silver, and Gold layer standards, data lineage, and governance patterns.
• Design, build, and maintain scalable batch and real-time pipelines that process member, clinical, claims, behavioral, engagement, and socioeconomic datasets.
• Lead the design and operation of Gold-layer feature tables and reusable data products that support model training, scoring, reinforcement learning, and decision intelligence workloads.
• Design and implement Kafka- and Spark Structured Streaming-based ingestion and processing patterns that enable near real-time decision-making.
• Establish platform-wide standards for data validation, monitoring, lineage, reconciliation, alerting, and operational visibility; treat data quality issues as production incidents.
• Drive optimization of Spark workloads, Delta Lake storage patterns, partitioning strategies, and query performance to support enterprise-scale volumes efficiently.
• Ensure compliance with PHI, HIPAA, auditability, retention, and governance requirements through Unity Catalog, lineage tooling, and secure data management practices.
• Lead and mentor engineers and contractors; conduct code reviews, establish engineering standards, and drive adoption of best practices across the pod.
• Partner with Decision Intelligence, Data Science, AI Engineering, and Platform Engineering teams to ensure reliable, governed data delivery across the NBA ecosystem.

Skills

SQL

Python

Spark

Databricks

Delta Lake

Medallion Architecture

Data governance

Leadership

Education

Bachelor's degree

Tools

Kafka

Unity Catalog

Spark Structured Streaming

Job description

Humana is seeking a Lead Data Engineer to own the NBA platform's data foundation on the Databricks Lakehouse. You will guide architecture, delivery, governance, and reliability of data products powering decision intelligence, machine learning, reinforcement learning, and real-time insights across the platform.

You will lead a small team of engineers and contractors, stay hands-on, and evolve pipelines, feature layers, streaming architectures, and data quality controls to turn raw healthcare data