Lead Data Engineer - Real-Time NBA Lakehouse

Humana Inc; Boston (MA); Hybrid; USD 142,000 - 196,000; Full time

Job summary

Humana Inc. seeks a Lead Data Engineer to own the NBA platform's data foundation on the Databricks Lakehouse.

You will lead architecture, delivery, quality, governance, and reliability of data products powering decision intelligence, ML, and real-time decisioning across the NBA ecosystem. You will guide a small team of engineers and contractors while remaining hands-on, shaping pipelines, feature layers, streaming patterns, and data quality controls for trusted, production-ready assets used by

Qualifications

• Bachelor's degree in computer science or related field.
• 7+ years data engineering experience with 1-2 years in a lead engineer or technical leadership capacity.
• Expert-level SQL and Python skills with significant experience developing and operating Spark-based data platforms.
• Deep experience with Databricks, Delta Lake, and modern Lakehouse architectures.
• Experience designing and operating Medallion Architecture (Bronze/Silver/Gold) implementations at enterprise scale.
• Experience building and supporting both batch and streaming data pipelines in production environments.
• Strong understanding of data quality engineering, validation frameworks, lineage, monitoring, and operational support models.
• Experience designing secure and compliant data platforms in regulated environments.
• Demonstrated ability to lead a small engineering team while remaining an active hands-on contributor.
• Strong communication skills and the ability to explain technical tradeoffs and architecture decisions to engineering, business, and compliance stakeholders.

Responsibilities

• Own end-to-end pod delivery for data engineering.
• Design and evolve the NBA Databricks Lakehouse architecture across Bronze/Silver/Gold layers.
• Build and maintain scalable batch and real-time pipelines from healthcare datasets.
• Lead the design of Gold-layer feature tables and reusable data products for model training and decisioning.
• Design Kafka- and Spark-based streaming ingestion for near real-time decision making.
• Establish data quality, lineage, monitoring, and governance standards across the platform.
• Optimize Spark workloads and storage patterns for enterprise-scale volumes.
• Ensure HIPAA/PHI compliance, auditability, and governance through Unity Catalog and tooling.
• Mentor engineers and align with cross-functional teams on data delivery standards.
• Collaborate with Decision Intelligence, Data Science, AI Engineering, and Platform teams for reliable data delivery.

Skills

SQL

Python

Spark

Databricks

Delta Lake

Medallion Architecture

Data pipelines

Data quality

Leadership

Communication

Education

Bachelor's degree in computer science or related field

Tools

Databricks

Delta Lake

Unity Catalog

Kafka

Spark Structured Streaming

Azure Data Factory

Job description

Humana Inc. seeks a Lead Data Engineer to own the NBA platform's data foundation on the Databricks Lakehouse.

You will lead architecture, delivery, quality, governance, and reliability of data products powering decision intelligence, ML, and real-time decisioning across the NBA ecosystem. You will guide a small team of engineers and contractors while remaining hands-on, shaping pipelines, feature layers, streaming patterns, and data quality controls for trusted, production-ready assets used by