Lead Data Engineer - NBA
Humana Inc
Get more replies from employers
Send a job-specific resume in minutes.
Job summary
Humana Inc. seeks a Lead Data Engineer to own the NBA platform's data foundation on the Databricks Lakehouse.
You will lead architecture, delivery, quality, governance, and reliability of data products powering decision intelligence, ML, and real-time decisioning across the NBA ecosystem. You will guide a small team of engineers and contractors while remaining hands-on, shaping pipelines, feature layers, streaming patterns, and data quality controls for trusted, production-ready assets used by
Qualifications
- Bachelor's degree in computer science or related field.
- 7+ years data engineering experience with 1-2 years in a lead engineer or technical leadership capacity.
- Expert-level SQL and Python skills with significant experience developing and operating Spark-based data platforms.
- Deep experience with Databricks, Delta Lake, and modern Lakehouse architectures.
- Experience designing and operating Medallion Architecture (Bronze/Silver/Gold) implementations at enterprise scale.
- Experience building and supporting both batch and streaming data pipelines in production environments.
- Strong understanding of data quality engineering, validation frameworks, lineage, monitoring, and operational support models.
- Experience designing secure and compliant data platforms in regulated environments.
- Demonstrated ability to lead a small engineering team while remaining an active hands-on contributor.
- Strong communication skills and the ability to explain technical tradeoffs and architecture decisions to engineering, business, and compliance stakeholders.
Responsibilities
- Own end-to-end pod delivery for data engineering.
- Design and evolve the NBA Databricks Lakehouse architecture across Bronze/Silver/Gold layers.
- Build and maintain scalable batch and real-time pipelines from healthcare datasets.
- Lead the design of Gold-layer feature tables and reusable data products for model training and decisioning.
- Design Kafka- and Spark-based streaming ingestion for near real-time decision making.
- Establish data quality, lineage, monitoring, and governance standards across the platform.
- Optimize Spark workloads and storage patterns for enterprise-scale volumes.
- Ensure HIPAA/PHI compliance, auditability, and governance through Unity Catalog and tooling.
- Mentor engineers and align with cross-functional teams on data delivery standards.
- Collaborate with Decision Intelligence, Data Science, AI Engineering, and Platform teams for reliable data delivery.
Skills
Education
Tools
Job description
Become a part of our caring community
The Lead Data Engineer owns the NBA platform's data foundation on the Databricks Lakehouse. This role is responsible for the architecture, delivery, quality, governance, and operational reliability of the data products that power decision intelligence, machine learning, reinforcement learning, agentic AI, and real-time decisioning across the platform. You lead a small team of engineers and contractors while remaining deeply hands-on, building and evolving the pipelines, feature layers, streaming architectures, and data quality controls that turn raw healthcare data into trusted, production-ready assets consumed across the NBA ecosystem.
Pod delivery - Own end-to-end delivery for the data engineering pod, including planning, execution, quality, and operational readiness.
Lakehouse architecture - Own the design and evolution of the NBA Databricks Lakehouse, including Bronze, Silver, and Gold layer standards, data lineage, and governance patterns.
Data pipeline engineering - Design, build, and maintain scalable batch and real-time pipelines that process member, clinical, claims, behavioral, engagement, and socioeconomic datasets.
Feature platform ownership - Lead the design and operation of Gold-layer feature tables and reusable data products that support model training, scoring, reinforcement learning, and decision intelligence workloads.
Streaming and event architecture - Design and implement Kafka- and Spark Structured Streaming-based ingestion and processing patterns that enable near real-time decision-making.
Data quality and observability - Establish platform-wide standards for data validation, monitoring, lineage, reconciliation, alerting, and operational visibility; treat data quality issues as production incidents.
Performance and optimization - Drive optimization of Spark workloads, Delta Lake storage patterns, partitioning strategies, and query performance to support enterprise-scale volumes efficiently.
Data governance - Ensure compliance with PHI, HIPAA, auditability, retention, and governance requirements through Unity Catalog, lineage tooling, and secure data management practices.
Team leadership - Lead and mentor engineers and contractors; conduct code reviews, establish engineering standards, and drive adoption of best practices across the pod.
Cross-team coordination - Partner with Decision Intelligence, Data Science, AI Engineering, and Platform Engineering teams to ensure reliable, governed, and performant data delivery throughout the NBA ecosystem.
Use your skills to make an impact
- Bachelor's degree in computer science or related field
- 7 years of data engineering experience with at least 1-2 years in a lead engineer or technical leadership capacity.
- Expert-level SQL and Python skills with significant experience developing and operating Spark-based data platforms.
- Deep experience with Databricks, Delta Lake, and modern Lakehouse architectures.
- Experience designing and operating Medallion Architecture (Bronze/Silver/Gold) implementations at enterprise scale.
- Experience building and supporting both batch and streaming data pipelines in production environments.
- Strong understanding of data quality engineering, validation frameworks, lineage, monitoring, and operational support models.
- Experience designing secure and compliant data platforms in regulated environments.
- Demonstrated ability to lead a small engineering team while remaining an active hands-on contributor.
- Strong communication skills and the ability to explain technical tradeoffs and architecture decisions to engineering, business, and compliance stakeholders.
- Deep experience with Databricks Feature Store, Delta Live Tables, Unity Catalog, and Databricks Workflows.
- Experience supporting machine learning, reinforcement learning, recommendation engines, or decision intelligence platforms.
- Experience with Kafka, event streaming architectures, and real-time feature engineering pipelines.
- Experience designing feature stores, feature-serving architectures, and model-data integration patterns.
- Familiarity with Azure Data Factory, Azure Event Hubs, Azure Data Lake Storage, and broader Azure data platform services.
- Experience implementing data observability and quality platforms such as Great Expectations, Monte Carlo, or equivalent solutions.
- Experience integrating large external datasets including CMS, CDC, USDA, consumer, behavioral, or social determinants of health data sources.
- Background in healthcare, insurance, or another regulated industry with PHI/HIPAA requirements.
We build with modern AI development tools (such as Claude and GitHub Copilot) and expect everyone on the team to use them to work faster and at higher quality.
- Architect, implement, and operate microservices that deliver:
- Action and variant metadata
- Context-aware policy and eligibility evaluation
- Versioned, read-optimized APIs for high-performance runtime consumption
- Guarantee that services are:
- Highly available, with low latency
- Horizontally scalable for increased demand
- Backward compatible to support safe evolution and upgrades
- Apply industry best practices for API design, schema evolution, service isolation, and secure integration.
- Design, deploy, and maintain resilient database schemas to support:
- Comprehensive action and variant catalogs
- Versioning, lifecycle management, and effective dating
- Rule bindings and complex metadata relationships
- Select and operate appropriate data stores (relational, document, key-value) tailored to workload and scalability requirements.
- Implement and monitor:
- Schema migration and backward compatibility strategies
- Indexing and query optimization for performance
- Data integrity, consistency, and reliability
- Auditability and traceability for compliance and governance
- Integrate and manage enterprise-grade rules engines to support:
- Eligibility, constraints, and business policies
- Suppression, cooldowns, exclusions, and other operational guardrails
- Policy-driven allow/deny logic
- Work with technologies such as Drools (DRL/DMN), IBM ODM, DMN-based services, OPA/Rego, or similar.
- Ensure rule execution is deterministic, versioned, stateless, and free from unintended side effects.
- Utilize AI-powered and agentic tools to:
- Generate and refactor database schemas and service logic
- Streamline rule authoring, validation, and ongoing refactoring
- Detect and address conflicting or redundant rules early in the development cycle
- Automatically produce comprehensive test cases and explore edge scenarios
- Apply AI responsibly to reduce manual effort while safeguarding clarity, correctness, and strong governance.
- Develop automated tests for:
- Database migrations and schema changes
- Service-level contracts and API backward compatibility
- Rules behavior, precedence, and edge cases
- Ensure platforms are fully observable and resilient, featuring:
- Structured logging, metrics, and alerting
- Clear error handling, fallback strategies, and robust incident management
- Support audit and compliance requirements through traceable and reproducible system behavior
- Work closely with platform, data, and machine learning teams to maintain clean integration points and shared standards.
- Participate actively in architecture and design reviews to uphold platform quality.
- Mentor junior engineers and contribute to technical standards and best practices.
- Bachelor's degree in computer science or related field
- 8 or more years of progressive IT experience as a senior developer in large IT projects
- 2 or more years of project leadership experience
- Must be passionate about contributing to an organization focused on continuously improving consumer experiences
- Master's Degree
Work Style: Remote/Hybrid - Preferably Boston, MA. Occasional travel to Humana's offices for training or meetings may be required.
Work Hours : Typical business hours are Monday-Friday, 8 hours/day, 5 days/week-- some flexibility might be possible, depending on business needs.
Very minimal travel might be required for training, meetings, and/or conferences
As part of our hiring process, we will be using on-demand technology provided by Hire Vue, a third-party vendor. This technology provides our team of recruiters and hiring managers with an enhanced method for decision-making through on-demand candidate assessments.
If you are selected to move forward from your application prescreen, you will receive correspondence inviting you to participate in an on-demand assessment with pre-determined questions. You should anticipate the assessment to take approximately 10-15 minutes.
Your on-demand assessment will be reviewed, and you will subsequently be informed if you will be moving forward to next round of interviews.
Should you be extended a formal employment offer you will receive a request to enter your SSN into our Workday system to scan for duplicate profiles.
Work at Home Requirements: To ensure Home or Hybrid Home/Office employees' ability to work effectively, the self-provided internet service of Home or Hybrid Home/Office employees must meet the following criteria: At minimum, a download speed of 25 Mbps and an upload speed of 10 Mbps is required; wireless, wired cable or DSL connection is suggested. In certain roles, the minimum recommended internet speed required by Humana may not be sufficient for business needs. Humana reserves the right to require associates to upgrade their internet service if necessary. Work from a dedicated space lacking ongoing interruptions to protect member PHI / HIPAA information.
Travel: While this is a remote position, occasional travel to Humana's offices for training or meetings may be required.
40
The compensation range below reflects a good faith estimate of starting base pay for full time (40 hours per week) employment at the time of posting. The pay range may be higher or lower based on geographic location and individual pay will vary based on demonstrated job related skills, knowledge, experience, education, certifications, etc.
$142,300 - $195,700 per year
This job is eligible for a bonus incentive plan. This incentive opportunity is based upon company and/or individual performance.
Humana, Inc. and its affiliated subsidiaries (collectively, "Humana") offers competitive benefits that support whole-person well-being. Associate benefits are designed to encourage personal wellness and smart healthcare decisions for you and your family while also knowing your life extends outside of work. Among our benefits, Humana provides medical, dental and vision benefits, 401(k) retirement savings plan, time off (including paid time off, company and personal holidays, paid parental and caregiver leave), short-term and long-term disability, life insurance and many other opportunities.
About us
About Humana: Humana Inc. (NYSE: HUM) is a leading U.S. healthcare company. Through our Humana insurance services and our CenterWell healthcare services, we make it easier for the millions of people we serve to achieve their best health - delivering the care and service they need to when they need it. These efforts are leading to a better quality of life for people with Medicare and Medicaid, families, individuals, military service personnel, and communities at large. Learn more about what we offer at Humana.com and at CenterWell.com.
It is the policy of Humana not to discriminate against any employee or applicant for employment because of race, color, religion, sex, sexual orientation, gender identity, national origin, age, marital status, genetic information, disability or protected veteran status. It is also the policy of Humana to take affirmative action, in compliance with Section 503 of the Rehabilitation Act and VEVRAA, to employ and to advance in employment individuals with disability or protected veteran status, and to base all employment decisions only on valid job requirements. This policy shall apply to all employment actions, including but not limited to recruitment, hiring, upgrading, promotion, transfer, demotion, layoff, recall, termination, rates of pay or other forms of compensation and selection for training, at all levels of employment.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
