Senior Data Engineer: Scalable Pipelines & AI-Driven Data

Humana Inc; City of Albany (NY); Remote; USD 118,000 - 162,000; Full time

Benefits offered by this job

Health benefits day 1

Annual incentive plan

401(k) match

Well-being program

Paid volunteer time off

Generous PTO

Job summary

Humana is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines for ingestion and transformation. You will work with Azure Databricks, SAS, SSIS, SQL, Python, and Spark to deliver data-driven insights across healthcare data ecosystems.

The role emphasizes collaboration with software engineering and analytics teams to ensure data quality, reliability, and performance while aligning with enterprise architecture standards.

Qualifications

• Bachelor’s degree in a science, engineering, or computing related discipline or equivalent practical experience.
• 8+ years of experience designing and building production-grade batch data pipelines for data ingestion and transformation.
• 8+ years of overall experience in data engineering or related roles.
• Strong proficiency in SQL, Databricks, Python, Apache Spark, Git, and Azure Services (Synapse, Azure ADLS Gen 2).
• Strong understanding of relational database principles, data modeling, database management, and data warehousing.
• Strong knowledge of Big Data querying tools.
• Experience optimizing large-scale data pipelines and ETL workflows.
• Excellent communication and interpersonal skills.

Responsibilities

• Design, build, and maintain scalable data pipelines and workflows using Azure Databricks.
• Develop and maintain SAS programs and SSIS packages for intermediate and downstream data processing.
• Write, optimize, and maintain complex SQL queries to support data transformation and analytical use cases.
• Design and build ETL/ELT pipelines integrating data from multiple internal and external source systems.
• Select, implement, and support Big Data tools and frameworks for data processing and integration.
• Monitor, troubleshoot, and resolve complex data processing issues with root-cause analysis and preventative fixes.
• Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
• Recommend and support infrastructure improvements to meet evolving data volume and processing demands.
• Partner with product management and SRE teams to define requirements and translate them into actionable engineering work.
• Collaborate with engineers, stakeholders, and cross-functional teams to align data strategy, architecture, and outcomes.

Skills

SQL

Databricks

Python

Apache Spark

Git

Azure Synapse

Azure ADLS Gen 2

Big Data querying tools

Education

Bachelor’s degree in science, engineering, or computing related discipline or equivalent practical experience

Tools

SAS

SSIS

Snowflake

Job description

Humana is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines for ingestion and transformation. You will work with Azure Databricks, SAS, SSIS, SQL, Python, and Spark to deliver data-driven insights across healthcare data ecosystems.

The role emphasizes collaboration with software engineering and analytics teams to ensure data quality, reliability, and performance while aligning with enterprise architecture standards.