Data Engineer (AI/ML)

001_BCBSA Blue Cross and Blue Shield Association; Chicago (IL); On-site; USD 100,800 - 138,600; Full time

Benefits offered by this job

Paid time off

Medical/dental/vision insurance

Generous 401(k) matching

Lifestyle spending account

Job summary

The 001_BCBSA Blue Cross and Blue Shield Association is seeking a skilled Data Engineer to design and optimize scalable data pipelines for analytics and product platforms. The ideal candidate will have over 5 years of experience in data engineering, particularly in cloud environments, and will contribute to innovative Machine Learning and Generative AI projects.

Responsibilities include working with cross-functional teams to build high-performance pipelines, ensuring data integrity, and supporting compliance with healthcare standards. The position offers a competitive salary and comprehensive benefits.

Qualifications

• 5+ years of experience in data engineering, including building and managing pipelines in cloud-based environments.
• Hands-on experience with AWS AI/ML data services.
• Strong analytical and problem-solving skills.

Responsibilities

• Design, build, and maintain reliable, high-performance data pipelines.
• Collaborate with Data Architects, Data Scientists, and Analysts.
• Implement and maintain data validation frameworks.

Skills

Data engineering

Machine Learning

Generative Artificial Intelligence

PySpark

AWS

Data pipelines

Python

SQL

Airflow

Kubernetes

Education

Bachelor's or Master's degree in Computer Science or related field

Tools

Databricks

AWS Glue

EMR

Snowflake

Job description

Job Overview

The Data Engineer will design, build, and optimize scalable, secure data pipelines that power analytics and product platforms. The focus will be on Machine Learning (ML) and Generative Artificial Intelligence (GenAI) workloads, contributing to innovation and ensuring compliance with healthcare industry standards.

Responsibilities

• Design, build, and maintain reliable, high-performance data pipelines for large-scale structured and unstructured healthcare data.
• Use PySpark and modern cloud-based tools (Databricks, AWS Glue, EMR, Snowflake) to transform and process data efficiently.
• Support ingestion, transformation, and validation processes that ensure data consistency, integrity, and availability.
• Partner with Data Architects, Data Scientists, and Analysts to translate business needs into scalable engineering solutions.
• Collaborate with platform and DevOps teams to deploy, scale, and monitor data pipelines using Airflow and Kubernetes.
• Participate in code reviews, documentation, and continuous improvement efforts across the engineering team.
• Implement and maintain data validation frameworks to ensure pipeline accuracy and completeness.
• Contribute to best practices in version control, metadata management, and reproducibility.
• Stay current with emerging technologies in data engineering and cloud computing, recommending improvements to existing infrastructure.
• Participate in performance tuning, cost optimization, and scaling strategies for cloud-based data systems.
• Identify automation opportunities to streamline ETL/ELT processes and reduce operational overhead.
• Share knowledge and mentor junior team members on tools, techniques, and best practices.
• Promote a culture of collaboration, innovation, and continuous learning within the engineering organization.
• Support compliance with SOC 2, HIPAA, and GDPR by adhering to established data privacy and security practices.

Qualifications

Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.

Experience: 5+ years of experience in data engineering, including building and managing pipelines in cloud-based environments.

Technical Skills & Knowledge

• Experience with building and operationalizing data foundations that support ML and generative AI use cases, including feature pipelines, training/inference data preparation, and retrieval-ready datasets (e.g., embeddings and vector stores).
• Familiarity with GenAI skills and adjacent tooling (foundation models, prompt engineering, RAG, embeddings/vector databases, GenAI orchestration frameworks).
• Hands‑on experience with AWS AI/ML and data services, including Amazon Bedrock, Bedrock Agent Core, SageMaker, Glue, and EMR.
• Experience designing and optimizing data architectures, including data foundations that support ML and GenAI workloads.
• Hands‑on experience with workflow orchestration (Airflow) and containerization (Kubernetes).
• Proficiency in Python, SQL, and distributed data frameworks (PySpark, Databricks, AWS Glue, EMR).
• Working knowledge of cloud platforms (AWS or Azure) and data warehouses (Snowflake).
• Familiarity with NoSQL and relational databases, as well as data modeling best practices.
• Strong analytical, problem‑solving, and communication skills.
• Understanding of compliance frameworks (SOC 2, HIPAA) and secure data management principles.
• Experience working with healthcare datasets or knowledge of healthcare standards (HIPAA, HL7, FHIR) preferred.

Compliance & Healthcare Knowledge: Knowledge of SOC 2, HIPAA, GDPR compliance; understanding of healthcare data security principles and standards such as HL7 and FHIR.

Compensation

Salary range: 100,800.00 - 138,600.00

Benefits

Paid time off, 11 holidays, medical/dental/vision insurance, generous 401(k) matching, lifestyle spending account and many other benefits to eligible employees.