Senior Data Engineer - AWS & Databricks (Remote)

UST; Chicago (IL); Remote; USD 92,000 - 138,000; Full time

Benefits offered by this job

401(k) matching

Medical, dental, and vision

Paid time off

HSA/FSA for healthcare

Job summary

UST is seeking a Lead Data Engineer to architect, build, and optimize cloud-native data platforms. You will lead enterprise data pipelines on Databricks and AWS, leveraging Iceberg lakehouses and Unity Catalog for governance, while ensuring HIPAA compliance and PHI protection.

You will mentor engineers, drive best practices, and shape scalable data architectures for Medicare Advantage datasets and healthcare interoperability standards such as HL7 and FHIR.

Qualifications

• 7+ years data engineering experience.
• At least 2 years in a technical lead role.
• Production experience building and optimizing large-scale pipelines with Databricks.
• Production experience deploying data infrastructure on AWS (S3, EMR, Lambda, IAM, Glue).
• Hands-on experience processing Medicare Advantage data (CMS enrollment, RAPS/EDPS submissions, MMR/MOR files, HCC risk models).
• Experience with healthcare interoperability standards (FHIR and HL7).
• RBAC and PHI security implementation experience.

Responsibilities

• Lead design, implementation, and maintenance of robust data pipelines on AWS using Databricks.
• Architect scalable lakehouse structures with Apache Iceberg for healthcare data.
• Establish data governance and lineage using Databricks Unity Catalog.
• Implement RBAC, data masking, and row-level security for PHI.
• Architect and scale ingestion for HL7 and FHIR data feeds into the lakehouse.
• Oversee end-to-end processing of Medicare Advantage data assets and reporting.
• Ensure HIPAA compliance and robust data governance.
• Mentor junior engineers and drive best practices across the team.

Skills

Python

PySpark

SQL

Apache Iceberg

Databricks Unity Catalog

Healthcare data

Data pipelines

Security & compliance

Tools

Databricks

AWS

S3

EMR

Lambda

IAM

Glue

Job description

UST is seeking a Lead Data Engineer to architect, build, and optimize cloud-native data platforms. You will lead enterprise data pipelines on Databricks and AWS, leveraging Iceberg lakehouses and Unity Catalog for governance, while ensuring HIPAA compliance and PHI protection.

You will mentor engineers, drive best practices, and shape scalable data architectures for Medicare Advantage datasets and healthcare interoperability standards such as HL7 and FHIR.