Lead Clinical Data Engineer — Remote ETL & Pipelines

Yale University; New Haven (CT); Remote; USD 90,000 - 166,000; Full time

Benefits offered by this job

Yale Benefits

Job summary

Yale University is seeking a Lead Clinical Data Engineer to advance the Aligned Data Warehouse (ADW) for biomedical research. You will design and develop data pipelines, ETL jobs, and data integration across Yale School of Medicine and Yale New Haven Health System.

Role emphasizes DevOps standards, data privacy, and security in healthcare environments, with exposure to Fabric, PySpark, and BI tools. Remote work options available in New Haven, CT area.

Qualifications

• Extensive experience developing, automating, and optimizing data integration workflows with Microsoft Fabric Data Pipelines.
• Experience with Fabric Notebooks using PySpark, Spark SQL, and related technologies for transformation, enrichment, validation, and automation.
• Designing and implementing Microsoft Fabric Lakehouses and Warehouses for governance, performance, security, and scalability.
• Collaborating with business stakeholders to translate complex requirements into scalable data engineering solutions.
• Bachelor's degree in a related area and a minimum of four years of experience in the field or a related area.

Responsibilities

• Design, build, test, maintain, and control data pipelines and ETL jobs for the ADW and other research databases.
• Develop standards and procedures for database and data pipeline design, operations, and maintenance.
• Handle loading of flat files in formats such as csv, tsv, pipe-delimited, XML, and JSON.
• Produce technical documentation, SOPs, and KBAs.
• Performance tune database objects, ETL jobs, and scripts to optimize queries and processing.
• Monitor scheduled ETL jobs and ensure pipeline uptime.
• Lead troubleshooting and remediation of ETL job failures.
• Maintain patient privacy and IT security across healthcare data systems.
• Research new technologies to improve capabilities and reduce costs.
• Lead projects and data stewardship strategies.
• Collaborate with DBA teams and vendors to optimize pipelines.
• Assist with reports and dashboards in SSRS, Power BI, or similar tools.

Skills

Microsoft Fabric Data Pipelines

Fabric Notebooks

PySpark

Spark SQL

Fabric Lakehouses

Fabric Warehouses

ETL Development

Data Modeling

Education

Bachelor's degree

Master's degree

Tools

Microsoft SQL Server

SSIS

SSRS

Power BI

Fabric

Job description

Yale University is seeking a Lead Clinical Data Engineer to advance the Aligned Data Warehouse (ADW) for biomedical research. You will design and develop data pipelines, ETL jobs, and data integration across Yale School of Medicine and Yale New Haven Health System.

Role emphasizes DevOps standards, data privacy, and security in healthcare environments, with exposure to Fabric, PySpark, and BI tools. Remote work options available in New Haven, CT area.