Lead Clinical Data Engineer

Yale University; New Haven (CT); Remote; USD 90,000 - 166,000; Full time

Benefits offered by this job

Yale Benefits

Job summary

Yale University is seeking a Lead Clinical Data Engineer to advance the Aligned Data Warehouse (ADW) for biomedical research. You will design and develop data pipelines, ETL jobs, and data integration across Yale School of Medicine and Yale New Haven Health System.

Role emphasizes DevOps standards, data privacy, and security in healthcare environments, with exposure to Fabric, PySpark, and BI tools. Remote work options available in New Haven, CT area.

Qualifications

• Extensive experience developing, automating, and optimizing data integration workflows with Microsoft Fabric Data Pipelines.
• Experience with Fabric Notebooks using PySpark, Spark SQL, and related technologies for transformation, enrichment, validation, and automation.
• Designing and implementing Microsoft Fabric Lakehouses and Warehouses for governance, performance, security, and scalability.
• Collaborating with business stakeholders to translate complex requirements into scalable data engineering solutions.
• Bachelor's degree in a related area and a minimum of four years of experience in the field or a related area.

Responsibilities

• Design, build, test, maintain, and control data pipelines and ETL jobs for the ADW and other research databases.
• Develop standards and procedures for database and data pipeline design, operations, and maintenance.
• Handle loading of flat files in formats such as csv, tsv, pipe-delimited, XML, and JSON.
• Produce technical documentation, SOPs, and KBAs.
• Performance tune database objects, ETL jobs, and scripts to optimize queries and processing.
• Monitor scheduled ETL jobs and ensure pipeline uptime.
• Lead troubleshooting and remediation of ETL job failures.
• Maintain patient privacy and IT security across healthcare data systems.
• Research new technologies to improve capabilities and reduce costs.
• Lead projects and data stewardship strategies.
• Collaborate with DBA teams and vendors to optimize pipelines.
• Assist with reports and dashboards in SSRS, Power BI, or similar tools.

Skills

Microsoft Fabric Data Pipelines

Fabric Notebooks

PySpark

Spark SQL

Fabric Lakehouses

Fabric Warehouses

ETL Development

Data Modeling

Education

Bachelor's degree

Master's degree

Tools

Microsoft SQL Server

SSIS

SSRS

Power BI

Fabric

Job description

The Lead Clinical Data Engineer role at Yale University focuses on advancing the Aligned Data Warehouse (ADW) for biomedical research. This position supports data pipeline and ETL development, expands data integration capabilities, and enables downstream analytics and research data sharing initiatives for Yale School of Medicine and Yale New Haven Health System.

Role Summary

As a senior technical specialist, you will lead the design and development of the ADW and extend its functionality. The work includes integrating new data sources, building and maintaining ETL and data pipelines, and supporting analytics and research data sharing efforts. You will also help manage troubleshooting for ETL failures and ensure healthcare data privacy and security across relevant systems.

Key Responsibilities

• Design, build, test, maintain, and control data pipelines and ETL jobs to integrate data into the ADW and other research databases.
• Contribute to DevOps and documentation by developing standards and procedures for database and data pipeline design, operational management, and ongoing maintenance.
• Design, build, test, maintain, and control additional data management processes, including management and loading of flat files in formats such as csv, tsv, pipe-delimited, XML, and JSON.
• Facilitate design sessions, code walkthroughs, and peer reviews; produce technical documentation, SOPs, and KBAs.
• Performance tune database objects, stored procedures, ETL jobs, and related scripts to optimize end-user queries and pipeline ETL processing.
• Monitor scheduled ETL jobs and related processes to ensure expected operation and pipeline uptime.
• Lead and coordinate troubleshooting and remediation of ETL job failures in a timely manner.
• Provide world-class customer service in interactions with customers, stakeholders, and other teams.
• Maintain patient privacy and the integrity and security of healthcare data in all databases and systems, including compliance with applicable laws, regulations, and institutional policies related to the Institutional Review Board (IRB), patient privacy, and IT cybersecurity.
• Independently research and stay current on new and emerging technologies that Scientific Computing & Data could use to improve capabilities, efficiencies, quality, or operating costs.
• Lead projects and data stewardship strategies.
• Work with the DBA team and vendors to optimize data pipelines.
• Assist with the design, build, test, and performance tuning of reports and dashboards in SSRS, Microsoft Power BI, or similar business intelligence tools.
• Perform related duties as assigned or requested.

Required Qualifications

• Extensive experience developing, automating, and optimizing data integration workflows with a focus on Microsoft Fabric Data Pipelines supporting large-scale ingestion, transformation, and integration workloads.
• Proven ability to design, develop, and optimize scalable data models, semantic layers, and storage architectures for enterprise reporting, self-service analytics, operational intelligence, and advanced analytics initiatives.
• Experience developing and maintaining Fabric Notebooks using PySpark, Spark SQL, and related technologies for transformation, enrichment, validation, and automation.
• Demonstrated experience designing and implementing Microsoft Fabric Lakehouses, Warehouses, and enterprise data organization strategies supporting governance, performance, security, and scalability requirements.
• Ability to collaborate with business stakeholders, analysts, architects, and technical teams to translate complex business requirements into scalable data engineering solutions.
• A bachelor's degree in a related area and a minimum of four years of experience in the field or a related area.

Technologies

• Microsoft SQL Server, T-SQL, SSIS, SQL Agent
• Microsoft Fabric, Microsoft Fabric Data Pipelines, Microsoft Fabric Lakehouses, Microsoft Fabric Warehouses
• Metadata-driven dynamic SQL
• Linux, Windows
• OMOP Common Data Model, OHDSI's Observational Medical Outcomes Partnership (OMOP) Common Data Model
• SSRS, Microsoft Power BI
• PySpark, Spark SQL, Fabric Notebooks
• Large Language Models, GPU-based software
• csv, tsv, pipe-delimited, XML, JSON
• Epic's data warehouse; Epic's Clarity & Caboodle databases
• PCORnet, OnCore, IRES - IRB

Preferred Qualifications

• In-depth knowledge of technology areas impacting the role, including healthcare technology experience.
• Experience integrating electronic health record (EHR), clinical, research, financial, or operational healthcare data in cloud-based analytics platforms.
• Certification and experience with Epic’s Clarity & Caboodle databases.
• Experience with the OMOP Common Data Model, PCORnet, and other research data models, including familiarity with research administrative systems such as OnCore or IRES - IRB.

Preferred Education and Experience

• Master's degree.
• Experience in data engineering, analytics engineering, or data integration, including at least five (5) years designing, developing, and maintaining enterprise-scale healthcare data platforms, ETL processes, and cloud-based data solutions.

Compensation, Work Model, and Employment Details

• Location: New Haven, CT (remote)
• Salary: USD 90,000 - 165,750 per year
• Time type: Full time
• Duration type: Staff
• Work model: Remote
• Job category: Professional
• Compensation grade: Administration & Operations (Manager; Program Leader (P6))
• Salary range: $90,000.00 - $165,750.00
• Background check requirements: Pre-employment background screening may include motor vehicle, DOT certification, drug testing, and credit checks based on position description and job requirements; offers are contingent on successful completion.
• Health requirements: Certain positions may require vaccinations, tests, or examinations as required by law, regulation, or university policy.

Benefits

A general description of benefits for this Managerial & Professional position is available at: Yale Benefits.

Additional Information

• Job posting date: 10/13/2026
• Bargaining unit: NON
• Compensation grade profile: Manager; Program Leader (P6)
• Posting disclaimer: Salary offers are determined by qualifications, experience, skills, and education relative to role requirements, along with the role’s grade profile and internal/external market conditions. The description provides a representative summary of essential functions and does not declare specific duties and responsibilities. Employees are assigned specific duties by the hiring department.
• Note: Yale University is a tobacco-free campus.