Lead Data Engineer – Databricks & Data Governance

UCSF Health, AG; California; On-site; USD 140,000 - 190,000; Full time

Job summary

UCSF Health is seeking an Informatics Specialist IV to spearhead engineering, reliability, data access, and governance for a cloud-native Databricks Lakehouse. The role leads teams developing scalable data pipelines, reusable data models, and automated quality systems to support statewide analytics and program needs.

The candidate will collaborate across departments, apply advanced informatics, and drive continuous improvement of data infrastructure, security, and interoperability within a

Qualifications

• Bachelor's degree in public health, health care IT, computer science or related field or equivalent experience
• 5+ years of experience in computer or data pipeline programming and database management or public health
• Proficiency in logical data design, data warehouse design, data integration, and data modeling
• Advanced skills in SQL, Python, R, and Spark APIs (e.g., PySpark)
• Knowledge of health informatics software, standards, and public health data management
• Strong project management, leadership, and collaboration skills
• Ability to communicate complex technical information to diverse stakeholders

Responsibilities

• Lead engineering, reliability, data access, and data governance for cloud-native Databricks Lakehouse
• Manage teams building data pipelines, data models, data access frameworks, and data quality systems
• Integrate multiple enterprise data sources for statewide analytics and operations
• Develop protocols for data governance, access, interoperability, and metadata standards
• Guide scalable data pipelines, data marts, and secure data access; support continuous improvement
• Act as technical lead to engineers, informaticists, analysts, and program partners across CDPH
• Collaborate across teams to enhance enterprise data architecture and systems

Skills

SQL

Python

PySpark

Data governance

Public health informatics

Education

Bachelor's degree in public health, health care information technology, computer science

Master’s degree in computer science, informatics, public health, biostatistics

Doctorate degree in computer science, informatics, public health, biostatistics

Tools

Databricks Lakehouse

Apache Spark (PySpark)

Job description

UCSF Health is seeking an Informatics Specialist IV to spearhead engineering, reliability, data access, and governance for a cloud-native Databricks Lakehouse. The role leads teams developing scalable data pipelines, reusable data models, and automated quality systems to support statewide analytics and program needs.

The candidate will collaborate across departments, apply advanced informatics, and drive continuous improvement of data infrastructure, security, and interoperability within a