Senior Public Health Data Engineer (Databricks)

UCSF Health, AG; Oakland (CA); On-site; USD 120,000 - 150,000; Full time

Job summary

UCSF Health, AG in Oakland, CA seeks an experienced Clinical Informatics Specialist III to lead cloud-native data engineering initiatives in a public health context. You will design scalable pipelines and data marts in Databricks, collaborating with data scientists and program partners to deliver interoperable data products for analytics and decision-making.

The role requires strong programming and data engineering skills (SQL, Python, PySpark), experience with Databricks/Delta Lake, and

Qualifications

• Bachelor's degree in public health, health IT, CS or related field, plus 3 years in data pipeline programming and/or public health.
• Strong SQL, R, Python, and PySpark; familiarity with Spark APIs.
• Experience with Databricks, Delta Lake, or cloud-based data lake architectures.
• Knowledge of clinical/public health informatics and data governance.
• Excellent written and verbal communication; ability to work with senior staff.

Responsibilities

• Design, develop, test, and maintain scalable data pipelines and enterprise data marts in Databricks.
• Support enhanced public health surveillance and operational data systems.
• Create standardized data models and curated datasets.
• Improve interoperability across CDPH data systems and with LHJs.
• Ensure secure, automated ingestion, transformation, storage, and distribution of data.
• Participate in data architecture, modeling, and data quality efforts.
• Develop tools and monitoring capabilities for reliable pipelines.
• Prototype new engineering approaches to improve scalability and maintainability.
• Apply informatics knowledge to support data governance and data quality.

Skills

SQL

Python

PySpark

Databricks

Delta Lake

Data Modeling

Data governance

Cloud platforms

Communication skills

Education

Bachelor's degree in public health / computer science / information technology

Master's degree in public health, informatics, or related field

Tools

Databricks

Delta Lake

Job description

UCSF Health, AG in Oakland, CA seeks an experienced Clinical Informatics Specialist III to lead cloud-native data engineering initiatives in a public health context. You will design scalable pipelines and data marts in Databricks, collaborating with data scientists and program partners to deliver interoperable data products for analytics and decision-making.

The role requires strong programming and data engineering skills (SQL, Python, PySpark), experience with Databricks/Delta Lake, and