Senior Platform Data Engineer

Remotedxb; Dubai; On-site; AED 300,000 - 500,000; Full time

Benefits offered by this job

Health benefits (vision & dental)

Domestic partner benefits

Job summary

Geisinger is seeking a senior data engineer to stream data from Epic SDE, ADT feeds, and clinical sources into Databricks, and curate shared clinical feature tables for AI model training and monitoring.

You will own RAG infrastructure, including ingestion, chunking, and embedding pipelines, administer vector databases, and build retrieval pipelines with hybrid search and reranking. Strong Databricks and real-time data skills are required.

Qualifications

• 5+ years of data engineering with batch and streaming pipelines.
• Expert proficiency with Databricks (Delta Live Tables, PySpark, Unity Catalog, Feature Store).
• Hands-on experience with real-time data ingestion frameworks like Kafka or Spark Structured Streaming.
• Strong SQL and Python (pandas, PySpark) skills.
• Experience administering Databricks workspaces and compute management.
• Understanding of data governance, lineage, and quality monitoring.
• Bachelor's degree in a related field; Master's degree preferred.

Responsibilities

• Stream data from Epic SDE, ADT feeds, and clinical sources into Databricks.
• Curate shared clinical feature tables in Databricks/Unity Catalog for AI model training and monitoring.
• Own and operate RAG infrastructure, including ingestion, chunking, and embedding pipelines.
• Administer vector databases, including schema design, indexing, and performance tuning.
• Build and maintain retrieval pipelines using hybrid search and reranking.
• Manage Databricks workspace configuration, Unity Catalog governance, and cost monitoring.
• Administer the Feature Store and manage user access controls.

Skills

Databricks

PySpark

SQL

Python

Streaming data

Data governance

Education

Bachelor's degree

Master's degree preferred

Tools

Databricks

Delta Live Tables

Unity Catalog

Feature Store

Kafka

Spark Structured Streaming

PySpark

Job description

Responsibilities

• Stream data from Epic SDE, ADT feeds, and clinical sources into Databricks
• Curate shared clinical feature tables in Databricks/Unity Catalog for AI model training and monitoring
• Own and operate RAG (Retrieval-Augmented Generation) infrastructure, including ingestion, chunking, and embedding pipelines
• Administer vector databases, including schema design, indexing, and performance tuning
• Build and maintain retrieval pipelines using hybrid search and reranking
• Manage Databricks workspace configuration, Unity Catalog governance, and cost monitoring
• Administer the Feature Store and manage user access controls

Requirements

• 5+ years of experience in data engineering with batch and streaming pipelines
• Expert proficiency with Databricks (Delta Live Tables, PySpark, Unity Catalog, Feature Store)
• Hands-on experience with real-time data ingestion frameworks like Kafka or Spark Structured Streaming
• Strong SQL and Python (pandas, PySpark) skills
Experience administering Databricks workspaces and compute management
• Understanding of data governance, lineage, and quality monitoring
• Bachelor's degree in a related field; Master's degree preferred

Preferred Qualifications

• Familiarity with clinical data models and healthcare data sources (EHR, ADT, claims)
• Experience with Epic data extraction methods (SDE, FHIR, epic-ws)

Benefits

• Healthcare benefits including vision and dental from day one
• Benefits for domestic partners

About the Company

Geisinger is a healthcare system serving over 1 million people, comprising nine hospital campuses, a health plan, two research centers, and the Geisinger Commonwealth School of Medicine.