Senior Data Engineer: Databricks on AWS (Remote)

Geisinger; United States; Remote; USD 120,000 - 160,000; Full time

Benefits offered by this job

Healthcare benefits

Vision

Dental

Domestic partners

Job summary

The Geisinger Research Data Core is seeking a Senior Data Engineer to design and maintain scalable data pipelines in Databricks and AWS, transforming clinical data for research use. You will model clinical data, modernize on-prem ETL to the cloud, and ensure data quality and performance across large research datasets.

You will collaborate with a multidisciplinary team, provide guidance on data architecture, and engage in the full software development life cycle to deliver robust analytical

Qualifications

• Databricks and cloud-based data engineering platforms experience required.
• Strong SQL and Python/PySpark for data pipelines.
• Experience with Apache Spark or similar distributed processing.
• Experience designing data models and data warehouse schemas for large datasets.
• ETL processes and data pipelines development/modernization.
• Relational databases and SQL Server experience.
• AWS cloud services familiarity for data architectures.
• Data quality, validation, monitoring, logging, troubleshooting practices.
• Understanding of pipeline performance optimization for scalable processing.
• Excellent communication to relay progress and issues to teams.
• Ability to collaborate with multidisciplinary stakeholders.

Responsibilities

• Designs, develops, and maintains PySpark and SQL data pipelines for transforming clinical data for research use.
• Develops and maintains data models for clinical and research data within Databricks.
• Reviews and optimizes data ingestion and processing pipelines with modern tooling.
• Defines and optimizes data warehouse schemas for efficient querying and analytics.
• Ensures scalability and reliability of pipelines and architecture for large datasets.
• Implements data quality checks and validation across pipelines.
• Troubleshoots Spark/SQL workloads and optimizes Databricks compute resources.
• Modernizes on-prem ETL processes to cloud-based Databricks equivalents.
• Provides Databricks support and resource optimization guidance to users.

Skills

Databricks experience

SQL

Python/PySpark

Apache Spark

Data modeling

ETL pipelines

Data quality & monitoring

Communication

Team collaboration

Mentoring

Education

High School Diploma or GED

Bachelor’s Degree in Technology Related Field

Tools

Databricks

AWS

SQL Server

PySpark

Job description

The Geisinger Research Data Core is seeking a Senior Data Engineer to design and maintain scalable data pipelines in Databricks and AWS, transforming clinical data for research use. You will model clinical data, modernize on-prem ETL to the cloud, and ensure data quality and performance across large research datasets.

You will collaborate with a multidisciplinary team, provide guidance on data architecture, and engage in the full software development life cycle to deliver robust analytical