Senior Data Engineer: Databricks & Cloud Pipelines

Geisinger; Danville (PA); On-site; USD 120,000 - 150,000; Full time

Benefits offered by this job

Healthcare benefits from day one

Vision and dental coverage

Domestic partner benefits

Job summary

Geisinger is seeking a Senior Data Engineer to design, build, and maintain cloud-based data pipelines using Databricks, PySpark, and SQL for large-scale clinical and genomic datasets.

The role focuses on modeling data within Databricks, modernizing legacy ETL processes, and ensuring data quality, performance, and reliability across research workflows. Collaboration with multidisciplinary teams is essential.

Qualifications

• Proven experience with Databricks and cloud-based data engineering platforms.
• Strong experience with SQL and Python/PySpark for developing and optimizing data processing pipelines.
• Experience with Apache Spark or other distributed data processing technologies.
• Experience designing and implementing data models and data warehouse schemas for large-scale datasets.
• Experience developing, maintaining, or modernizing ETL processes and data pipelines.

Responsibilities

• Designs, develops, and maintains PySpark and SQL data pipelines for transforming clinical data for research use.
• Develops and maintains data models for clinical and other research data within the Databricks environment.
• Reviews, analyzes, modifies, creates, debugs, designs, and tests data ingestion and processing pipelines using modern development methodologies and tools.
• Defines and optimizes data warehouse schemas and data models to support efficient querying, storage, and analytical workflows.
• Ensures scalability, reliability, and maintainability of data pipelines and data architecture supporting large clinical and research datasets.
• Implements and monitors data quality checks and validation processes to ensure accuracy, completeness, and integrity of data throughout the pipeline.
• Troubleshoots and optimizes Spark and SQL workloads, data pipelines, and Databricks compute resources to improve performance and reliability.
• Modernizes existing on-premises Microsoft SQL Server ETL processes by designing and implementing equivalent cloud-based data pipelines and workflows in Databricks.
• Provides general Databricks computing support, including assisting users with compute resource selection, troubleshooting, and optimization of Spark and SQL workloads.
• Participates in design and application architecture discussions and assists in the development and documentation of technical approaches and solutions for team stakeholders.
• Responsible for the process, life cycle and creation of the software to assist users in completing a computer task or program.
• Implements code and documents all system changes based on assignments and timelines.
• Develops and follows department coding and user interface guidelines.
• Provides code review, testing, debugging, technical documentation, general testing instructions, go-live planning, assistance in go-live moves, and post-live support.
• Communicates all progress, roadblocks, and issues to the team and management in a timely manner.
• Collaborates with team members to identify and resolve issues, improve system performance, and ensure code quality.
• Provides training and documentation to team members on new development assignments.
• Trains and mentors staff members.
• Assists in the development and documentation of a technical approach for a solution to the team stakeholders.

Skills

Databricks

SQL

Python/PySpark

Apache Spark

Data modeling

ETL

AWS

Data quality

Communication

Education

Bachelor’s Degree-Technology Related Degree

High School Diploma or GED

Tools

SQL Server

AWS Cloud Services

Job description

Geisinger is seeking a Senior Data Engineer to design, build, and maintain cloud-based data pipelines using Databricks, PySpark, and SQL for large-scale clinical and genomic datasets.

The role focuses on modeling data within Databricks, modernizing legacy ETL processes, and ensuring data quality, performance, and reliability across research workflows. Collaboration with multidisciplinary teams is essential.