Senior Data Engineering Developer

Geisinger; United States; Remote; USD 120,000 - 160,000; Full time

Benefits offered by this job

Healthcare benefits

Vision

Dental

Domestic partners

Job summary

The Geisinger Research Data Core is seeking a Senior Data Engineer to design and maintain scalable data pipelines in Databricks and AWS, transforming clinical data for research use. You will model clinical data, modernize on-prem ETL to the cloud, and ensure data quality and performance across large research datasets.

You will collaborate with a multidisciplinary team, provide guidance on data architecture, and engage in the full software development life cycle to deliver robust analytical

Qualifications

• Databricks and cloud-based data engineering platforms experience required.
• Strong SQL and Python/PySpark for data pipelines.
• Experience with Apache Spark or similar distributed processing.
• Experience designing data models and data warehouse schemas for large datasets.
• ETL processes and data pipelines development/modernization.
• Relational databases and SQL Server experience.
• AWS cloud services familiarity for data architectures.
• Data quality, validation, monitoring, logging, troubleshooting practices.
• Understanding of pipeline performance optimization for scalable processing.
• Excellent communication to relay progress and issues to teams.
• Ability to collaborate with multidisciplinary stakeholders.

Responsibilities

• Designs, develops, and maintains PySpark and SQL data pipelines for transforming clinical data for research use.
• Develops and maintains data models for clinical and research data within Databricks.
• Reviews and optimizes data ingestion and processing pipelines with modern tooling.
• Defines and optimizes data warehouse schemas for efficient querying and analytics.
• Ensures scalability and reliability of pipelines and architecture for large datasets.
• Implements data quality checks and validation across pipelines.
• Troubleshoots Spark/SQL workloads and optimizes Databricks compute resources.
• Modernizes on-prem ETL processes to cloud-based Databricks equivalents.
• Provides Databricks support and resource optimization guidance to users.

Skills

Databricks experience

SQL

Python/PySpark

Apache Spark

Data modeling

ETL pipelines

Data quality & monitoring

Communication

Team collaboration

Mentoring

Education

High School Diploma or GED

Bachelor’s Degree in Technology Related Field

Tools

Databricks

AWS

SQL Server

PySpark

Job description

Location:

Work from home (Pennsylvania)

Shift:

Days (United States of America)

Scheduled Weekly Hours:

40

Worker Type:

Regular

Exemption Status:

Yes

Job Summary:

The Geisinger Research Data Core is responsible for transforming and managing clinical and genomic data for research use. As part of a collaborative and multidisciplinary team, this position develops and maintains data systems and pipelines that support research applications and the use of large-scale clinical and genomic datasets. The role provides key insight into data architecture, data modeling, and application development and engages in all aspects of the development life cycle as needed for each project to be successful.

We are seeking a skilled Senior Data Engineer with experience in cloud-based data platforms, particularly the Databricks platform and AWS. The ideal candidate will be responsible for modeling clinical data within the Databricks environment, developing and maintaining scalable data pipelines, and modernizing legacy ETL processes for cloud environments. The position will also support data quality, performance optimization, monitoring, and general Databricks computing needs for research and analytical applications.

Job Duties:

• Designs, develops, and maintains PySpark and SQL data pipelines for transforming clinical data for research use.

• Develops and maintains data models for clinical and other research data within the Databricks environment.

• Reviews, analyzes, modifies, creates, debugs, designs, and tests data ingestion and processing pipelines using modern development methodologies and tools.

• Defines and optimizes data warehouse schemas and data models to support efficient querying, storage, and analytical workflows.

• Ensures scalability, reliability, and maintainability of data pipelines and data architecture supporting large clinical and research datasets.

• Implements and monitors data quality checks and validation processes to ensure accuracy, completeness, and integrity of data throughout the pipeline.

• Troubleshoots and optimizes Spark and SQL workloads, data pipelines, and Databricks compute resources to improve performance and reliability.

• Modernizes existing on-premises Microsoft SQL Server ETL processes by designing and implementing equivalent cloud-based data pipelines and workflows in Databricks.

• Provides general Databricks computing support, including assisting users with compute resource selection, troubleshooting, and optimization of Spark and SQL workloads.

• Participates in design and application architecture discussions and assists in the development and documentation of technical approaches and solutions for team stakeholders.

• Responsible for the process, life cycle and creation of the software to assist users in completing a computer task or program.

• Implements code and documents all system changes based on assignments and timelines.

• Develops and follows department coding and user interface guidelines.

• Provides code review, testing, debugging, technical documentation, general testing instructions, go-live planning, assistance in go-live moves, and post-live support.

• Communicates all progress, roadblocks, and issues to the team and management in a timely manner.

• Collaborates with team members to identify and resolve issues, improve system performance, and ensure code quality.

• Provides training and documentation to team members on new development assignments.

• Trains and mentors staff members.

• Assists in the development and documentation of a technical approach for a solution to the team stakeholders.

Position Details:

Requirements :

• Proven experience with Databricks and cloud-based data engineering platforms.
• Strong experience with SQL and Python/PySpark for developing and optimizing data processing pipelines.
• Experience with Apache Spark or other distributed data processing technologies.
• Experience designing and implementing data models and data warehouse schemas for large-scale datasets.
• Experience developing, maintaining, or modernizing ETL processes and data pipelines.
• Experience working with relational databases and SQL Server
• Experience migrating or modernizing on-premises ETL processes is preferred.
• Familiarity with AWS cloud services and cloud-based data architectures.
• Experience with data quality, validation, monitoring, logging, and troubleshooting practices.
• Understanding of data pipeline performance optimization and scalable data processing.
• Excellent communication skills to convey progress, roadblocks, technical issues, and solutions to team members and management.
• Ability to work effectively as part of a multidisciplinary team and collaborate with technical and research stakeholders.

Education:

High School Diploma or Equivalent (GED)- (Required), Bachelor’s Degree-Technology Related Degree (Preferred)

Experience:

Minimum of 10 years-Related work experience (Required)

Certification(s) and License(s):

Skills:

Critical Thinking, Interpersonal Communication, Leads Others

OUR PURPOSE & VALUES:

Everything we do is about caring for our patients, our members, our students, our Geisinger family and our communities.

• KINDNESS: We strive to treat everyone as we would hope to be treated ourselves.

• EXCELLENCE: We treasure colleagues who humbly strive for excellence.

• LEARNING: We share our knowledge with the best and brightest to better prepare the caregivers for tomorrow.

• INNOVATION : We constantly seek new and better ways to care for our patients, our members, our community, and the nation.

• SAFETY: We provide a safe environment for our patients and members and the Geisinger family.

We offer healthcare benefits for full time and part time positions from day one, including vision, dental and domestic partners. Perhaps just as important, we encourage an atmosphere of collaboration, cooperation and collegiality.

Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, pregnancy, genetic information, disability, status as a protected veteran, or any other protected category under applicable federal, state, and local laws.