Remote Data Engineer - Databricks & PySpark Pipelines

Sentara Health; Virginia Beach (VA); Remote; USD 80.000 - 134.000; Vollzeit

Benefits dieser Stelle

Medical, Dental, Vision plans

401k/403B with employer match

Paid Time Off

Zusammenfassung

Sentara Health is seeking a Data Engineer to support a modern data platform built on Databricks. The role focuses on building scalable data pipelines, metadata-driven ingestion, and embedding data quality and governance into the platform.

Requirements include 3+ years of relevant experience, strong PySpark/Databricks skills, and familiarity with Delta Lake and Azure. Remote work options are available with eligibility across multiple states.

Qualifikationen

• Experience in building scalable data pipelines with Databricks and PySpark.
• Knowledge of Delta Lake concepts such as merge, schema evolution, and partitions.
• Familiarity with metadata-driven ingestion frameworks and data governance practices.

Aufgaben

• Develop and maintain data pipelines using PySpark and Databricks
• Work within a metadata-driven ingestion framework to onboard new datasets
• Implement data quality checks and validation rules within pipelines
• Support ingestion from file-based sources and ingestion tools (e.g., Fivetran)
• Handle schema changes, incremental loads, and file processing patterns
• Contribute to data governance practices including tagging, metadata, and lineage
• Troubleshoot and resolve pipeline failures and performance issues
• Collaborate with architects and stakeholders on data onboarding and requirements
• Follow and contribute to coding standards, reusable components, and best practices

Kenntnisse

PySpark

Databricks

SQL

ETL/ELT

Delta Lake

Azure

Git

Data governance

Fivetran

Metadata-driven

Ausbildung

Experience in lieu of a Bachelor's Degree

3+ years with degree

5+ years without degree

Tools

Databricks Platform

Fivetran

Git

Delta Lake

DataHub

Azure Cloud

Jobbeschreibung

Sentara Health is seeking a Data Engineer to support a modern data platform built on Databricks. The role focuses on building scalable data pipelines, metadata-driven ingestion, and embedding data quality and governance into the platform.

Requirements include 3+ years of relevant experience, strong PySpark/Databricks skills, and familiarity with Delta Lake and Azure. Remote work options are available with eligibility across multiple states.