Remote Data Engineer: PySpark & Databricks Pipelines

Sentara Hospitals; United States; Remote; USD 80.000 - 156.000; Vollzeit

Benefits dieser Stelle

Medical Insurance

Dental Insurance

Vision Insurance

401k with Employer Match

Paid Time Off

Education Assistance

Student Debt Pay Down

Zusammenfassung

Sentara Health is hiring a Data Engineer to support a modern data platform built on Databricks. This remote role focuses on scalable data pipelines, metadata-driven ingestion, and strong data quality and governance embedded in the platform.

You will develop pipelines with PySpark, onboard new datasets via a metadata framework, and ensure data quality through validation rules while supporting file-based ingestion tools like Fivetran.

Qualifikationen

• Experience with PySpark and Databricks in data pipelines.
• Experience building ETL/ELT pipelines and data integration.
• Understanding of Delta Lake concepts and schema evolution.
• Experience with data quality and governance concepts.
• Exposure to metadata-driven ingestion frameworks.

Aufgaben

• Develop and maintain data pipelines using PySpark and Databricks.
• Work within a metadata-driven ingestion framework to onboard new datasets.
• Implement data quality checks and validation rules within pipelines.
• Support ingestion from file-based sources and ingestion tools (e.g., Fivetran).
• Handle schema changes, incremental loads, and file processing patterns.
• Contribute to data governance practices including tagging, metadata, and lineage.
• Troubleshoot and resolve pipeline failures and performance issues.
• Collaborate with architects and stakeholders on data onboarding and requirements.
• Follow and contribute to coding standards, reusable components, and best practices.

Kenntnisse

PySpark

Databricks

SQL

ETL/ELT

Fivetran

Delta Lake

Git

Data governance

Metadata

DataHub

Ausbildung

Bachelor’s degree or equivalent experience

Tools

Git

DataHub

Fivetran

Azure

Jobbeschreibung

Sentara Health is hiring a Data Engineer to support a modern data platform built on Databricks. This remote role focuses on scalable data pipelines, metadata-driven ingestion, and strong data quality and governance embedded in the platform.

You will develop pipelines with PySpark, onboard new datasets via a metadata framework, and ensure data quality through validation rules while supporting file-based ingestion tools like Fivetran.