Senior Data Engineer - Hybrid, Delta Lake & Spark

Premise Health; Brentwood (TN); Hybrid; USD 110,000 - 140,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Life and disability insurance

401(k) with company match

Paid holidays & vacation

Wellness program

EAP

Virtual primary care & behavioral care

Job summary

Premise Health is hiring a Senior Data Engineering Analyst in Brentwood, TN, for a hybrid full-time role. You will own ingestion jobs and data pipelines within the bronze layer of the medallion architecture, using Spark, ADF/Airflow, and Delta Lake to ensure reliable, scalable data delivery to downstream consumers.

Requirements include a BS in a related field and 2–4 years of data engineering experience, plus skills in SQL, Python, Spark, and cloud platforms (Azure/AWS).

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or related field.
• 2-4 years of experience in data engineering or related field.
• Experience with orchestration tools (Airflow, ADF, or similar).
• Experience with Databricks and cloud data platforms (Azure, AWS).
• Experience creating runbooks and operational documentation.
• Healthcare data experience preferred.
• Strong proficiency in SQL, Python, and Spark.
• Familiarity with Delta Lake and lakehouse patterns.
• Exposure to streaming or CDC patterns.
• Understanding of data pipeline reliability and monitoring concepts.

Responsibilities

• Pipeline Development & Ownership.
• Own ingestion jobs and data pipelines for assigned domains.
• Work with advanced Spark, ADF/Airflow, and orchestration tools.
• Implement Delta Lake and lakehouse patterns for data storage and processing.
• Ensure data pipelines are reliable, performant, and scalable.
• Documentation & Operations.
• Create and maintain runbooks and SLAs for pipeline operations.
• Manage domain requirements and communicate data availability timelines.
• Document architecture notes and technical decisions.
• Support pipeline monitoring, alerting, and incident response.
• Collaboration & Communication.
• Manage domain requirements with business and technical stakeholders.
• Collaborate with Analytics Engineering and Data Governance to ensure data quality and adherence to standards.
• Work with senior team members to prioritize work and resolve technical challenges.
• Receive and respond to feedback in a professional manner.

Skills

SQL

Python

Spark

Education

Bachelor's degree in Computer Science, Information Systems, Engineering, or related field

Tools

Airflow

ADF

Databricks

Azure

AWS

Delta Lake

Lakehouse

Job description

Premise Health is hiring a Senior Data Engineering Analyst in Brentwood, TN, for a hybrid full-time role. You will own ingestion jobs and data pipelines within the bronze layer of the medallion architecture, using Spark, ADF/Airflow, and Delta Lake to ensure reliable, scalable data delivery to downstream consumers.

Requirements include a BS in a related field and 2–4 years of data engineering experience, plus skills in SQL, Python, Spark, and cloud platforms (Azure/AWS).