Data Engineer II - Data Architect

Premise Health; Brentwood (TN); Hybrid; USD 110,000 - 150,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Life and disability insurance

401(k) with company match

Paid holidays and vacation time

Job summary

Premise Health is seeking an experienced Data Engineer II - Data Architect in Brentwood, TN to design scalable data models and build pipelines with Azure Databricks, enabling Lakehouse architecture across the organization.

You will collaborate with data science teams, govern data quality and security, and contribute to CI/CD and monitoring efforts in a hybrid, full-time role.

Qualifications

• Bachelor’s or Master’s degree in CS, IT, or a related field.
• 5+ years of cloud data engineering experience.
• Strong Azure Databricks, Delta Lake, and CI/CD experience.
• Experience with EPIC/Clarity/Caboodle data sources.
• Experience with data governance and security.

Responsibilities

• Design and maintain data models for operational, analytical, reporting, and data science use cases.
• Develop data pipelines using Azure Databricks, Spark, and Python.
• Integrate data from EPIC, Clarity, and Caboodle to Azure.
• Ensure data quality, security, and governance.
• Develop and maintain Lakehouse architecture with Delta Lake.
• Set up monitoring for Databricks pipelines and dashboards.
• Collaborate with data science teams for workflow integration.
• Participate in code reviews and CI/CD improvements.
• Provide on-call/off-hours support for critical deployments.

Skills

Azure Databricks

Python

PySpark

Scala

Data modeling

CI/CD

Data governance

SQL

Leadership

Education

Bachelor's or Master's in CS/IT

Tools

Delta Lake

EPIC

Clarity

Caboodle

Unity Catalog

Azure Monitor

Databricks

Azure Data Factory (ADF)

Splunk

Datadog

Prometheus

Power BI

Qlik

Terraform

Kubernetes

Linux

Job description

Premise Health is Different on Purpose

Premise Health serves large organizations and their people with exceptional healthcare. The result? Better experiences, better health, and better value, all while helping organizations lower their healthcare costs.

Premise's mission is to help people get, stay, and be well. Come join us and see for yourself why amazing health starts with amazing healthcare. For more information, visit www.premisehealth.com.

The Data Engineer II will be responsible for building, and maintaining scalable data pipelines, managing our azure delta lake architecture, and supporting analytics initiatives across the organization. The ideal candidate will have a strong background in Azure Databricks, data engineering, and data architecture. The successful candidate will work closely with our platform architects and data science teams to ensure seamless integration of data pipelines and Lakehouse architecture. This role should ensure coding standard and coding migration to higher environment and CI/CD.

This is a Full Time, hybrid, Data Engineer II - Data Architect role located in Brentwood, TN.

What You'll Do

• Design and maintain conceptual, logical, and physical data models for operational, analytical, reporting, and data science use cases.
• Design, develop, and maintain data pipelines using Azure Databricks, Apache Spark, and Python.
• Collaborate with data engineering teams to integrate data from EPIC, Clarity, and Caboodle and other on-premise source systems to Azure.
• Ensure data quality, integrity, and security throughout the data pipeline.
• Automate unit test cases as part of Data Pipeline.
• Contribute to the development and maintenance of our Lakehouse architecture using Azure Databricks and Delta Lake.
• Ensure data governance and security in the Lakehouse environment.
• Able to execute POCs to evaluate new tool, concepts introduced by Azure/ Databricks.
• Troubleshoot data pipeline and Lakehouse architecture issues, identifying and resolving bottlenecks and errors.
• Optimize data pipeline and Lakehouse architecture performance, scalability, cost and reliability.
• Collaborate with data science teams to integrate data pipelines and Lakehouse architecture with data science workflows.
• Ensure seamless data exchange between data engineering and data science teams.
• Participate in peer review for coding standards.
• Subject matter expert for pySpark and Scala.
• Implement and maintain monitoring solutions for Databricks pipelines (Bronze, Silver, Gold layers) to ensure data quality, performance, and reliability.
• Develop dashboards and alerts for tracking DLT pipeline health, job execution status, and SLA adherence using Databricks metrics and Azure Monitor.
• Configure and manage logging frameworks across ADF pipelines, Databricks jobs, and Storage Account to enable end-to-end traceability.
• Set up proactive alerting mechanisms for failures, latency issues, and resource utilization using Azure Monitor, Log Analytics, and custom metrics.
• Ensure lineage and auditability by leveraging Unity Catalog for tracking data movement and access patterns across the Medallion architecture.
• Collaborate with Data Engineering teams to define observability standards and integrate them into CI/CD workflows.
• Perform root cause analysis (RCA) for pipeline failures and maintain operational documentation for recurring issues.
• Optimize cost and performance monitoring for Databricks clusters and ADF activities through usage analytics and reporting.
• Provide extended support during off-hours or weekends for critical deployments or troubleshooting production issues to ensure platform stability.
• Other duties as assigned.

What You'll Bring

• Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.
• Certifications such as AWS Certified Solutions Architect, Cisco CCNP, Microsoft Azure Solutions Architect, or ITIL preferred.
• Databricks and Azure certification preferred.
• 5+ years of experience in Cloud development practice.
• Demonstrated experience designing conceptual, logical, and physical data models.
• Demonstrated experience with dimensional modeling, including facts, dimensions, conformed dimensions, and slowly changing dimensions.
• Strong expertise in cloud platforms (AWS, Azure, GCP), virtualization, and networking.
• Preferred experience with infrastructure automation tools (Terraform, Ansible, Kubernetes).
• Proficiency in Linux and Windows server administration.
• Solid knowledge of cybersecurity best practices and disaster recovery planning.
• Strong leadership experience in managing technical teams and project management skills.
• Knowledge of log management and monitoring tools (Splunk, Datadog, Prometheus).
• Knowledge on Talend, Power BI and Qlik.
• Familiarity with containerization and microservices.

Work-life balance is at the foundation of how decisions are made and where Premise is headed. We can only help people get, stay, and be well if we do the same for ourselves. In addition to competitive pay, Premise offers full-time team members benefits including medical, dental, vision, life and disability insurance, a 401(k) program with company match, paid holidays and vacation time, a company-sponsored wellness program, EAP, access to virtual primary care and virtual behavioral health at no cost for team members and their dependents. Additional benefits can be viewed here: https://jobs.premisehealth.com/benefits.

Premise is an equal opportunity employer; we value inclusion and do not discriminate based on race, color, religion, creed, national origin or ancestry, ethnicity, sex (including pregnancy and related conditions), gender identity or expression, sexual orientation, age, physical or mental disability, genetic information, past, current or prospective service in the uniformed services, or any other characteristic protected under applicable federal, state, or local law.