Senior Solutions Architect – Healthcare Data Migration

Jobtailor; Mesa (AZ); On-site; USD 180,000 - 240,000; Full time

Job summary

Jobtailor is seeking a senior data engineering/architecture leader to drive cloud migration and Databricks-based workloads in healthcare. You will decommission legacy Teradata systems, architect scalable Databricks environments with Terraform, and enforce HIPAA/PHI-compliant data security using Unity Catalog.

You will mentor teams and lead PoCs to demonstrate the vision. The role requires deep Spark/DBSQL tuning expertise, PySpark, Delta Lake, and strong problem-solving skills in regulated

Qualifications

• 15+ years of dedicated experience in data engineering, enterprise architecture, and major cloud migrations.
• Deep, hands-on history migrating legacy Teradata estates, BTEQ scripts, and enterprise warehouses.
• 4+ years of hands-on expertise in Databricks including job optimization, debugging, and workload governance.
• Deep understanding of Spark Internals, PySpark, Databricks SQL (DBSQL) performance tuning, and Delta Lake.
• Proven experience using Terraform to provision and manage cloud infrastructure in highly regulated environments.
• Solid understanding of patient data complexities, data models, and strict compliance (HIPAA/PHI) guardrails.
• Proven ability to identify and solve complex technical problems with minimal oversight.

Responsibilities

• Lead engagements from discovery to delivery with full ownership and accountability for enterprise healthcare accounts.
• Architect and execute the decommissioning of legacy Teradata environments, shifting workloads into highly scalable Databricks structures.
• Design and build automated, reproducible cloud environments using Terraform to manage workspaces and networking.
• Apply expert-level Spark fundamentals and Spark internals knowledge to troubleshoot, tune, and optimize large-scale distributed workloads and SQL jobs.
• Implement strict data security models complying with HIPAA and PHI regulations using Unity Catalog for fine‑grained access control.
• Design and execute PoCs and architecture workshops to demonstrate solution vision and inspire customer confidence.
• Champion best practices in PySpark, Delta Lake, and Lakehouse design while mentoring engineering teams.

Skills

Data Engineering

Enterprise Architecture

Spark Internals

PySpark

Delta Lake

SQL Performance Tuning

Workload Governance

Tools

Databricks

Terraform

Unity Catalog

BTEQ Scripts

Job description

Responsibilities

• Lead engagements from discovery to delivery with full ownership and accountability for enterprise healthcare accounts.
• Architect and execute the decommissioning of legacy Teradata environments, shifting workloads into highly scalable Databricks structures.
• Design and build automated, reproducible cloud environments using Terraform to manage workspaces and networking.
• Apply expert-level Spark fundamentals and Spark internals knowledge to troubleshoot, tune, and optimize large-scale distributed workloads and SQL jobs.
• Implement strict data security models complying with HIPAA and PHI regulations using Unity Catalog for fine‑grained access control.
• Design and execute PoCs and architecture workshops to demonstrate solution vision and inspire customer confidence.
• Champion best practices in PySpark, Delta Lake, and Lakehouse design while mentoring engineering teams.

Requirements

• 15+ years of dedicated experience in data engineering, enterprise architecture, and major cloud migrations.
• Deep, hands‑on history migrating legacy Teradata estates, BTEQ scripts, and enterprise warehouses.
• 4+ years of hands‑on expertise in Databricks including job optimization, debugging, and workload governance.
• Deep understanding of Spark Internals, PySpark, Databricks SQL (DBSQL) performance tuning, and Delta Lake.
• Proven experience using Terraform to provision and manage cloud infrastructure in highly regulated environments.
• Solid understanding of patient data complexities, data models, and strict compliance (HIPAA/PHI) guardrails.
• Proven ability to identify and solve complex technical problems with minimal oversight.

Core Competencies

Demonstrates extensive expertise in data engineering and enterprise architecture, with a strong focus on migrating legacy systems to cloud environments while ensuring compliance with HIPAA and PHI regulations. Proficient in architecting scalable solutions using Databricks, Terraform, and Spark technologies.

Hard Skills

• Data Engineering
• Enterprise Architecture
• Databricks
• Spark Internals
• PySpark
• Delta Lake
• Terraform
• SQL Performance Tuning
• Workload Governance
• BTEQ Scripts

Soft Skills

• Problem Solving
• Mentoring
• Leadership

Tools & Technologies

• Databricks
• Terraform
• Unity Catalog