Senior Data Platform Engineer - Databricks & Healthcare AI

CodaMetrix; Boston; Hybrid; PHP 10,822,511 - 12,368,584; Full time

Benefits offered by this job

Health Insurance

401(k) plan

Paid Time Off

Performance evaluations

Company awards

Office in Boston (Back Bay)

Employer-paid life insurance

Job summary

CodaMetrix is seeking a Principal Data Engineer to lead the Data Platform team. You will own the architecture of a Databricks-based data platform serving 30+ healthcare customers, guiding streaming and batch workloads and data governance across teams.

You will drive cost optimization, CI/CD pipelines, and scalable data solutions while mentoring engineers and collaborating with ML, Analytics, DevOps, and Product stakeholders.

Qualifications

• Bachelor's, Master's, or Ph.D. in CS or related field or equivalent experience.
• 8+ years data engineering with progressive responsibility.
• 5+ years hands-on Databricks platform experience (Unity Catalog, Delta Lake, Structured Streaming, Spark SQL, cluster management).
• Expert PySpark and Python; working knowledge of Scala.
• Expert Terraform for infrastructure-as-code (state management, modular structures, multi-environment deployments).
• Expert Apache Kafka / AWS MSK for streaming ingestion and security.
• Deep understanding of medallion/lakehouse architecture patterns (bronze/silver/gold, SCD2, MV, SCD).
• Proven CI/CD pipelines for data platform deployments (Jenkins, GitHub Actions).
• Expert SQL with complex queries on petabyte-scale data.
• Strong AWS experience (S3, IAM, Secrets Manager, VPC).
• HIPAA-regulated environment experience with PHI handling requirements.
• Disaster recovery architecture and cross-region replication.
• Experience with AI tools and agentic workflows to augment design, development, and testing.
• Nice to have healthcare data experience (EHR, ICD-10, HL7/FHIR) and build-vs-buy capability.

Responsibilities

• Own the technical strategy and roadmap for the CodaMetrix data platform, ensuring scalability and regulatory alignment.
• Design, build, and maintain scalable streaming and batch platforms using Databricks and Terraform-based infrastructure.
• Implement data governance, least-privilege access, and HIPAA/SOC 2 compliance across environments.
• Drive cost optimization and operational excellence with monitoring, budgets, and runbooks for platform SLAs.
• Mentor engineers, enable ML/BI/Analytics teams, and establish organization-wide standards.

Skills

PySpark

Python

Scala

Databricks platform

Apache Kafka / AWS MSK

SQL

AWS

HIPAA compliance

CI/CD pipelines

Disaster recovery

Data governance

AI tools and agentic workflows

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Terraform

Jenkins

GitHub Actions

Unity Catalog

Delta Lake

Spark SQL

AWS services

Job description

CodaMetrix is seeking a Principal Data Engineer to lead the Data Platform team. You will own the architecture of a Databricks-based data platform serving 30+ healthcare customers, guiding streaming and batch workloads and data governance across teams.

You will drive cost optimization, CI/CD pipelines, and scalable data solutions while mentoring engineers and collaborating with ML, Analytics, DevOps, and Product stakeholders.