Lead Data Engineer

Astrana Health

United States

Remote

USD 140,000 - 190,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Astrana Health is seeking a seasoned Data Engineer to lead the design and delivery of core platform components, including a Databricks lakehouse and a GraphQL API layer. You will architect scalable batch and incremental pipelines, ingest data from diverse sources, and maintain unified healthcare data models across membership, providers, claims, and clinical data.

You will optimize data access layers, implement governance with Unity Catalog, and mentor engineers while driving reliability,

Qualifications

  • 5+ years of data engineering experience building and supporting enterprise data platforms and pipelines.
  • 3+ years of hands-on experience with Databricks in production environments.
  • Strong proficiency in Python, PySpark, SQL, and Apache Spark.
  • Experience with Delta Lake and modern lakehouse architectures.
  • Hands-on experience with Azure data services and cloud-based data platforms.
  • Strong knowledge of data modeling, relational databases, and ETL/ELT best practices.

Responsibilities

  • Lead design, development, and delivery of core platform components (Databricks Lakehouse, GraphQL API layer, Lakebase-backed apps).
  • Build and maintain scalable batch and incremental data pipelines using Databricks, Spark, PySpark, SQL, and Delta Lake.
  • Integrate data from various sources into governed datasets and unified models across domains.
  • Develop ingestion/Transformation processes including CDC, schema evolution, and reconciliation.
  • Mentor engineers, conduct code/design reviews, and enforce engineering standards.
  • Ensure data governance, Unity Catalog, and access controls; implement data quality checks.

Skills

Python
PySpark
SQL
Apache Spark
Leadership
Cross-functional collaboration
Data modeling
Communication

Education

Bachelor's degree in Computer Science, Engineering, Data Science, or a related field

Tools

Databricks
Azure data services
PostgreSQL
GraphQL
Unity Catalog
Delta Lake

Job description

  • Lead the design, development, and delivery of core platform components, including the Constellation Lakehouse in Databricks, Connect API (GraphQL) layer, and Lakebase-backed applications
  • Build and maintain scalable batch and incremental data pipelines using Databricks, Apache Spark, PySpark, SQL, and Delta Lake
  • Integrate data from transactional databases, APIs, files, and external partners into governed datasets
  • Design and maintain unified healthcare data models spanning membership, providers, claims, encounters, clinical data, and other business domains
  • Develop ingestion and transformation processes including change data capture, schema evolution, deduplication, historical tracking, and reconciliation
  • Build and optimize data access layers and APIs with software engineering teams
  • Develop and support Lakebase-backed applications and PostgreSQL data structures
  • Maintain and optimize Databricks workspaces, compute resources, job orchestration, and environment configurations
  • Implement data governance and access controls through Unity Catalog
  • Establish automated data quality checks, reconciliation, freshness monitoring, and alerting
  • Apply DevOps practices including Git, automated testing, CI/CD, infrastructure as code, and repeatable deployments
  • Tune pipelines, Spark workloads, SQL queries, and compute configurations for performance, scalability, and cost efficiency
  • Investigate and resolve platform incidents and pipeline failures; perform root-cause analysis
  • Partner on backup, recovery, and business continuity procedures; maintain documentation and runbooks
  • Conduct code and design reviews, establish engineering standards, and mentor engineers
  • Bachelor’s degree in Computer Science, Engineering, Data Science, or a related field
  • 5+ years of data engineering experience building and supporting enterprise data platforms and pipelines
  • 3+ years of hands‑on experience with Databricks in production environments
  • Strong proficiency in Python, PySpark, SQL, and Apache Spark
  • Experience with Delta Lake and modern lakehouse architectures
  • Hands‑on experience with Azure data services and cloud‑based data platforms
  • Strong knowledge of data modeling, relational databases, and ETL/ELT best practices
  • Experience integrating enterprise databases and APIs into scalable data solutions
  • Experience with Git, CI/CD pipelines, and automated deployment practices
  • Proven ability to monitor, troubleshoot, and optimize data pipelines and platform performance
  • Demonstrated leadership in solution design, code reviews, mentoring, and cross‑functional collaboration
  • Strong problem‑solving and communication skills
  • Preferred: Experience administering and optimizing Databricks environments, including Unity Catalog
  • Preferred: Experience with PostgreSQL, GraphQL APIs, and operational applications leveraging lakehouse data
  • Preferred: Healthcare data experience, including claims, eligibility, provider, or clinical datasets
  • Preferred: Familiarity with HL7, FHIR, and X12
  • Preferred: Experience with Terraform or other infrastructure‑as‑code tools
  • Preferred: Experience with streaming data pipelines and event‑driven architectures
  • Preferred: Experience supporting AI, machine learning, or NLP data workloads
  • Preferred: Familiarity with regulated data environments and governance best practices
  • Preferred: Experience with cloud data platform migrations and modernization initiatives
  • Preferred: Azure, Databricks, or AWS certifications
  • Remote position based in the U.S.
  • Equal Employment Opportunity and Affi…

Disability accommodation support during application

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Databricks Engineer
Senior Databricks Engineer

Remote • San Francisco (CA)

On-site
USD 150,000 - 190,000
Senior Data Engineer - Databricks
Senior Data Engineer - Databricks

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 160,000
Databricks Data Architect
Databricks Data Architect

NewVision • San Francisco (CA)

Hybrid
USD 140,000 - 190,000
Hybrid work model
Databricks Data Architect
Databricks Data Architect

NewVision Software • United States

Remote
USD 120,000 - 180,000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

Fa Ewjt Saasfaprod1 • New York (NY)

On-site
USD 140,000 - 170,000
Data Engineer
Data Engineer

GSquared Group • Atlanta (GA)

On-site
USD 120,000 - 160,000
Databricks Developer
Databricks Developer

Vservice It Solutions • United States

Remote
USD 140,000 - 190,000
Data Engineer - Databricks
Data Engineer - Databricks

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 170,000
Senior Data Engineer
Senior Data Engineer

SiloSmashers, Inc. • Alexandria (VA)

On-site
USD 140,000 - 155,000
Senior Data Engineer
Senior Data Engineer

Tap Growth ai • Pittsburgh

On-site
USD 140,000 - 180,000