Senior Data Platform Engineer: AI-Driven Pipelines

Vanna Health; United States; Remote; USD 180,000 - 220,000; Full time

Job summary

Vanna Health is seeking a Senior Data Engineer, Platform to own the data infrastructure that powers our health insights. You will decide where and how jobs run, manage orchestration and deployments, and ensure data moves securely and cost-effectively to support AI-enabled analytics.

You will collaborate with coaches, clinicians and researchers to ensure data quality and trust, while guiding the platform toward scalable, observable and compliant operation.

Qualifications

• Typically seven or more years building production software or data systems.
• Hands-on experience with distributed data processing engines (e.g. Spark/Databricks).
• Strong Python and SQL you can defend with reasoning and checks.
• Experience in major cloud (AWS) with automated testing, CI/CD and IaC.
• Led technical work, set direction, and mentored engineers.
• Experience using AI coding agents and validating outputs in real data.

Responsibilities

• Shape the data platform roadmap with data leadership and deliver the work you lead.
• Set technical direction for data engineering: standards, mentoring and unblock engineers.
• Own platform architecture on Databricks and AWS: ingestion, storage, compute, access control.
• Own connectivity to partners and systems via SFTP, S3, private networking and feeds.
• Decide where and how jobs run; optimize for speed, memory predictability and cost.
• Own orchestration and deployments; treat infrastructure and pipelines as code.
• Make the platform reliable and observable; detect and recover from failures.
• Make data usable by AI; govern metrics and ensure trusted data for AI agents.
• Represent data engineering with partners, vendors and colleagues.

Skills

Python

SQL

Distributed data processing

Databricks

AWS

Data platform design

Tools

Databricks

AWS

Spark

SQL

Job description

Vanna Health is seeking a Senior Data Engineer, Platform to own the data infrastructure that powers our health insights. You will decide where and how jobs run, manage orchestration and deployments, and ensure data moves securely and cost-effectively to support AI-enabled analytics.

You will collaborate with coaches, clinicians and researchers to ensure data quality and trust, while guiding the platform toward scalable, observable and compliant operation.