Senior Data Platform Engineer - AI-Driven & Cloud-Native

Health Data Innovations, Inc.; United States; On-site; USD 180,000 - 220,000; Full time

Job summary

Health Data Innovations, Inc. seeks a Senior Data Engineer, Platform to own and evolve the data infrastructure that powers our health insights platform. You will design scalable compute, orchestration, and data movement across SFTP/S3, private networks, and partner feeds, while ensuring reliability and observability.

You will lead architectural decisions, mentor engineers, and collaborate with analytics teams to deliver timely, cost-efficient gold-tier data for AI and analytics use cases.

Qualifications

• Seven or more years building production software or data systems.
• Experience owning data infrastructure or pipelines end to end.
• Strong Python and SQL with defendable results.
• Experience designing systems on a major cloud, preferably AWS.
• Led technical work or teams.

Responsibilities

• Shape the data platform roadmap with leadership and oversee delivery.
• Set technical direction for data engineering and code reviews.
• Own the platform architecture on Databricks and AWS.
• Manage connectivity to partners and systems (SFTP, S3, private networking, feeds).
• Decide where and how jobs run to optimize cost and performance.
• Own orchestration and deploys with Terraform, GitLab CI/CD.
• Make platform reliable and observable, handle failures gracefully.
• Make data usable by AI with governance and Genie/MCP components.
• Validate arrivals against promises (row counts, date spans).
• Replace manual work with automation and AI agents.
• Participate in data operations rotation.
• Protect PHI by design with access controls.
• Support analytics engineering with a platform.
• Represent data engineering with partners and vendors.

Skills

Python

SQL

Spark/Databricks

Tools

Databricks

AWS

Terraform

Job description

Health Data Innovations, Inc. seeks a Senior Data Engineer, Platform to own and evolve the data infrastructure that powers our health insights platform. You will design scalable compute, orchestration, and data movement across SFTP/S3, private networks, and partner feeds, while ensuring reliability and observability.

You will lead architectural decisions, mentor engineers, and collaborate with analytics teams to deliver timely, cost-efficient gold-tier data for AI and analytics use cases.