Senior Data Platform Engineer – Databricks, AI/RAG (Cloud-Native)

Halo Media; United States; Remote; USD 120,000 - 190,000; Full time

Job summary

Halo Media is seeking an experienced Data Engineer to lead a modern data platform for healthcare applications. You will define the data architecture, set engineering standards, and provide technical leadership to ensure scalable, reliable data solutions.

You will partner with product stakeholders and client teams to translate complex business needs into robust data architectures. You will design scalable cloud-native architectures, lead pipelines with Databricks, Spark, Python and SQL, and

Qualifications

• 7+ years in Data Engineering or related roles.
• Expert-level Python and SQL with Spark experience.
• Databricks proficiency and data architecture experience.
• Experience with Airflow and scalable data platforms.
• Strong stakeholder communication and mentoring abilities.

Responsibilities

• Design and evolve scalable, cloud-native data architectures.
• Define data models, integration patterns, and standards.
• Lead design of data pipelines using Databricks, Spark, Python, SQL.
• Establish data quality, governance, observability, and performance.
• Mentor engineers and guide architecture reviews.
• Translate business needs into technical solutions with stakeholders.
• Ingest, transform, and orchestrate large-scale datasets.
• Contribute to AI-ready data platforms supporting RAG & Generative AI.

Skills

Databricks

Python

SQL

Apache Spark

Airflow

Data Modeling

ETL/ELT

Data Platform Architecture

Leadership

Client Communication

Mentoring

Cloud Architecture

RAG/Generative AI

LangChain

LlamaIndex

Vector Databases

Tools

Docker

Kubernetes

Pinecone

Weaviate

Chroma

Qdrant

Milvus

Job description

Halo Media is seeking an experienced Data Engineer to lead a modern data platform for healthcare applications. You will define the data architecture, set engineering standards, and provide technical leadership to ensure scalable, reliable data solutions.

You will partner with product stakeholders and client teams to translate complex business needs into robust data architectures. You will design scalable cloud-native architectures, lead pipelines with Databricks, Spark, Python and SQL, and