Data Engineer (Databricks | AI/RAG)

Halo Media; United States; Remote; USD 120.000 - 190.000; Vollzeit

Zusammenfassung

Halo Media is seeking an experienced Data Engineer to lead a modern data platform for healthcare applications. You will define the data architecture, set engineering standards, and provide technical leadership to ensure scalable, reliable data solutions.

You will partner with product stakeholders and client teams to translate complex business needs into robust data architectures. You will design scalable cloud-native architectures, lead pipelines with Databricks, Spark, Python and SQL, and

Qualifikationen

• 7+ years in Data Engineering or related roles.
• Expert-level Python and SQL with Spark experience.
• Databricks proficiency and data architecture experience.
• Experience with Airflow and scalable data platforms.
• Strong stakeholder communication and mentoring abilities.

Aufgaben

• Design and evolve scalable, cloud-native data architectures.
• Define data models, integration patterns, and standards.
• Lead design of data pipelines using Databricks, Spark, Python, SQL.
• Establish data quality, governance, observability, and performance.
• Mentor engineers and guide architecture reviews.
• Translate business needs into technical solutions with stakeholders.
• Ingest, transform, and orchestrate large-scale datasets.
• Contribute to AI-ready data platforms supporting RAG & Generative AI.

Kenntnisse

Databricks

Python

SQL

Apache Spark

Airflow

Data Modeling

ETL/ELT

Data Platform Architecture

Leadership

Client Communication

Mentoring

Cloud Architecture

RAG/Generative AI

LangChain

LlamaIndex

Vector Databases

Tools

Docker

Kubernetes

Pinecone

Weaviate

Chroma

Qdrant

Milvus

Jobbeschreibung

About the Role

We are looking for an experienced Data Engineer to lead the design and evolution of a modern data platform supporting critical business applications within the healthcare industry. This is a highly strategic role where you will define the data architecture, establish engineering best practices, and provide technical leadership to ensure scalable, reliable, and high-quality data solutions. You will work closely with engineering teams, product stakeholders, and client leadership to transform complex business requirements into robust data architectures. Beyond designing data pipelines, you will serve as the technical authority for data engineering decisions, helping shape the long-term architecture while mentoring engineers and driving engineering excellence.

What You'll Do

Design and evolve scalable, cloud-native data architectures.

Define data models, integration patterns, and architecture standards across multiple data sources.

Lead the design of modern data pipelines using Databricks, Spark, Python, and SQL.

Establish best practices for data quality, governance, scalability, performance, and observability.

Guide Data Engineers through technical leadership, architecture reviews, and mentoring.

Partner with product managers, business stakeholders, and client teams to understand business needs and translate them into technical solutions.

Design efficient ingestion, transformation, and orchestration strategies for large-scale datasets.

Collaborate on AI-ready data platforms supporting Retrieval-Augmented Generation (RAG) and Generative AI initiatives.

Drive technical decisions that improve reliability, maintainability, and long-term scalability of the platform.

Required Qualifications

7+ years of experience in Data Engineering, Data Architecture, or related roles.

Strong hands-on experience with Databricks.

Expert-level Python and SQL.

Strong experience with Apache Spark.

Experience with workflow orchestration tools such as Airflow.

Strong understanding of data modeling, ETL/ELT design, and distributed data processing.

Experience designing highly scalable and reliable data platforms.

Strong communication skills with experience working directly with clients and business stakeholders.

Experience leading technical initiatives and mentoring engineering teams.

Advanced English Level

Preferred Qualifications

Experience with AI/ML data platforms.

Familiarity with Retrieval-Augmented Generation (RAG) architectures.

Experience working with Vector Databases (Pinecone, Weaviate, Chroma, Qdrant, Milvus, or similar).

Knowledge of LangChain, LlamaIndex, or similar frameworks.

Experience with Docker and Kubernetes.

Experience implementing CI/CD and Infrastructure as Code.

Background working in consulting or client-facing environments.

What We're Looking For

We're looking for someone who enjoys solving complex architectural challenges, influencing technical direction, and helping engineering teams succeed. The ideal candidate combines deep technical expertise with strong communication and leadership skills, and is comfortable collaborating with both engineers and business stakeholders to build scalable, future-ready data platforms.