Senior Data Engineer

Careforth; United States; Remote; USD 112,000 - 170,000; Full time

Benefits offered by this job

Remote-first culture

Wellness program

Flexible schedules

Job summary

Careforth in the United States is seeking a Senior Data Engineer to architect and operate our cloud-native lakehouse data platform powering AI and analytics products for clinical and caregiver workflows.

You will own end-to-end data pipelines using Databricks, Spark, and AWS services, ensuring HIPAA‑compliant, scalable data infrastructure, while mentoring engineers and collaborating with data science teams.

Qualifications

• 7–10 years of professional data engineering experience.
• 3+ years with modern cloud-based data lake/lakehouse architectures.
• Deep expertise in Apache Spark and Databricks across Delta Lake, Unity Catalog, and Workflows.
• AWS data services familiarity (S3, Redshift, Athena, Glue, Lambda, Step Functions, Kinesis).
• Experience in healthcare or regulated data environments; HIPAA safeguards required.
• Experience building ML feature stores and model pipelines.

Responsibilities

• Architect scalable data pipelines using Databricks and AWS.
• Evolve lakehouse architecture and semantic data layer.
• Build ETL/ELT and streaming pipelines ingesting healthcare data.
• Collaborate with data scientists on feature stores and models.
• Develop data infra for LLM/NLP use cases and audits.
• Create APIs and integrations for dashboards and CRMs.
• Ensure HIPAA-compliant data handling and security.
• Lead code reviews, mentor juniors, explore new tools.
• Participate in design discussions across product lines.

Skills

Python

SQL

Spark

Databricks

Airflow

dbt

Kubernetes

AWS

HIPAA

Communication

Education

Bachelor's or Master's in Computer Science / Data Engineering

Tools

Delta Lake

Unity Catalog

Workflows

S3

Redshift

Athena

Glue

Lambda

Step Functions

Kinesis

Job description

About Us

A pioneer in the caregiving space, Careforth supports family caregivers across the United States to confidently care for their loved ones at home. Through a combination of in-person home visits, remote coaching and our proprietary digital collaboration app, we provide caregivers with support, guidance, confidence, and connection to resources they need. The Caregivers and families we support stay with Careforth for many years, building lasting relationships along the way. Join us today and live our values: lead with heart, cultivate trust, go beyond.

Position Summary

The Senior Data Engineer will be a cornerstone of Careforth’s data platform team, architecting and operating the cloud-native, lakehouse-based infrastructure that powers our full suite of AI and analytics products — from clinical and caregiver risk intelligence to LLM-enabled chatbots, real-time signal capture, and payer-facing outcome reporting.

The ideal candidate brings deep, hands‑on expertise with Databricks, Apache Spark, and AWS Cloud services, and can serve as both a technical leader and mentor. You will own end‑to‑end pipeline design from raw source ingestion through curated analytics‑ready datasets, ensuring data systems are reliable, scalable, and HIPAA‑compliant.

What You Will Do

• Architect and build robust, scalable, and secure data pipelines leveraging Databricks (Delta Lake, Spark, Unity Catalog, Workflows) and AWS (S3, Redshift, Athena, Glue, Lambda, Step Functions, Kinesis), supporting both batch and real‑time processing modes.
• Design and evolve the enterprise lakehouse architecture, including data modeling, Delta table optimization, data versioning, and a semantic data layer and metrics store serving BI tools and downstream ML feature engineering.
• Build and maintain ETL/ELT orchestration (Airflow, dbt) and stream processing pipelines (Kafka/Kinesis) ingesting claims, EHR/EMR, pharmacy, IoT, survey, CRM, and digital engagement event data.
• Partner with Data Scientists to design and maintain feature store architecture, real‑time member context assembly, and temporal feature computation services supporting risk and predictive models.
• Develop data infrastructure for LLM and NLP use cases: vector databases, knowledge base ingestion, transcript storage and indexing, secure audio storage, and LLM response logging and audit trails.
• Build APIs and integrations for downstream consumers: care team dashboards, lead scoring, pathway recommendation, queue prioritization, CRM connectors (Salesforce, HubSpot), and payer‑facing data exchange.
• Implement HIPAA‑compliant data handling across all pipelines including PHI detection, redaction, audit trails, row‑level security, SSO, and secrets management (AWS Secrets Manager, Vault).
• Build reporting infrastructure including a report template engine, multi‑format export (PDF, Excel, CSV), scheduled report generation, and delivery APIs for internal and payer‑facing reporting.
• Conduct pull request reviews, enforce engineering best practices, mentor junior engineers, and research and apply emerging data engineering tools and patterns.
• Participate in design discussions with technical leads across product lines; perform other duties and special projects as assigned.

What You Will Bring

Education

• Bachelor’s or Master’s Degree in Computer Science, Data Engineering, or a related field.

Experience

• 7–10 years of professional data engineering experience, including 3+ years with modern cloud‑based data lake or lakehouse architectures.
• Deep expertise in Apache Spark (PySpark, Scala, or Java) for large‑scale distributed data processing and strong hands‑on Databricks experience across Delta Lake, Unity Catalog, and Workflows.
• Hands‑on experience with AWS data services: S3, Redshift, Athena, Glue, Lambda, Step Functions, Kinesis, DynamoDB, EMR, Bedrock, Transcribe, and Comprehend Medical.
• Experience in healthcare or regulated data environments; familiarity with HIPAA technical safeguards required.
• Demonstrated experience building data infrastructure in support of ML feature stores and model serving pipelines.

Technical Skills

• Strong Python (preferred); Java or Scala also valued. Proficient SQL across relational, NoSQL, and lakehouse environments.
• ETL/ELT orchestration with Airflow, dbt, and AWS Step Functions; stream processing with Kafka/Kinesis.
• Experience with vector databases, embedding stores, document ingestion pipelines, and RAG pipeline data layers for LLM use cases.
• Container orchestration with Docker and Kubernetes/EKS; CI/CD for data pipelines; familiarity with data quality and data contract frameworks.
• BI tool integration, ad‑hoc query execution engine design, query result caching, and multi‑format report delivery pipelines.

Soft Skills

• Strong communicator able to translate data architecture decisions to clinical, operational, and executive stakeholders.
• Collaborative by nature; comfortable driving alignment across data science, product, engineering, and clinical teams.
• Detail‑oriented and committed to data quality, HIPAA compliance, and operational excellence; self‑directed with a continuous learning mindset.

You'll Benefit From

At Careforth your well‑being matters. With flexible schedules, a remote‑first culture, and a nationally recognized wellness program, our benefits are designed to help you thrive, both professionally and personally. Discover how we invest in you: https://careforth.com/careers/#benefits

The pay range for this position is $112000 - $170000. The actual wage offered may be lower or higher depending on budget and candidate experience, knowledge, skills, qualifications, and geographic location.

Join Our Award Winning Team

Founded in Boston, Careforth's caregiver programs and services improve health outcomes, keeping care at home longer. Additionally, our programs provide financial benefit to caregivers and cost savings to state agencies and health plans. At Careforth, we understand the challenges of caregiving and are committed to supporting family caregivers at every turn.

Caregivers play a critical role in the future of healthcare—and so can you.

Careforth is an Equal Opportunity Employer*

DISCLAIMER: Please note this job description is not designed to cover or contain a comprehensive listing of activities, duties or responsibilities that are required of the employee for this job.

*Careforth supports families with diverse backgrounds and as an equal opportunity employer, we seek employees who reflect the diverse population we serve. Careforth complies with all applicable laws concerning hiring and employment practices and is firmly committed to fostering and maintaining a workplace free from discrimination. We pledge to hire, train, and promote our employees without regard to race, religion, gender, gender identity, genetic information, age, national origin, sexual orientation, disability, veteran status, or any other category protected by applicable law.

Careforth strives to create experiences that are accessible and welcoming to everyone, including making www.careforth.com and the careers site accessible to any and all users. If you would like to contact us regarding the company’s diversity, equity and inclusion initiatives, inquire about a specific accessibility need or the accessibility of our website, or if you need assistance completing an application process, please contact People & Culture at 866-797-2333.