Sr. Data Engineer (Healthcare, Spark, Scala) - Remote

ConsultNet Technology Services and Solutions; United States; Remote; USD 100,000 - 150,000; Full time

Job summary

A leading technology services company seeks a Sr. Data Engineer specialized in healthcare to design and execute advanced data processing pipelines. This 100% remote role requires expertise in Apache Spark, Scala, and cloud environments, focusing on improving data integrity and auditability. Candidates should possess strong analytical skills, communicate effectively, and manage multiple tasks under pressure. Join a dedicated team striving to enhance healthcare through technology.

Qualifications

• 5+ years of experience with big data technologies including Spark, Scala, and SQL.
• Deep experience with Scala for Spark applications and cloud expertise.
• Experience in healthcare data preferred.

Responsibilities

• Design and execute Spark jobs for data ingestion, transformation, and machine learning.
• Create and optimize SQL scripts for ETL operations.
• Collaborate with teams to troubleshoot and manage data pipelines.

Skills

Data Analysis

Apache Spark

Scala

SQL

Cloud Computing

Healthcare Domain Expertise

Education

Bachelor's degree in Computer Science or Information Technology

Tools

Apache Spark

AWS

Azure

GCP

Databricks

Snowflake

Job description

Title: Sr. Data Engineer, Healthcare
Location : 100% Remote
Target Start Date : ASAP
Type: contract or contract to hire
Pay Rate: DOE

Are you driven by working to improve our healthcare system through advanced technology? We need your expertise and integrity!

This role is 100% remote and we are looking for candidates who have healthcare domain expertise.

Responsibilities

• Design, maintain, and execute intermediate-to-advanced Apache Spark jobs (using RDDs, DataFrames, Datasets, and Structured Streaming) for data ingestion, transformation, validation, integration, and machine learning pipelines
• Develop, optimize, and refactor Scala-based Spark applications, including writing unit tests and conducting peer code reviews with a focus on performance and production-readiness
• Create, maintain, and enhance basic-to-intermediate SQL scripts for orchestration, data validation, and ETL operations. Tune SQL for efficient daily execution.
• Integrate LangChain into data workflows to support retrieval-augmented generation (RAG), prompt templates, and conversational or embedding-based data queries
• Use LangChain indexes and chains to interface with data stores like RDBMS, DataFrames, or CSV—enabling automated document analysis, summarization, and structured prompt pipelines
• Collaborate closely with engineering, DBAs, cloud ops, and data science teams to troubleshoot Spark cluster (e.g., Databricks / EMR), environmental, or network issues; support may include after‐hours or weekend duty.
• Construct and sustain scalable data pipelines in cloud ecosystems (AWS, Azure, GCP, OCI), encompassing S3, Glue/EMR, Databricks, Kafka/Spark Streaming.
• Validate pipeline outputs to meet business and compliance requirements, ensuring data integrity and auditability.
• Follow all industry and organizational compliance guidelines.
• Track productivity, complete tasks via JIRA or similar ticketing systems, and deliver against performance goals.
• Support special projects (e.g., migration to Databricks, Snowflake integration, RAG workflows), and adapt responsibly to organizational change.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or equivalent experience.
• 5+ years experience with big data technologies (Spark, Scala, Hadoop, Kafka, Ray, etc.).
• Deep experience with Scala in Spark—writing idiomatic functional pipelines, optimizing Spark apps, leveraging DataFrame / Dataset APIs
• Intermediate to advanced Spark SQL, including query optimization and streaming design.
• LangChain proficiency: implementation of chains, prompt templates, retrieval workflows, and integration with LLMs/data systems .
• Cloud expertise (AWS/Azure/GCP/OCI), including Databricks, EMR, S3, Glue, Lambdas, Snowflake.
• Solid RDBMS skills (Oracle, MS SQL, Vertica), with strong SQL/PL-SQL or ETL experience for data integration.
• Prior experience in healthcare data (enrollment, medical/pharmacy claims) is preferred.
• Proficiency in Microsoft Office and project tools like JIRA.
• Familiarity with Databricks and/or Snowflake ecosystems.

Additional Requirements

• Highly analytical with a strong attention to detail.
• Exceptional verbal, written, and listening communication skills.
• Capable of independently managing multiple concurrent tasks and prioritizing under pressure.
• Comfortable working in both independent and team-based environments.

Welcome to ConsultNet and the family of companies, Tekne, SaltClick, TechBridge, and OmniMedia. As a premier national provider of technology talent and solutions, our expertise spans across project services, contract-to-hire, direct placement, and managed services both onshore and nearshore.

Celebrating more than 25 years of partnership with a diverse client base, we've crafted rewarding opportunities for our consultants, fostering high-performing teams that deliver impactful results.

Over the last few years thousands of consultants have found their calling with us in roles that have made a meaningful impact on their lives, enhanced their career, challenged them, and propelled them towards achieving their personal and professional goals. At the ConsultNet family of companies, we believe effective communication is crucial in aligning the right job with your unique skills and professional aspirations. To us, it's all about the personal approach we take and the values we uphold.

Our comprehensive service offerings cover a wide range of technology positions across key markets nationwide. Client more at www.consultnet.com .

We champion equality and inclusivity, proudly supporting an Equal Opportunity Employer policy. We welcome applicants regardless of Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other status protected by law.