Senior Data Engineer - Healthcare EDI Platform (Databricks)

Quantum Technologies. LLC; New York (NY); On-site; USD 120,000 - 170,000; Full time

Job summary

Quantum Technologies LLC invites an experienced Senior Data Engineer to join our Enrolment as a Product Platform team in New York. You will build a modern, scalable healthcare enrollment data platform using Databricks and PySpark.

The role focuses on delivering X12 EDI 837/835 processing capabilities, working with Delta Lake and Apache Spark to handle high-volume enrollment, eligibility, and claims data while ensuring HIPAA compliance and data security.

Qualifications

• 6+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks, Delta Lake, Spark, and Notebooks.
• Proficiency with X12 EDI standards – 837, 835, and 834.
• Python and PySpark experience required.
• Experience building and maintaining high-volume data pipelines.
• Healthcare domain experience with HIPAA, PII, PHI compliance.
• Familiarity with healthcare enrollment and eligibility data workflows.
• Ability to work independently in a fast-paced environment.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build and support X12 EDI processing capabilities, particularly 837, 835, and 834 transactions.
• Work with Delta Lake, Apache Spark, Databricks Notebooks, and Pipelines.
• Process and transform healthcare enrollment, eligibility, and claims-related data.
• Ensure data security and compliance with HIPAA guidelines, including PII and PHI handling.
• Troubleshoot, optimize, and maintain existing data pipelines.
• Work independently in a fast-paced program environment.

Skills

Databricks

Delta Lake

Apache Spark

Notebooks

Pipelines

X12 EDI 837/835/834

Python

PySpark

HIPAA

PII/PHI handling

Healthcare domain

Tools

Databricks Notebooks

Job description

Quantum Technologies LLC invites an experienced Senior Data Engineer to join our Enrolment as a Product Platform team in New York. You will build a modern, scalable healthcare enrollment data platform using Databricks and PySpark.

The role focuses on delivering X12 EDI 837/835 processing capabilities, working with Delta Lake and Apache Spark to handle high-volume enrollment, eligibility, and claims data while ensuring HIPAA compliance and data security.