Senior Databricks Data Engineer - Healthcare EDI & Pipelines

Valiant Technologies LLC; New York (NY); On-site; USD 140,000 - 200,000; Full time

Job summary

Valiant Technologies LLC is seeking an experienced Senior Data Engineer to join our Enrolment as a Product Platform team. The role focuses on building a modern, scalable healthcare enrollment data platform using Databricks.

Key priorities include delivering X12 EDI 837/835 processing capabilities with PySpark, Python, and Databricks expertise. You will work with Delta Lake, Spark, notebooks, and pipelines to transform enrollment and claims data while ensuring HIPAA compliance.

Qualifications

• 6+ years of experience in Data Engineering.
• Hands-on Databricks experience with Delta Lake, Spark and notebooks.
• Strong Python and PySpark skills.
• Experience with X12 EDI standards 837/835/834.
• Healthcare data handling with HIPAA compliance knowledge.
• Familiarity with healthcare enrollment and eligibility data workflows.
• Ability to work independently in a fast-paced environment.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build and support X12 EDI processing capabilities, particularly 837, 835, and 834 transactions.
• Work with Delta Lake, Apache Spark, Databricks Notebooks, and Pipelines.
• Process and transform healthcare enrollment, eligibility, and claims-related data.
• Ensure data security and compliance with HIPAA guidelines, including PII and PHI handling.
• Troubleshoot, optimize, and maintain existing data pipelines.
• Work independently in a fast-paced program environment.

Skills

Databricks

Python

PySpark

HIPAA compliance

X12 EDI

Delta Lake

Spark

Tools

Delta Lake

Spark

Databricks Notebooks

Pipelines

Job description

Valiant Technologies LLC is seeking an experienced Senior Data Engineer to join our Enrolment as a Product Platform team. The role focuses on building a modern, scalable healthcare enrollment data platform using Databricks.

Key priorities include delivering X12 EDI 837/835 processing capabilities with PySpark, Python, and Databricks expertise. You will work with Delta Lake, Spark, notebooks, and pipelines to transform enrollment and claims data while ensuring HIPAA compliance.