Senior Databricks Engineer - RCM Data Pipelines Lead

Aarista Technologies; Houston (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Aarista Technologies seeks a Senior Databricks Engineer to design and optimize large-scale healthcare data pipelines on Azure Databricks. You will own end-to-end engineering across EDI processing, remittance workflows, and real-time delivery within a Medallion architecture.

Key duties include building Delta Lake tables, MongoDB syncs, and Scala-based orchestration while integrating with Azure services and Unity Catalog.

Qualifications

• 6–7 years of data engineering with 3+ years on Databricks/Spark
• Proficient in PySpark and Scala
• Deep Delta Lake expertise including medallion architecture and schema evolution
• Production experience with MongoDB and SQL Server via JDBC
• Familiarity with HIPAA EDI standards (X12 837/835/277/999) or quick learner
• DevOps experience with Azure Pipelines/CI/CD for Databricks
• Data quality validation and reconciliation experience

Responsibilities

• Design and maintain production data pipelines on Azure Databricks using PySpark/Scala
• Implement Delta Lake with ACID, time travel, partitioning, and Z-ordering
• Build MongoDB syncs for Gold layer to real-time collections
• Develop Scala-based reporting and workflow orchestration
• Integrate with Azure storage, Unity Catalog, AD, and CI/CD pipelines
• Manage multi-tenant data architectures with per-client configurations and secrets
• Build SQL Server integrations via JDBC for metadata and configs
• Implement data quality validation at each stage
• Write ad-hoc scripts for data correction and backfills
• Participate in code reviews and PR pipelines

Skills

Databricks/Spark

PySpark

Scala

Delta Lake

MongoDB

SQL Server

JDBC

Azure Pipelines

Data Quality

Unity Catalog

LLM Orchestration

Tools

Azure Databricks

Delta Lake

Unity Catalog

MongoDB

SQL Server

Job description

Aarista Technologies seeks a Senior Databricks Engineer to design and optimize large-scale healthcare data pipelines on Azure Databricks. You will own end-to-end engineering across EDI processing, remittance workflows, and real-time delivery within a Medallion architecture.

Key duties include building Delta Lake tables, MongoDB syncs, and Scala-based orchestration while integrating with Azure services and Unity Catalog.