Senior Databricks Engineer - Revenue Cycle Management (RCM)

Aarista Technologies; Houston (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Aarista Technologies seeks a Senior Databricks Engineer to design and optimize large-scale healthcare data pipelines on Azure Databricks. You will own end-to-end engineering across EDI processing, remittance workflows, and real-time delivery within a Medallion architecture.

Key duties include building Delta Lake tables, MongoDB syncs, and Scala-based orchestration while integrating with Azure services and Unity Catalog.

Qualifications

• 6–7 years of data engineering with 3+ years on Databricks/Spark
• Proficient in PySpark and Scala
• Deep Delta Lake expertise including medallion architecture and schema evolution
• Production experience with MongoDB and SQL Server via JDBC
• Familiarity with HIPAA EDI standards (X12 837/835/277/999) or quick learner
• DevOps experience with Azure Pipelines/CI/CD for Databricks
• Data quality validation and reconciliation experience

Responsibilities

• Design and maintain production data pipelines on Azure Databricks using PySpark/Scala
• Implement Delta Lake with ACID, time travel, partitioning, and Z-ordering
• Build MongoDB syncs for Gold layer to real-time collections
• Develop Scala-based reporting and workflow orchestration
• Integrate with Azure storage, Unity Catalog, AD, and CI/CD pipelines
• Manage multi-tenant data architectures with per-client configurations and secrets
• Build SQL Server integrations via JDBC for metadata and configs
• Implement data quality validation at each stage
• Write ad-hoc scripts for data correction and backfills
• Participate in code reviews and PR pipelines

Skills

Databricks/Spark

PySpark

Scala

Delta Lake

MongoDB

SQL Server

JDBC

Azure Pipelines

Data Quality

Unity Catalog

LLM Orchestration

Tools

Azure Databricks

Delta Lake

Unity Catalog

MongoDB

SQL Server

Job description

We are seeking a Senior Databricks Engineer to design, build, and optimize large-scale data pipelines powering our healthcare Revenue Cycle Management platform. You will own end-to-end data engineering across EDI claim processing (837, 835, 277, 999), remittance workflows, AI-driven medical coding, and real-time operational data delivery using the Medallion architecture on Azure Databricks.

Key Responsibilities:

• Design & maintain production data pipelines using PySpark and Scala on Azure Databricks, processing healthcare EDI transactions (837 Claims, 835 Remittances, 277/999 Acknowledgments)
• Implement and optimize Delta Lake tables with ACID transactions, Z-ordering, partitioning strategies, and data versioning (time travel) for petabyte-scale healthcare data
• Build MongoDB sync pipelines to deliver Gold-layer data into operational MongoDB collections for real-time application access
• Develop/Maintain Scala-based reporting and workflow orchestration engines, including dynamic report generation, AP analysis, and notification triggers
• Integrate with Azure ecosystem: Azure Blob Storage, Azure Data Lake, Azure AD, Azure DevOps CI/CD pipelines, and Databricks Unity Catalog
• Manage multi-tenant data architectures with per-client configurations, secrets management (Databricks secrets scopes), and feature flag-driven rollouts
• Build and maintain SQL Server integrations via JDBC for workflow metadata, feature flags, report configurations, and state management
• Implement data quality frameworks with validation layers at each pipeline stage (patient info, diagnosis codes, CPT codes, facility mappings, rendering providers)
• Write ad-hoc data correction, migration, and backfill scripts for production data reconciliation
• Participate in code reviews, PR validation pipelines, and engineering best practices (commit/lint, conventional commits)

Required Skills & Qualifications:

• Core: 6–7 years of data engineering experience with 3+ years on Databricks/Spark
• Languages: Proficient in PySpark (primary) and Scala/Spark; Python 3.11+
• Data Lake: Deep expertise in Delta Lake — medallion architecture, ACID transactions, schema evolution, Z-ordering, partition pruning
• Databases: Production experience with MongoDB (aggregation pipelines, indexing, bulk sync) and SQL Server (JDBC, stored procedures)
• EDI/Healthcare: Familiarity with HIPAA EDI standards (X12 837, 835, 277, 999) or willingness to learn quickly
• DevOps: Azure Pipelines (YAML), CI/CD for Databricks artifacts, PR validation
• Data Quality: Experience implementing validation frameworks, data reconciliation, and error tracking

Preferred Qualifications:

• Experience in healthcare/RCM domain — claims lifecycle, payer/provider workflows, CPT/ICD-10 coding would be a plus
• Hands-on with Databricks Unity Catalog for data governance
• Exposure to AI/ML integration in data pipelines — LLM orchestration, model observability (Langfuse or similar)
• Experience with feature flag systems for gradual data pipeline rollouts
• Performance tuning Spark jobs at scale (shuffle optimization, broadcast joins, adaptive query execution)

What You'll Impact:

• Process thousands of healthcare claims daily across multiple clients
• Reduce claim processing time through pipeline optimization
• Enable AI-powered medical coding that improves coder productivity
• Deliver real-time operational dashboards for revenue cycle teams
• Ensure HIPAA-compliant data processing at every layer