Senior Databricks Platform Engineer - Medicare (Remote)
BigRio, LLC
11 days ago
A complete application in a minute
Tailored resume and cover letter, ready to send.
Get past ATS filters
Job summary
BigRio is seeking a Senior Databricks Developer specializing in Medicare Advantage data to build and maintain lakehouse pipelines on the Databricks platform.
You will work remotely from the U.S., focusing on Delta Live Tables, Unity Catalog, and Spark tuning to deliver compliant, production-grade data solutions for claims, eligibility, and provider datasets.
Qualifications
- 5+ years of hands-on Databricks experience.
- Strong command of Spark (PySpark/Scala) and Databricks SQL.
- Hands-on experience building Delta Live Tables pipelines.
- Experience with Unity Catalog for governance and data lineage.
- Experience with Databricks Workflows/Jobs for orchestration and scheduling.
- Tuning cluster and job performance and cost optimization within Databricks.
- CI/CD for notebook/pipeline deployment (Databricks Repos, Azure DevOps/Git).
- 3–10+ years of healthcare payer domain experience, ideally Medicare Advantage data.
Responsibilities
- Build, deploy, and maintain data pipelines within Databricks.
- Develop and optimize Spark logic for Medicare Advantage data.
- Manage Delta Lake architecture (bronze/silver/gold) for clean, governed data.
- Configure Unity Catalog for governance and access control across PHI datasets.
- Tune clusters, jobs, and costs; monitor pipeline health and troubleshoot issues.
Skills
Databricks
Spark
Delta Live Tables
Unity Catalog
Workflows/Jobs
Azure
PHI/HIPAA security
Python/Scala
Tools
Databricks Repos
CI/CD
Azure DevOps
Git
Job description
Databricks Developer – Medicare
< Back
About the job
Databricks Developer – Medicare
- Advantage (Payer)
- Work Model: Remote (U.S.-based candidates only)
- Level: Senior
- Domain: Healthcare Payer / Medicare Advantage
Overview
- Our client, a leading Medicare Advantage health plan, is seeking an experienced Databricks Developer to build and
- operate data solutions on the Databricks lakehouse platform, supporting claims, eligibility, and member/provider
- data used across Medicare Advantage reporting and compliance. This is a Databricks platform-focused role: the
- ideal candidate lives inside Databricks day to day — notebooks, Delta Live Tables, Workflows, Unity Catalog, and
- cluster/job tuning — rather than a general data engineer who happens to use Databricks as one of several tools.
- Candidates should be comfortable working independently in a remote setup and available to start promptly.
Key Responsibilities
- Build, deploy, and maintain data pipelines natively within Databricks, using notebooks, Delta Live Tables
- (DLT), and Databricks Workflows/Jobs for orchestration.
- Develop and optimize Spark (PySpark/Scala/SQL) logic for processing large-scale Medicare Advantage
- claims (EDI 837/835), eligibility, and provider data within the Databricks environment.
- Manage and evolve the Delta Lake medallion architecture (bronze/silver/gold) to ensure clean, reliable,
- well-governed data.
- Configure and maintain Unity Catalog for data governance, lineage, and access control across PHI/HIPAA-
- regulated datasets.
- Tune cluster configurations, job performance, and cost efficiency (autoscaling, cluster policies, Photon,
- job/task-level optimization) within Databricks.
- Use Databricks SQL and SQL warehouses to support ad hoc analysis, validation, and reconciliation of
- Medicare Advantage claims and eligibility data.
- Monitor pipeline health, troubleshoot job failures, and resolve performance issues directly within the
- Databricks platform (Jobs UI, cluster logs, DLT event logs).
- Manage code and deployments via Databricks Repos and CI/CD integration (Databricks Asset Bundles, Azure DevOps/Git).
- Apply PHI/HIPAA-compliant data handling and role-based access practices throughout all Databricks workspace and pipeline configurations.
- Document Databricks pipeline design, workspace structure, and job configurations.
Required Experience & Skills
- 5+ years of hands-on Databricks experience — this is a Databricks-first role; deep platform fluency is the primary requirement.
- Strong command of Spark (PySpark/Scala) and Databricks SQL for pipeline development.
- Hands-on experience building and managing Delta Live Tables (DLT) pipelines.
- Practical experience with Unity Catalog for data governance, access control, and lineage.
- Experience with Databricks Workflows/Jobs for orchestration and scheduling.
- Demonstrated ability to tune cluster and job performance (autoscaling, cluster policies, Photon, cost optimization) within Databricks.
- Experience with Databricks Repos and Git-based CI/CD for notebook/pipeline deployment.
- 3–10+ years of healthcare payer domain experience, ideally with Medicare Advantage data (claims ingestion, eligibility logic, EDI 837/835, CMS-aligned reporting).
- Working knowledge of PHI/HIPAA-regulated data handling and role-based security within a data platform.
- Comfortable working within an Azure or equivalent cloud environment (ADLS, workspace/networking basics) as the underlying infrastructure for Databricks.
- Strong problem-solving skills and ability to work independently in a remote environment.
- U.S.-based, remote-capable (U.S. resident, citizen, or Green Card holder).
Nice to Have
- Databricks Certified Data Engineer (Associate or Professional).
- Experience with MLflow or Databricks’ ML/AI tooling.
- Exposure to Azure Data Factory, Power BI, or Snowflake in a supporting/downstream capacity.
- Experience migrating legacy ETL workloads into Databricks.
- Prior experience in a client-facing or consulting environment.
- Experience with EHR-adjacent systems (e.g., Epic Clarity) or provider-side data in addition to payer data.
Ideal Candidate Profile
“Senior Databricks Developer who lives in the platform day to day — Delta Live Tables, Unity Catalog, Workflows,
and Spark performance tuning — with real Medicare Advantage claims and eligibility experience, and a track
record of running governed, production-grade lakehouse pipelines end to end within Databricks.
Apply NowBigRio is a leading AI, Gen AI, Data and Analytics professional services company. We are focused on Healthcare, Pharma, Digital Health, Provider, and Payer Industry segments with several innovative solutions.
Company
- Our Team
- Careers
- Portfolio Partners
- Contact Us
Knowledge Center
- Events
- Case Studies
- White Papers
- Blogs
GET IN TOUCH
- Harvard Square, One Mifflin Place
Suite 400
Cambridge, MA 02138 - info@bigr.io
mmMwWLliI0fiflO&1
mmMwWLliI0fiflO&1
mmMwWLliI0fiflO&1
mmMwWLliI0fiflO&1
mmMwWLliI0fiflO&1
mmMwWLliI0fiflO&1
mmMwWLliI0fiflO&1
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
