Senior CDC & Incremental Pipelines Engineer (Databricks)

Curate Partners LLC

United States

On-site

USD 90,000 - 130,000

Full time

2 days ago

Be an early applicant

Application generator

Turn this role into an interview — a resume and cover letter built around what this employer wants.

Get past ATS filters

Job summary

Curate Partners LLC is seeking a data engineering professional to analyze and redesign the Provider data pipelines for incremental processing. You will assess architecture, data flows, and CDC capabilities, and collaborate with data teams to deliver scalable solutions in a healthcare domain.

You will develop patterns, validation strategies, and documentation while working in an Agile environment using Jira for delivery management.

Qualifications

Proven expertise in SQL development and data analysis.
Experience designing and implementing incremental processing and CDC solutions.
Ability to perform data lineage analysis and document mappings.
Strong understanding of ETL pipeline development and optimization.
Experience with large-scale healthcare datasets and master data domains.
Familiarity with cloud data platforms (Databricks, Spark).
Excellent troubleshooting, profiling, and data quality skills.
Ability to create technical documentation and architecture diagrams.
Experience supporting healthcare analytics and reporting platforms.

Responsibilities

Analyze current Provider subject area architecture, data flows, and processing patterns.
Assess feasibility of converting Provider area to incremental processing.
Identify source-system capabilities (CDC, timestamps, etc.).
Evaluate impacts to downstream applications, reporting, and analytics consumers.
Develop recommended incremental processing design patterns and implementations.
Create reconciliation and validation strategies for outputs.
Collaborate with data architecture and modeling teams to align future-state designs.
Identify technical debt, bottlenecks, and optimization opportunities in pipelines.
Produce technical documentation, architecture diagrams, findings, and recommendations.
Participate in Agile ceremonies and Jira-based delivery activities.