Senior Data Engineer (SQL, Databricks, CDC)

Curate Partners LLC; United States; Presencial; USD 90 000 - 130 000; Tempo integral

Resumo da oferta

Curate Partners LLC is seeking a data engineering professional to analyze and redesign the Provider data pipelines for incremental processing. You will assess architecture, data flows, and CDC capabilities, and collaborate with data teams to deliver scalable solutions in a healthcare domain.

You will develop patterns, validation strategies, and documentation while working in an Agile environment using Jira for delivery management.

Qualificações

• Proven expertise in SQL development and data analysis.
• Experience designing and implementing incremental processing and CDC solutions.
• Ability to perform data lineage analysis and document mappings.
• Strong understanding of ETL pipeline development and optimization.
• Experience with large-scale healthcare datasets and master data domains.
• Familiarity with cloud data platforms (Databricks, Spark).
• Excellent troubleshooting, profiling, and data quality skills.
• Ability to create technical documentation and architecture diagrams.
• Experience supporting healthcare analytics and reporting platforms.

Responsabilidades

• Analyze current Provider subject area architecture, data flows, and processing patterns.
• Assess feasibility of converting Provider area to incremental processing.
• Identify source-system capabilities (CDC, timestamps, etc.).
• Evaluate impacts to downstream applications, reporting, and analytics consumers.
• Develop recommended incremental processing design patterns and implementations.
• Create reconciliation and validation strategies for outputs.
• Collaborate with data architecture and modeling teams to align future-state designs.
• Identify technical debt, bottlenecks, and optimization opportunities in pipelines.
• Produce technical documentation, architecture diagrams, findings, and recommendations.
• Participate in Agile ceremonies and Jira-based delivery activities.

Conhecimentos

Advanced SQL

CDC / incremental processing

Data lineage analysis

ETL pipeline development

Healthcare data experience

Healthcare master data domains

Databricks / Spark

Troubleshooting & data quality

Documentation & architecture diagrams

Agile & Jira

Ferramentas

Databricks

Spark

Descrição da oferta de emprego

• Analyze the current Provider subject area architecture, data flows, and processing patterns.
• Assess the feasibility of converting the Provider subject area from full-refresh processing to incremental processing.
• Identify source-system capabilities and determine availability of change data capture (CDC), timestamps, or other incremental processing mechanisms.
• Evaluate impacts to downstream applications, reporting, and analytics consumers.
• Develop recommended incremental processing design patterns and implementation approaches.
• Create reconciliation and validation strategies to compare incremental and full-refresh outputs.
• Collaborate with data architecture and data modeling teams to align future-state designs with enterprise standards.
• Identify technical debt, performance bottlenecks, and opportunities for optimization within the Provider data pipelines.
• Produce technical documentation, architecture diagrams, findings, and implementation recommendations.
• Participate in Agile ceremonies, working sessions, backlog refinement, and technical design reviews.

Required Skills:

• Advanced SQL development and data analysis skills.
• Experience designing and implementing incremental processing and change data capture (CDC) solutions.
• Experience performing data lineage analysis and documenting source-to-target mappings.
• Strong understanding of ETL pipeline development and optimization.
• Experience working with large-scale healthcare datasets.
• Familiarity with provider, claims, enrollment, or healthcare master data domains.
• Experience with cloud data platforms such as Databricks, Spark, or equivalent distributed processing technologies.
• Strong troubleshooting, data profiling, and data quality analysis skills.
• Experience creating technical documentation, architecture diagrams, and engineering recommendations.
• Experience supporting healthcare analytics and reporting platforms.
• Experience working within Agile teams and utilizing Jira for delivery management.