Data Engineer

Sedona Digital; United States; Remote; USD 130,000 - 190,000; Full time

Job summary

Sedona Digital is seeking a Senior Data Engineer to design and implement scalable data platforms and pipelines in a cloud data environment. The role emphasizes PySpark, SQL and Databricks, with a focus on production-grade data solutions and governance within the insurance domain.

You will collaborate with engineers, data professionals and business SMEs to deliver robust data pipelines, reconciliations and well-governed data.

Qualifications

• 5+ years of professional experience in Data Engineering or closely related data roles with hands-on data pipelines experience.
• Strong hands-on experience with PySpark, SQL and Databricks, including production-grade pipelines.
• Experience with GCP data technologies and data pipelines.
• Experience with data pipeline testing, optimization and operational support.
• Experience with dimensional, master/reference or hierarchical data modelling.
• Experience implementing automated data-quality checks and reconciliation processes.
• Experience with observability, audit history, version control and CI/CD within data solutions.
• Experience documenting data contracts, mappings, lineage and dependencies.
• Experience managing controlled data pipeline or platform transitions.
• Ability to translate ambiguous reporting needs into tested engineering solutions.
• Strong communication and ability to produce technical documentation and handover materials.

Responsibilities

• Assess and re-engineer ingestion pipelines from operational and source-system data, reducing dependencies on manual feeds where possible.
• Design and implement robust data mappings, lineage, dependencies and controlled migration approaches.
• Develop and maintain detailed data reconciliation processes to validate changes and explain movements in values and counts.
• Refactor PySpark-based data processing and hierarchy roll-up capabilities with testing, versioning and observability.
• Collaborate with business SMEs to expand data coverage based on priorities.
• Assess GCP data pipeline options and contribute to implementation when platform is established.
• Source mappings and reference data from operational systems like Microsoft Dynamics365 when feasible.
• Support changes to reporting hierarchies and filtering logic with testing and reconciliation.
• Build automated data-quality checks, monitoring and operational controls.
• Support centralised reporting and data capabilities with prioritized data feeds and requirements.
• Produce concise technical documentation and handover materials.
• Deliver via backlogs and short sprints with other engineers and stakeholders.

Skills

Analytical thinking

Communication

Team collaboration

Self-motivated

Problem solving

Agile

Tools

GCP Databricks

PySpark

SQL

Microsoft Dynamics365

Power BI

Git

CI/CD

Data Quality & Reconciliation

Data Lineage

Monitoring

Job description

Описание

Accelerate your development and exposure to high‑performance data platforms and cloud infrastructure. Join Sedona Digital, a fast‑growing scale‑up with the ambition to be recognised as one of the leading technology companies in Romania. Our global client base needs builders, engineers who enjoy designing and implementing scalable data platforms, have deep expertise in cloud data technologies, and take pride in delivering reliable, well‑governed solutions. At Sedona, we: - Obsess about our customers - Build robust, scalable technical solutions - Create an open, collaborative culture - Invest in learning and long‑term careers We are looking for a Senior Data Engineer with strong hands‑on expertise in PySpark, SQL and Databricks to support a strategic data engineering engagement within the insurance sector. The role focuses on transitioning existing data and reporting capabilities towards controlled, reusable and sustainable solutions. You will work within a small product team, collaborating with engineers, data professionals and business subject‑matter experts to deliver robust data pipelines, reconciliation capabilities and well‑governed data solutions. This is a hands‑on engineering role with a strong focus on working code, automated testing, data quality, reconciliation, technical documentation and effective handover.

Задачи

• Assess and re‑engineer ingestion pipelines from operational and source‑system data, reducing dependencies on tactical or manually maintained data feeds where appropriate.
• Design and implement robust data mappings, lineage, dependencies and controlled migration or cutover approaches.
• Develop and maintain detailed data reconciliation processes to validate engineering changes and explain movements in values, record counts, inclusion rules and hierarchy assignments.
• Refactor and enhance PySpark‑based data processing and hierarchy/roll‑up capabilities, introducing appropriate testing, versioning, metadata, observability and deployment controls.
• Work closely with business and technical SMEs to expand and improve data coverage based on business priorities and consumer requirements.
• Assess GCP data pipeline options and contribute to implementation where the platform and operating model are appropriately established.
• Assess opportunities to source mappings and reference data directly from operational systems such as Microsoft Dynamics365, reducing reliance on manual data submissions.
• Support changes to reporting hierarchies and filtering logic, including impact assessment, testing and reconciliation.
• Build and maintain automated data‑quality checks, monitoring and operational controls.
• Support the transition and adoption of centralised reporting and data capabilities by incorporating prioritised data feeds and requirements.
• Produce concise technical documentation, decision records, reconciliation evidence, variance explanations and handover materials.
• Deliver through a prioritised backlog and short iterative sprints, working collaboratively with other engineers and stakeholders.

Требования

• 5+ years of professional experience in Data Engineering or closely related data roles, with significant hands‑on experience building and supporting production data solutions.
• Strong hands‑on experience with PySpark, SQL and Databricks, including the development and support of production‑grade data pipelines.
• Hands‑on experience with GCP data technologies and data pipelines.
• Experience with data pipeline testing, optimisation and operational support.
• Strong experience with dimensional, master/reference or hierarchical data modelling.
• Experience implementing complex entity matching, data mappings and roll‑up logic with appropriate traceability.
• Practical experience implementing automated data‑quality checks and reconciliation processes.
• Experience with observability, audit history, version control and CI/CD within data engineering solutions.
• Experience assessing source‑system data and documenting data contracts, mappings, lineage and dependencies.
• Experience managing controlled data pipeline or platform transitions.
• Ability to investigate data issues and clearly explain variances in values, record counts, mappings and hierarchy outcomes.
• Ability to work directly with platform teams, analysts and business SMEs, translating ambiguous reporting and mapping needs into tested engineering solutions.
• Strong communication skills with the ability to produce concise technical documentation, decision records, variance explanations and handover materials.
• Self‑motivated and comfortable working within a collaborative, iterative delivery environment.
• Experience working within Insurance, London Market, Reinsurance, broker, distribution or policy‑level data environments.
• Experience integrating with Microsoft Dynamics365 or comparable operational source systems.
• Experience with Power BI, including semantic models and reporting dependencies.
• Experience with Slowly Changing Dimensions (SCDs) and historical hierarchy management.
• Experience supporting complex master/reference data, entity matching and hierarchy solutions.
• Experience investigating historical data and reporting discrepancies.
• Experience supporting controlled data migrations, source‑system transitions or platform changes.
• Experience working within distributed Agile teams and collaborating with UK‑based business and technical stakeholders.

Key Tools & Technologies

• GCP Databricks
• PySpark
• SQL
• Microsoft Dynamics365
• Power BI
• Git / Version Control
• CI/CD
• Data Quality & Reconciliation
• Data Lineage & Metadata Management
• Monitoring & Observability