Data QA - Test Engineer

Morgan McKinley; City Of London; Presencial; GBP 65.000 - 110.000; Jornada completa

Descripción de la vacante

Morgan McKinley seeks a hands-on Senior Data QA / Data Quality Engineer to lead automated migration testing for an investment firm's eFront to Azure Databricks Lakehouse migration. You will own source-to-target reconciliation, validate financial calculations, and ensure 100% data integrity across Bronze/Silver/Gold layers.

Responsibilities include building automated frameworks with PyTest, PySpark, and advanced SQL, integrating checks into Azure DevOps pipelines, and monitoring schema drift and

Formación

• Hands-on experience testing data from complex pipelines and migrations.
• Experience with automated validation and reconciliation between source and target systems.
• Familiarity with private markets terminology and financial data structures.

Responsabilidades

• Lead automated migration testing and validation for data from eFront to Databricks Lakehouse.
• Design and implement source-to-target reconciliation queries and checks.
• Develop automated data quality tests using PyTest, PySpark, and advanced SQL.

Conocimientos

Python

PyTest

PySpark

Advanced SQL

Azure DevOps

Data validation

Reconciliation

Herramientas

Azure Databricks

Delta Lake

Descripción del empleo

Role Overview

We are partnering with an established investment firm undertaking a core data migration from eFront Invest FIA into an Azure Databricks Lakehouse.

We are seeking a hands-on Senior Data QA / Data Quality Engineer to lead automated migration testing, source-to-target reconciliation, and pipeline validation. You will be responsible for ensuring that all data flowing from eFront lands in Databricks with 100% mathematical and schema integrity.

Key Responsibilities

• Source-to-Target Reconciliation: Design, build, and execute automated validation queries to compare legacy eFront extracts against target Databricks Delta Lake tables (Bronze, Silver, Gold).
• Business Logic & Metrics Testing: Verify complex private equity calculation logic (cash flows, commitments, valuations, NAV, and return metrics) across data pipelines.
• Automated Data Quality Frameworks: Develop automated test suites using Python (PyTest), PySpark, and advanced SQL to catch discrepancies, nulls, and duplicate records.
• CI/CD Quality Gates: Integrate automated testing and data validation checks into Azure DevOps (ADO) deployment pipelines.
• Schema Drift & Anomaly Detection: Monitor and resolve schema evolution, edge-case discrepancies, and late-arriving financial records.

Required Experience & Skills

• Experience testing data extracted from or integrated with eFront (eFront Invest / FIA / FrontScript).
• Proven track record as a Data QA Engineer, Data Quality Engineer, or Data Test Automation Engineer on enterprise cloud migrations.
• Hands-on testing and reconciliation experience within Azure Databricks / Delta Lake environments.
• Familiarity with private markets, fund accounting, or private equity financial structures (NAV, cash flows, capital calls, distributions).