Data Engineer

Nigel Frank; Greater London; On-site; GBP 50,000 - 70,000; Full time

Job summary

A leading recruitment firm seeks a skilled Data Engineer in Greater London. You will support a major data transformation project within a clinical environment, focusing on modernizing a screening service. Responsibilities include constructing ingestion pipelines and addressing data quality issues while leveraging your skills in SQL and Python. The role requires eligibility for UK Security Clearance due to the nature of the data involved.

Responsibilities

• Support major data transformation workstream in a clinical setting.
• Build secure, repeatable data ingestion and transformation pipelines.
• Resolve data quality challenges and shape future data models.

Skills

Data transformation pipelines

Advanced SQL expertise

Python scripting

Data tooling (Spark, Azure Data Factory)

Geospatial datasets

Communication and collaboration skills

Proactive and independent working

Job description

Data Engineer

A skilled Data Engineer is required to support a major data‑transformation workstream within a clinical screening and diagnostics environment. The project focuses on modernising an end‑to‑end screening service through modern, data‑driven digital capabilities. You will join a small specialist team responsible for analysing complex datasets spread across multiple system instances, resolving data‑quality challenges, and shaping future data models and structures.

The core responsibility is to build secure, repeatable ingestion and transformation pipelines, apply robust data‑cleansing logic, and produce auditable, reproducible outputs.

Essential Skills

• Experience establishing import/export patterns, including handling data extracts, schema discovery, incremental loading and normalising data across multiple source systems.
• Strong capability in data‑transformation‑heavy pipelines covering profiling, cleansing, standardisation, conformance and final data publishing.
• Advanced SQL expertise including profiling, joins/merges, deduplication, anomaly detection and performance tuning.
• Practical Python scripting experience for automation, parsing, rules engines and data‑quality checks, using libraries such as Pandas/Polars, scikit‑learn or matplotlib.
• Experience with modern data tooling (e.g., Spark, Azure Data Factory) or the ability to deliver equivalent functionality in code‑based environments.
• Proven experience working with geospatial datasets (vector, raster, GeoJSON, shapefiles), including coordinate systems, spatial data handling and geospatial analysis workflows.
• Ability to interpret geographical context and aggregate/upscale local or regional geospatial insights into coherent national‑ or region‑level datasets.
• Experience working with publicly available official datasets (e.g., census boundaries, geographic lookups, deprivation indices, population estimates).
• Capability to design rules for completeness, validity and consistency, and to implement exception handling and reconciliation flows.
• Ability to build version‑controlled pipelines with deterministic transformations, logging, lineage and full traceability of data changes.
• Comfortable working in a secure environment with least‑privilege access principles, secure storage/transfer practices and handling of sensitive personal data.

Soft Skills

• Strong communication and collaboration skills.
• A team‑focused, cooperative approach.
• Enthusiasm, engagement and a positive attitude.
• Proactivity‑comfortable working independently without constant direction.
• Ability to handle ambiguity and adapt to change effectively.

Nice‑to‑Have Skills

• Experience working with healthcare or medical‑sector datasets, including patient/episode‑style records or longitudinal histories.
• Experience building automated data‑profiling dashboards or reporting frameworks.

Additional Requirements

Candidates must be eligible for UK Security Clearance due to the sensitive nature of the data involved.

BPSS eligible also