Senior Data Engineer: Cloud Data Pipelines & Migration

Netcompany1; Leeds; Hybrid; GBP 65,000 - 90,000; Full time

Benefits offered by this job

Hybrid working model

25 days holiday

Private Medical Health care via Vitaly

Pension contribution

Life Assurance

Professional certifications supported

Employee discounts

ERGs

Job summary

Netcompany is seeking a data engineer to design and operate repeatable data pipelines for a large-scale migration. You will own ingestion, validation and transformation from multiple sources into a unified platform, using SQL, Python, PySpark and dbt.

You'll work with Azure, AWS or GCP cloud environments, implementing automated checks and auditable data lineage while collaborating with cross-functional teams to deliver reliable data processing throughout the migration lifecycle.

Qualifications

• Hands-on experience building and operating data pipelines in a structured migration environment.
• Experience delivering reliable data processing across ingestion, validation and loading stages.
• Strong SQL and Python skills for data transformations, validation and reconciliation.

Responsibilities

• Build repeatable processes to receive, unpack, validate and ingest data from multiple sources.
• Transform source extracts into consistent structures for migration to a single platform.
• Develop maintainable ingestion and transformation code using SQL, Python, PySpark and dbt.
• Implement automated checks for completeness and accuracy across the pipeline.
• Monitor, diagnose and recover failed or incomplete data processing.
• Maintain auditable records of data sources, mappings, transformations and outcomes.

Skills

SQL

Python

PySpark

dbt

Spark

Databricks

Terraform

Azure

AWS

GCP

Tools

Databricks

Delta Lake

dbt

Terraform

Job description

Netcompany is seeking a data engineer to design and operate repeatable data pipelines for a large-scale migration. You will own ingestion, validation and transformation from multiple sources into a unified platform, using SQL, Python, PySpark and dbt.

You'll work with Azure, AWS or GCP cloud environments, implementing automated checks and auditable data lineage while collaborating with cross-functional teams to deliver reliable data processing throughout the migration lifecycle.