Senior Data Engineer: Migration & Data Pipelines (Hybrid)

Netcompany; Greater London; Hybrid; GBP 75,000 - 110,000; Full time

Benefits offered by this job

Hybrid working model

Private medical health care

Pension contribution

Professional certifications support

Employee discounts

Employee Resource Groups

Job summary

Netcompany is seeking a data engineering professional to build the data foundations for a large-scale migration. You’ll design and operate repeatable pipelines moving varied source data into a single platform, taking ownership of ingestion to delivery.

You’ll work with the migration team, develop modular transformations using dbt and PySpark, and ensure data completeness with automated checks. Hybrid working with modern cloud stacks is offered.

Qualifications

• Hands-on experience building and operating data pipelines in a migration environment.
• Experience building and operating data pipelines on Azure, AWS or Google Cloud.
• Experience with Spark, PySpark, Databricks or comparable distributed processing technologies.
• Experience building modular, tested and documented transformations using dbt or an equivalent approach.
• Strong SQL and Python skills for writing and troubleshooting transformations, validation and reconciliation queries.
• Experience scheduling, monitoring and recovering pipelines using cloud-native or equivalent orchestration tools.
• Experience with Delta Lake, including transactional tables, schema changes and repeatable batch loading.
• Experience using Terraform or equivalent infrastructure as code to provision and maintain consistent cloud environments.

Responsibilities

• Design and operate repeatable data pipelines for large-scale migrations.
• Collaborate with data lead, analysts, platform team and source organisations.
• Build processes to ingest and validate data from multiple sources.
• Transform source extracts into consistent structures for migration.
• Develop maintainable ingestion and transformation code (SQL, Python, PySpark, dbt).
• Implement automated checks for data completeness and accuracy.
• Monitor, diagnose and recover failed or incomplete data processing.
• Maintain auditable data lineage and secure data handling controls.

Skills

Data pipelines

SQL

Python

Cloud platforms

Spark / PySpark

dbt

Orchestration tooling

Delta Lake

Terraform

Tools

Azure

AWS

Google Cloud

Databricks

Spark

PySpark

dbt

Terraform

Job description

Netcompany is seeking a data engineering professional to build the data foundations for a large-scale migration. You’ll design and operate repeatable pipelines moving varied source data into a single platform, taking ownership of ingestion to delivery.

You’ll work with the migration team, develop modular transformations using dbt and PySpark, and ensure data completeness with automated checks. Hybrid working with modern cloud stacks is offered.