Hybrid Data Engineer—Cloud Pipelines & Migrations

Netcompany1; Leeds; Hybrid; GBP 60,000 - 90,000; Full time

Benefits offered by this job

Hybrid working model

25 days holiday

Private medical health care

Pension contribution

Life Assurance

Professional certifications supported

Employee discounts

ERGs

Job summary

Netcompany is looking for a data engineer to build and operate data pipelines for a large-scale migration. You will design repeatable ingestion, validation and transformation processes, moving disparate sources into a single platform.

You’ll work with the data lead, analysts and cloud platform teams, developing code in SQL, Python, PySpark and dbt, implementing automated checks, monitoring pipelines and maintaining auditable data records.

Qualifications

• Hands-on experience building and operating data pipelines on Azure, AWS or Google Cloud.
• Experience with Spark, PySpark, Databricks or comparable distributed processing technologies.
• Experience building modular, tested and documented transformations using dbt or an equivalent approach.
• Strong SQL and Python skills for writing and troubleshooting transformations, validation and reconciliation queries.
• Experience scheduling, monitoring and recovering pipelines using cloud-native or equivalent orchestration tools.
• Experience with Delta Lake, including transactional tables, schema changes and repeatable batch loading.
• Experience using Terraform or equivalent infrastructure as code to provision and maintain consistent cloud environments.

Responsibilities

• Build repeatable processes to receive, unpack, validate and ingest data from multiple sources.
• Transform source extracts into consistent structures for migration to a single platform.
• Develop maintainable ingestion and transformation code using SQL, Python, PySpark and dbt.
• Implement automated checks that establish completeness and accuracy throughout the migration.
• Monitor, diagnose and recover failed or incomplete data processing.
• Maintain auditable records of data sources, mappings, transformations and outcomes, and apply agreed controls for secure data handling.

Skills

SQL

Python

Data pipelines

Cloud data platforms

PySpark

dbt

Terraform

Delta Lake

Tools

Spark

PySpark

Databricks

dbt

Delta Lake

Terraform

Azure

AWS

Google Cloud

Job description

Netcompany is looking for a data engineer to build and operate data pipelines for a large-scale migration. You will design repeatable ingestion, validation and transformation processes, moving disparate sources into a single platform.

You’ll work with the data lead, analysts and cloud platform teams, developing code in SQL, Python, PySpark and dbt, implementing automated checks, monitoring pipelines and maintaining auditable data records.