Senior Data Engineer: Cloud Data Pipelines & Migrations

Netcompany Group A/S; Leeds; Hybrid; GBP 60,000 - 90,000; Full time

Benefits offered by this job

Hybrid working model

Private Medical Health care via Vitaly

Pension contribution and Life Assuar

Professional certifications supported

Employee discounts

Employee Resource Groups

Job summary

Netcompany is a leading European technology partner delivering digital solutions across governments, industries, and businesses. The role focuses on building data foundations for large-scale migrations, owning end-to-end data processing, and developing robust ingestion/transformation pipelines using SQL, Python, PySpark and dbt.

Work within a migration team across data, platform and source organisations. Hybrid working model with strong focus on security, governance and accountability in data

Qualifications

• Hands-on experience building and operating data pipelines in a structured migration environment.
• Experience building and operating data pipelines on Azure, AWS or Google Cloud.
• Experience with Spark, PySpark, Databricks or comparable distributed processing technologies.
• Experience building modular, tested and documented transformations using dbt or an equivalent approach.
• Strong SQL and Python skills for writing and troubleshooting transformations, validation and reconciliation queries.
• Experience scheduling, monitoring and recovering pipelines using cloud-native or equivalent orchestration tools.
• Experience with Delta Lake, including transactional tables, schema changes and repeatable batch loading.
• Experience using Terraform or equivalent infrastructure as code to provision and maintain consistent cloud environments.

Responsibilities

• Build the data foundations for a large-scale migration.
• Design and operate repeatable pipelines moving source data into a single platform.
• Own data processing from ingestion to delivery.
• Develop maintainable ingestion and transformation code using SQL, Python, PySpark and dbt.
• Implement automated data quality checks for completeness and accuracy.
• Monitor, diagnose and recover failed or incomplete data processing.
• Maintain auditable records of data sources, mappings, transformations and outcomes.

Skills

SQL

Python

PySpark

dbt

Spark

Delta Lake

Terraform

Databricks

Cloud platforms

Data pipelines

Tools

Databricks

dbt

Airflow

Terraform

Job description

Netcompany is a leading European technology partner delivering digital solutions across governments, industries, and businesses. The role focuses on building data foundations for large-scale migrations, owning end-to-end data processing, and developing robust ingestion/transformation pipelines using SQL, Python, PySpark and dbt.

Work within a migration team across data, platform and source organisations. Hybrid working model with strong focus on security, governance and accountability in data