Hybrid Cloud Data Pipeline Engineer for Migrations

Netcompany; Leeds; Hybrid; GBP 60,000 - 90,000; Full time

Benefits offered by this job

Hybrid working model with some flexi

Private Medical Health care via Vitaly

Pension contribution

Life Assurance

Professional certifications

Retail discounts

Employee Resource Groups

Job summary

Netcompany is hiring a data engineer to build the data foundations for a large-scale migration. You will design and operate repeatable pipelines moving varied source data into a single platform, owning data processing from ingestion to delivery.

You’ll collaborate with the migration team and leverage Azure/AWS/GCP, Spark/PySpark, dbt, SQL and Python to ensure completeness, accuracy and auditable data lineage across the platform.

Qualifications

• Hands-on experience building and operating data pipelines in migration environments.
• Experience with cloud platforms Azure, AWS or Google Cloud.
• Proficiency with Spark, PySpark, Databricks or equivalent distributed processing tools.
• Experience designing modular, tested transformations with dbt or similar tooling.
• Strong SQL and Python for transformations, validation and reconciliation.
• Experience scheduling, monitoring and recovering pipelines using cloud orchestration tools.
• Knowledge of Delta Lake and transactional tables.
• Experience using infrastructure as code (Terraform) to manage cloud environments.

Responsibilities

• Design and operate repeatable data pipelines for multi-source migration.
• Transform source extracts into consistent structures for migration to a single platform.
• Develop ingestion and transformation code using SQL, Python, PySpark and dbt.
• Implement automated data quality checks for completeness and accuracy.
• Monitor, diagnose and recover failed data processing workflows.
• Maintain auditable data lineage and secure data handling controls.

Skills

Data pipelines

Azure

AWS

GCP

Spark/PySpark

dbt

SQL

Python

Cloud orchestration

Delta Lake

Tools

Databricks

Terraform

Job description

Netcompany is hiring a data engineer to build the data foundations for a large-scale migration. You will design and operate repeatable pipelines moving varied source data into a single platform, owning data processing from ingestion to delivery.

You’ll collaborate with the migration team and leverage Azure/AWS/GCP, Spark/PySpark, dbt, SQL and Python to ensure completeness, accuracy and auditable data lineage across the platform.