Senior Data Engineer

Netcompany1; Greater London; Hybrid; GBP 65,000 - 90,000; Full time

Benefits offered by this job

Hybrid working model

25 days' holiday

Private Medical Health care via Vital1

Pension contribution

Life Assurance

Professional certifications supported

Retail discounts

Employee Resource Groups

Job summary

Netcompany is seeking an experienced data engineer to build and operate scalable data pipelines for a large migration project. You will design repeatable ingestion, validation and transformation processes across cloud platforms, using SQL, Python, PySpark and dbt.

You will partner with data leads, analysts and platform teams to deliver reliable data processing, ensuring traceability, governance and auditable records throughout the migration lifecycle.

Qualifications

• Hands-on experience building and operating data pipelines in cloud environments.
• Experience with data validation, reconciliation queries and auditing.
• Proficiency in SQL and Python for data transformations and troubleshooting.

Responsibilities

• Build repeatable data ingestion and transformation processes from multiple sources.
• Transform source extracts into consistent structures for migration to a single platform.
• Develop maintainable ingestion and transformation code using SQL, Python, PySpark and dbt.

Skills

SQL

Python

PySpark

dbt

Delta Lake

Terraform

Cloud platforms

Orchestration tools

Tools

Databricks

Azure

AWS

Google Cloud

Job description

Build the data foundations for a large-scale migration. You'll design and operate repeatable pipelines that move varied source data into a single platform, taking ownership of data processing from ingestion to delivery.

Key responsibilities

• You'll work as part of a migration team, collaborating with the data lead, analysts, the platform team and source organisations throughout the delivery.
• Build repeatable processes to receive, unpack, validate and ingest data from multiple sources.
• Transform source extracts into consistent structures for migration to a single platform.
• Develop maintainable ingestion and transformation code using SQL, Python, PySpark and dbt.
• Implement automated checks that establish completeness and accuracy throughout the migration.
• Monitor, diagnose and recover failed or incomplete data processing.
• Maintain auditable records of data sources, mappings, transformations and outcomes, and apply agreed controls for secure data handling.

You bring hands‑on experience building and operating data pipelines, with the technical skills to deliver reliable data processing in a structured migration environment.

• Hands‑on experience building and operating data pipelines on Azure, AWS or Google Cloud.
• Experience with Spark, PySpark, Databricks or comparable distributed processing technologies.
• Experience building modular, tested and documented transformations using dbt or an equivalent approach.
• Strong SQL and Python skills for writing and troubleshooting transformations, validation and reconciliation queries.
• Experience scheduling, monitoring and recovering pipelines using cloud‑native or equivalent orchestration tools.
• Experience with Delta Lake, including transactional tables, schema changes and repeatable batch loading.
• Experience using Terraform or equivalent infrastructure as code to provision and maintain consistent cloud environments.

Build on your experience and keep developing

Your development continues through the responsibility you take, the projects you lead and the people you learn from. As part of a structured career model, you'll build on the experience you already bring, supported by a personal mentor from the leadership group who can help guide your professional development, career direction and personal goals.

Be part of a shared community

Community is built through the way we work together. You'll join teams where collaboration, trust and support are part of everyday work. We bring together colleagues with different backgrounds, identities, cultures and experiences, ensuring everyone is met with respect and has equal opportunities to develop and thrive.

Driven by responsible digitalisation

We believe digitalisation creates the most value when it is done responsibly. We build solutions that are secure, transparent, and designed to create lasting impact for people, businesses, and society. Responsibility is embedded in how we design, develop, and deliver every solution.

Benefits include

• Hybrid working model with some flexible working
• 25 days' holiday
• Private Medical Health care via Vitality
• Pension contribution, Life Assurance
• Professional certifications supported as part of learning and development.
• A range of retail discounts to enhance your lifestyle, encompassing restaurants, supermarkets, travel, leisure activities and health and well-being services.
• Access to our Employee Resource Groups, our groups represent diverse backgrounds and provide a platform for colleagues to connect, learn, and support one another.

Company information

We pride ourselves on our entrepreneurial spirit and our capacity for doing things differently. Our culture is built on fostering low bureaucracy, emphasizing high agility and promoting flexibility, enabling everyone to contribute their best.

Netcompany is committed to providing an inclusive and barrier‑free recruitment process for anyone wishing to apply for a job with us. We operate a Disability Confident Interview Scheme which means that we offer guaranteed interviews to all applicants with a disability who meet the minimum criteria for a role. Applicants are asked to opt in, so please let us know if you wish your application to be considered under this scheme.

If you need any reasonable adjustments or extra support during the recruitment process, please contact us at [e-mail] .