Senior Data Engineer - 12 month FTC

SmartestEnergy; London; Hybrid; GBP 60,000 - 85,000; Full time

Job summary

A leading energy solutions company in London is searching for a Data Engineer/Architect to design robust data architectures and manage data solutions. Candidates should have extensive experience in developing complex PySpark pipelines, Azure Data Factory, and CI/CD practices. This role offers a flexible hybrid working arrangement and the opportunity to impact global markets.

Benefits

Flexible working

Diversity and inclusion commitment

Global impact opportunities

Qualifications

• Significant experience as a Data Engineer/Architect or similar role in large-scale settings.
• Proven ability to develop complex PySpark pipelines and automated notebooks.
• Expert in building metadata-driven ETL frameworks.

Responsibilities

• Design and implement data architectures that support the data strategy.
• Develop and maintain the Data Lakehouse, Data Warehouse, and associated data solutions.
• Hands-on contribution to projects and BAU activities.

Skills

Data Engineering

PySpark

ETL Frameworks

Azure Data Factory

CI/CD

Agile/SCRUM

Tools

Azure DevOps

T-SQL

PowerShell

Job description

Overview

The purpose of this role is to design and implement robust data architectures that support the data strategy. Also, support the development and maintenance of the Data Lakehouse, Data Warehouse, Data Marts and associated Data Solutions. This is a hands-on role expected to directly contribute to development and delivery as part of projects and BAU activities. You will be involved in building and maintaining the data platform(s), ensuring data consistency, integrity, security and accessibility across the business.

Responsibilities

• Design and implement robust data architectures to support the data strategy.
• Support development and maintenance of the Data Lakehouse, Data Warehouse, Data Marts and associated data solutions.
• Contribute hands-on to development and delivery on projects and BAU activities.
• Build and maintain data platforms, ensuring data consistency, integrity, security and accessibility across the business.

Qualifications / Skills

• Substantial demonstrated commercial experience as a Data Engineer/architect or in a similar role within large-scale environments dealing with large data sets;
• Proven experience developing complex PySpark pipelines and automated notebooks;
• Expertise in building metadata-driven ETL frameworks;
• Extensive experience with one of the following technologies: Azure Data Factory, Microsoft Fabric, Synapse Analytics, strong T-SQL skills;
• Solid experience with release management, CI/CD, Azure Devops, ARM Templates and PowerShell scripting;
• Experience working within Agile/SCRUM delivery frameworks;
• Hands-on experience with Data Designs, data modelling and implementing Medallion architectures.

What sets us apart

• Global Impact: With offices in the UK, US, and Australia, and plans for further expansion, you'll be part of a dynamic, globally-minded team, with opportunities to explore new markets and make a difference on a global scale.
• Flexible Working: Embrace the freedom to work from anywhere in the world for up to 30 days a year. We prioritize work-life balance, recognizing that your well-being matters.
• Commitment to Diversity and Inclusion: We celebrate our diverse culture and value individuals irrespective of background, disability, religion, gender identity, sexuality, or ethnicity. Join a team where diversity is a driver of growth and innovation.

Working patterns

Hybrid working typically means 2 days in the office location listed on this advert and 3 days working at home each week. Some occasional travel to our other offices may be required.

What happens next?

We will review your application. The process typically includes 2 video interviews (via Teams) and may include a face-to-face meeting. Timing varies by role. Our Recruitment Team can assist with any questions or reasonable adjustments.