Data Engineer

Scrumconnect Consulting; Newcastle upon Tyne; On-site; GBP 45,000 - 65,000; Full time

Job summary

A leading technology consultancy in the UK is seeking a Data Engineer to develop and maintain data products. The role involves designing ETL pipelines, working in Agile teams, and ensuring operational stability of data solutions. Ideal candidates will have extensive experience in cloud data solutions, SQL, and data modeling, and will contribute to various data engineering projects.

Qualifications

• 5+ years of data engineering experience.
• Proficiency in SQL and cloud data solutions.
• Experience with ETL processes and data modeling.

Responsibilities

• Design and operate ETL data pipelines in cloud environments.
• Develop and maintain data products and code libraries.
• Collaborate with teams to transform business requirements into code.

Skills

SQL

Data Modeling

Collaboration

Education

Certifications in AWS

Certifications in Azure

Certifications in Databricks

Tools

Power BI

Git

Informatica

Tableau

NiFi

Kafka

Hadoop

Job description

Job Description

Job Title: Data Engineer

Location: UK

Job Type: Full-time

About Scrumconnect

Scrumconnect is a leading force in technology consultancy, contributing to over 20% of the UK’s significant citizen-facing public services. Our award-winning team has delivered more than 64 services in the past two years, reaching over 50 million citizens and saving taxpayers over £25 million. We foster a community of talented consultants focused on collaboration, knowledge sharing, and continuous learning to solve complex challenges. Our mission combines software engineering, human-focused design, and data-driven insights to deliver exceptional service.

About The Role And Responsibilities

We are hiring a Data Engineer to develop and maintain data products. The Data Engineering team ensures the delivery and operational stability of these products and provides ongoing support. You will work within Agile teams focused on one or more products, contributing to the entire data development lifecycle.

Role

1. Design, build, and operate ETL data pipelines within distributed and cloud environments.
2. Develop and productionize code for ETL, data validation, and data processes.
3. Understand tooling of cloud platforms like GCP, Azure, AWS.
4. Write and review code in languages such as Python, SQL, Java, Scala, R.
5. Develop data products including matching, rule development, scans, and operational outputs.
6. Maintain in-house code libraries and undertake unit testing.
7. Transform business requirements into production-ready code and data models.
8. Resolve technical issues in databases, data processes, and services.
9. Normalize data and design relational database structures.
10. Conduct basic data analysis for profiling, QA, or troubleshooting.
11. Support data quality queries and communicate issues.
12. Perform source system analysis and develop validation frameworks.
13. Collaborate with customers to update progress and confirm requirements.
14. Utilize cloud data engineering tools for pipeline development and automation.
15. Apply engineering best practices and monitor performance.
16. Advise on technology changes and support transition to modern data platforms.
17. Engage with professional communities to stay updated on emerging issues.
18. Communicate technical issues clearly to colleagues of different specialisms.
19. Monitor delivery expectations and escalate blockers proactively.
20. Develop data pipelines for large 'big data' volumes using native tools and coding approaches.
21. Adhere to engineering standards across platforms and tools.

Typical Data Engineering Experience (5+ years) includes:

• Knowledge of Azure/AWS cloud data solutions
• Proficiency in SQL
• Experience with visualization tools like Power BI
• Enterprise-scale ETL experience (e.g., Informatica)
• Data modeling and transformation skills
• Experience in large, complex projects and migration of legacy systems
• Agile methodology experience
• Ability to analyze, identify problems, and align with business objectives
• Mentoring and coaching capabilities
• Strong collaborative skills with non-technical colleagues
• Design and operate ETL pipelines in distributed/cloud environments
• Understanding of distributed data processing principles
• Write clean, efficient, documented code
• Use Git for version control
• Perform data and code analysis for QA and troubleshooting
• Proficiency in programming languages like Python, SQL, Scala, Java, R
• Investigate system issues and perform data profiling and analysis
• Knowledge of database principles and SQL across various platforms
• Awareness of innovative tools and data uses
• Experience with BI tools like Tableau, Power BI, etc.
• Familiarity with tools like NiFi, Kafka, Hadoop, and others listed
• Ability to explain technical concepts to non-technical stakeholders
• Design, build, and test data products from multiple sources
• Implement data modeling concepts and create unit tests

Desired Qualifications include certifications in AWS, Azure, Databricks, or related technologies, experience with public sector data initiatives, and knowledge of AI/ML concepts.