Senior Data Engineer

ZipRecruiter; Nottingham; Hybrid; GBP 40,000 - 55,000; Full time

Job summary

A major UK energy provider is seeking a Data Engineer to support their ambitious data transformation journey leveraging Databricks. The role involves developing efficient data pipelines, ensuring data quality, and mentoring team members. This is a full-time, initially 3-month contract with hybrid working options, geared towards professionals passionate about renewable energy and exceptional customer experiences.

Qualifications

• Extensive experience in Python, SQL, and data architecture.
• Strong understanding of data governance and quality principles.
• Familiarity with AWS and version control systems.

Responsibilities

• Develop and maintain scalable data pipelines within Databricks.
• Integrate new data sources and ensure data quality.
• Coach and mentor team members in data engineering best practices.

Skills

Python

SQL

Data governance

Data quality principles

Spark

Data extraction and transformation

Linux

Tools

Databricks

AWS

Git

Job description

Job Description

Join our client in embarking on an ambitious data transformation journey using Databricks, guided by best practice data governance and architectural principles. To support this, we are recruiting talented data engineers. As a major UK energy provider, our client is committed to 100% renewable energy and sustainability, focusing on delivering exceptional customer experiences.

It is initially a 3-month contract with potential extension. The role is hybrid, with one day a week based in their Nottingham office, negotiable. It is a full-time role, 37 hours per week.

Accountabilities:

1. Develop and maintain scalable, efficient data pipelines within Databricks, evolving them as requirements and technologies change.
2. Build and manage an enterprise data model within Databricks.
3. Integrate new data sources into the platform using batch and streaming processes, adhering to SLAs.
4. Create and maintain documentation for data pipelines and systems, following security and monitoring protocols.
5. Ensure data quality and reliability processes are effective to maintain trust in the data.
6. Take ownership of complex data engineering projects and develop solutions aligned with business needs.
7. Work closely with stakeholders to manage requirements.
8. Coach and mentor team members, fostering a culture of innovation and peer review to ensure best practices.

Knowledge and Skills:

1. Extensive experience with Python, including advanced concepts like decorators, protocols, functools, context managers, and comprehensions.
2. Strong understanding of SQL, database design, and data architecture.
3. Experience with Databricks and/or Spark.
4. Knowledge of data governance, data cataloguing, data quality principles, and related tools.
5. Skilled in data extraction, joining, and aggregation tasks, especially with big data and real-time data using Spark.
6. Proficient in data cleansing and transforming data for analysis.
7. Understanding of data storage concepts and logical structures like data warehousing.
8. Ability to write production-quality, repeatable code for data pipelines, using templating and parameterization.
9. Ability to recommend data pipeline designs based on business needs.
10. Experience with data migration is a plus.
11. Open to new technologies and ways of working.
12. Self-motivated, goal-oriented, and proactive.
13. Strong troubleshooting skills and problem-solving ability.
14. Experience with Git/version control, large legacy codebases, unit and integration testing, CI/CD, and software development best practices.
15. Attention to detail and curiosity about data.
16. Strong understanding of Linux tooling and concepts.
17. Knowledge and experience with AWS is essential.

Note: Successful applicants will undergo pre-employment checks, including a satisfactory DBS check. This vacancy is advertised by Rullion Ltd, an employment business. Rullion is committed to equal opportunities.