Senior Data Engineer

Rullion - Eon; East Midlands; On-site; GBP 40,000 - 70,000; Full time

Job summary

A major UK energy provider is seeking talented data engineers to join their data transformation journey. The role involves developing efficient data pipelines using Databricks, ensuring data quality, and collaborating with stakeholders. This full-time, hybrid position offers a chance to contribute to a sustainable energy future.

Qualifications

• Extensive experience with Python and SQL.
• Experience with Databricks and/or Spark.
• Knowledge of data governance and quality principles.

Responsibilities

• Develop and maintain scalable data pipelines within Databricks.
• Integrate new data sources using batch and streaming processes.
• Ensure data quality and reliability.

Skills

Python

SQL

Data Governance

Data Architecture

Databricks

Spark

Data Cleansing

Data Transformation

AWS

Linux

Tools

Git

Job description

Join our client on an ambitious data transformation journey using Databricks, guided by best practice data governance and architectural principles.

We are recruiting talented data engineers for a major UK energy provider committed to 100% renewable energy and sustainability, focusing on delivering exceptional customer experiences.

Contract Details: Initially 3 months with potential extension. Hybrid role with one day a week in Nottingham (negotiable). Full-time, 37 hours/week.

Accountabilities:

1. Develop and maintain scalable, efficient data pipelines within Databricks, evolving them as requirements and technologies change.
2. Build and manage an enterprise data model within Databricks.
3. Integrate new data sources using batch and streaming processes, adhering to SLAs.
4. Create and maintain documentation for data pipelines, following security and monitoring protocols.
5. Ensure data quality and reliability, maintaining trust in the data.
6. Take ownership of complex data engineering projects and develop solutions aligned with business needs.
7. Collaborate closely with stakeholders to manage requirements.
8. Coach and mentor team members, fostering innovation and best practices.

Knowledge and Skills:

• Extensive experience with Python, including advanced concepts.
• Strong SQL, database design, and data architecture skills.
• Experience with Databricks and/or Spark.
• Knowledge of data governance, cataloguing, quality principles, and tools.
• Skilled in data extraction, joining, and aggregation, especially with big data and real-time data using Spark.
• Proficient in data cleansing and transformation for analysis.
• Understanding of data storage concepts and warehousing.
• Ability to write production-quality, repeatable data pipeline code.
• Experience with data migration is advantageous.
• Open to new technologies and ways of working.
• Self-motivated and proactive problem solver.
• Experience with Git/version control, legacy codebases, testing, CI/CD, and software best practices.
• Attention to detail and curiosity about data.
• Strong Linux tooling knowledge.
• Experience with AWS is essential.

Additional Notes:

Successful applicants will undergo pre-employment checks, including DBS verification. This vacancy is advertised by Rullion Ltd, committed to diversity and equal opportunities.