Principal Data Engineer

Office for Students; United Kingdom; Hybrid; GBP 64,800 - 79,200; Full time

Job summary

An independent education regulator in the UK seeks a Principal Data Engineer to innovate data solutions on the Azure Databricks platform. The role involves building reusable data pipelines, leading IT and Data collaborations, and enhancing data quality. Ideal candidates will possess strong Python and Azure knowledge, and a flexible approach to evolving challenges.

Qualifications

• Experience in building and developing reusable pipelines for analytics and AI projects.
• Familiarity with monitoring execution of data workflows and setting service level indicators.
• Ability to develop coding standards for Python programming.

Responsibilities

• Build and develop pipelines for analytics and AI projects.
• Lead relationships between IT and Data for platform growth.
• Support continuous data quality improvements.

Skills

Strong Python programming knowledge, ideally PySpark

Strong knowledge of the Azure Databricks platform

Adaptable with a willingness to work flexibly

Ability to communicate technical concepts to nontechnical audience

Logical and analytical approach to problem-solving

Job description

Overview

Principal Data Engineer role at Office for Students

Base pay range up to £72,000. 12 months fixed term contract. Location: Bristol.

This range is provided by Office for Students. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.

About us

We are the independent regulator of higher education in England. We aim to ensure that every student, whatever their background, has a fulfilling experience of higher education that enriches their lives and careers. We’re proud of our people, our culture and the benefits we offer. We care about supporting and developing our staff. At the OfS we want to create a welcoming and engaging working environment which is free from discrimination.

We’re looking for skilled, committed people who care about students and their experience of higher education. Join us and you’ll be part of an energetic, forward-looking organisation with high ambitions and big opportunities for our people.

The analytics & data engineering team are a newly formed team in the OfS. The team is there to make sure the analyst team and subsequently the business has the right information to carry out the functions of the organisation.

Why work for the OfS

The OfS\'s Data function is currently undergoing an exciting and major transformation and we\'re looking for a talented data engineer to help lead the change.

We\'re moving away from a legacy on-premises platform to an Azure Databricks platform, unlocking new possibilities for automation and innovation. This is your opportunity to help develop new robust data products from the ground up, set best practice and introduce AI into the organisation to improve how we support the higher education sector and the needs of students.

Key responsibilities

You will be a key individual contributor and be responsible for:

• Building and developing reusable pipelines for analytics and AI projects
• Pushing for innovation within the platform to enable great efficiencies and detailed insights and outputs
• Leading key relationships between IT and Data to grow the platform and release new capabilities
• Deploying production AI models with automated monitoring from the data pipeline to the model runs and outputs

As well as supporting the team responsibilities on:

• Work to Extract, Load & Transform (ELT) data sets from a variety of data sources across the OfS enterprise technology stack. With a particular focus to the Extract & Load parts.
• Monitoring the execution of data workflows, including identifying and mitigating risks, setting service level indicators and configuring alerts.
• Adopting data governance best practice when processing raw data to develop, test and maintain datasets, including the use and maintenance of relevant resources (such as, but not limited to, a data catalogue, data dictionary, logical data models).
• Developing coding standards for Python programming to be used across the Data function.
• Supporting continuous data quality improvements across the organisation by exploring enhancements to: automated quality assurance processes; the reliability of data workflows; and the effectiveness of data transformations.
• Applying dimensional data modelling concepts and practices across the OfS data to develop conceptual, logical and physical data models that support efficiency and effectiveness of business operations
• Advocating and supporting an evidence-led culture across the organisation, assisting with driving enhancements to the organisation’s data capability

Essential experience & skills

• Strong Python programming knowledge, ideally PySpark
• Strong knowledge of the Azure Databricks platform and associated functionalities
• Adaptable, with a willingness to work flexibly as the needs of the organisation evolve
• Working well within a team, and able to work closely with internal and external stakeholders
• An ability to take a logical and analytical approach, and to take a pragmatic, collaborative approach to solving problems
• Adept at communicating technical concepts to a nontechnical audience
• Awareness of the modern data stack and associated methodologies

Working for us

The OfS regulates the higher education sector on behalf of all students. We value diversity and the wealth of perspectives, experience and ideas that it brings to our work, and we strive to embed equality of opportunity in everything we do.

We recruit based on fair and open competition and welcome applications from candidates regardless of age, disability, race and ethnicity, gender reassignment, marriage and civil partnership, pregnancy and maternity, religion or belief, sex or sexual orientation.

Flexibility is important to the OfS, as we recognise that it can enable people to work more effectively. A set of organisation-wide principles shape our approach to flexible and hybrid working, which our teams then use as guidance on how they deliver their business goals. This would usually mean working from the office for two days in a typical week.

To attract the widest possible field of qualified applicants, flexible working, for example job share and part time working apply to this role.

Important dates

Closing date for applications: 8th September 2025

Interviews: From 15 September 2025

Seniority level

• Mid-Senior level

Employment type

• Other

Job function

• Other

Industries

• Higher Education and Public Policy Offices

Note: This refined description removes irrelevant boilerplate and reorganises content for clarity while preserving original information.