Lead Data Engineer

National Grid; Warwick; Hybrid; GBP 60,000 - 90,000; Full time

Job summary

National Grid in Warwick is seeking an Associate Data Developer to partner with Electricity Transmission, translating business challenges into data products and dashboards on cloud platforms (Azure, Snowflake).

You will own ETL development, CI/CD pipelines, testing, deployment, and governance, while collaborating with ET teams and stakeholders to drive performance and actionable insight across the software lifecycle.

Job description

About Us

At National Grid, we keep people connected and society moving. But it’s so much more than that. National Grid provides the environment, support, and purpose that enable us to make real impact. As we drive momentum in the energy transition for all, we’re committed to ensuring none of our customers are left in the dark.

About Us

At National Grid, we keep people connected and society moving. But it’s so much more than that. National Grid provides the environment, support, and purpose that enable us to make real impact. As we drive momentum in the energy transition for all, we’re committed to ensuring none of our customers are left in the dark.

This role operates under a hybrid working model and is based out of the Warwick, UK office. Hybrid work expectations are subject to business needs and company policy.

About The Role

To work with Electricity Transmission to deliver actionable insight that drives business performance. Collaborating with key stakeholders the Associate Data Developer will work to understand key business challenges, help determine the most impactful insight and translate this into business requirements. They will support the management of insight delivery across the full software lifecycle — including development, testing, CI/CD and deployment, consider the business readiness for insight implementation and undertake necessary engagement to ensure insights drive action and are continuously improved. The role combines strong analytical capability with practical experience of building data products and dashboards on cloud platforms (particularly Azure).

What You'll Do

• Understanding of data fabric, lake and warehousing principles and full project involvement in one or more major technology platforms, e.g. Snowflake, Azure etc.
• Development of the Extract Transform Load (ETL) for the relevant data insights platform
• Working in partnership with ET teams, across the relevant areas of ET data to undertake advanced data mining and manipulation to understand the quality and completeness of data to advise and define key changes to technology architecture & data models to facilitate performance measurement & business insight.
• Lead the development of new installations and patches for the relevant insight platform and support upstream system developers and downstream stakeholders
• Support the data flow and mapping for solution design of IT Projects
• Produce support documentation ranging from requirement capture to end delivery
• Transform the quality of ET’s recorded information so it is universally recognised as the single source of the truth which is consistently utilised by the business without the need for extraction from core systems to cleanse before use
• Design, implement and maintain CI/CD pipelines, automation and test frameworks for data pipelines, models and dashboard deployments (e.g. Azure DevOps, GitHub Actions, or equivalent).
• Follow and promote good software lifecycle practices: version control, code review, automated testing, release management and rollback processes.
• Implement monitoring, logging and observability for data pipelines and dashboards (for example Application Insights, logging frameworks and pipeline metrics) and act on alerts.
• Work with ML/agentic automation teams to integrate or evaluate agent-based workflows where applicable (e.g. controlled LLM agents, orchestration of multi-step automated tasks).
• Apply ORM and database design principles (entity modelling, relationships, normalization, indexing, ORMs such as SQLAlchemy / Entity Framework where used) when designing data models and mappings.
• Ensure data quality and data contract validation are embedded within pipelines (unit tests, schema checks, monitoring) and work with stakeholders to remediate upstream issues.
• Continue existing accountabilities around data fabric/lake/warehouse design, ETL/ELT development, stakeholder engagement, documentation and promoting single-source-of-truth usage across the business.
• Provide day-to-day operational support for AI agents and associated automation capabilities.
• Monitor AI agent performance, availability, accuracy, and user adoption metrics.
• Investigate, troubleshoot, and resolve issues impacting AI agent functionality and outcomes.
• Collaborate with business stakeholders to identify opportunities for AI-driven process improvements.
• Maintain AI agent knowledge bases, prompts, workflows, and configuration settings.
• Ensure AI agents operate in compliance with organisational governance, security, privacy, and regulatory requirements.
• Support testing, validation, and deployment of new AI agent features and enhancements.
• Analyse user feedback and performance data to drive continuous improvement and optimisation of AI solutions.
• Ensure AI agents deliver reliable, accurate, and business-aligned outcomes.
• Monitor and report on AI agent performance, adoption, value realisation, and risk management.
• Drive continuous improvement initiatives to enhance AI effectiveness and user experience.
• Support the safe and responsible deployment of AI capabilities across the organisation.

About You

• Demonstrable evidence of delivering business benefits through data products and dashboards, from requirements through to deployed solutions and operational support.
• Proven experience designing and operating CI/CD for data platforms and analytics applications (pipeline automation, test automation, release orchestration).
• Practical familiarity with software development life-cycle practices (version control, code review, unit and integration testing, release management).
• Experience or demonstrable understanding of agentic LLM or automation frameworks, or experience integrating automated agents safely into production workflows.
• Knowledge of ORM database concepts and experience applying them in data models and service layers.
• Strong analytics and dashboarding experience (Power BI required; experience producing actionable dashboards and optimising UX for decision making).
• Hands-on experience building data products using Snowflake and/or Azure (Synapse, ADLS, Azure SQL, Azure Data Factory) and operationalising these on Azure.
• Experience working with stakeholders across business, product and engineering to define KPIs, SLAs and data contracts.
• Familiarity with DBT, DataOps practices and modern transformation tooling is desirable.
• Experience supporting digital products, automation platforms, AI solutions, or enterprise applications.
• Understanding of Generative AI, Large Language Models (LLMs), AI agents, and conversational AI concepts.
• Strong analytical and problem-solving skills with the ability to diagnose complex issues.
• Experience gathering and translating business requirements into technical solutions.
• Knowledge of data governance, information security, and responsible AI principles.
• Ability to interpret performance metrics and utilise data to drive improvements.