Senior Data Platform Engineer

Octoenergy; Greater London; On-site; GBP 90,000 - 120,000; Full time

Job summary

Octopus Energy Group seeks an engineering professional to design and implement enterprise-grade identity, access, and governance across human users, systems, and AI agents. You will build production-grade platform controls, IaC, and secure pipelines while collaborating with data engineers, application builders, and stakeholders across Finance, Asset Management, and Fund Operations.

You will harden API gateways, implement token exchange, and ensure full auditability.

Qualifications

• Federated identity mastery: design and implement enterprise-grade federated identity, token exchange, access control & SCIM provisioning.
• Platform foundations: hands-on IaC (Terraform) and production CI/CD pipelines with environment promotion and release controls.
• API gateways: harden patterns for authentication, routing and rate limiting.
• Cloud & data environments: experience across Azure, AWS or Databricks ecosystems.
• Pragmatic mindset: autonomous engineer; translate architecture into practical code and communicate trade-offs clearly.
• Experience with AI gateways, LLM proxies or model-routing patterns.
• Observability tools for AI systems, vector databases, RAG pipelines, or non-deterministic monitoring.
• Policy-as-code, action logging, or AI kill switches implementations.
• Direct experience with Okta, ZScaler, Entra ID, Unity Catalog, or M365 migration environments.
• Background in regulated or high-governance industries; FinOps practices like token budgeting and usage attribution.

Responsibilities

• Build and maintain IaC (Terraform) across dev, test, and prod environments.
• Strengthen CI/CD practices in GitHub Actions with repeatable deployment patterns and release controls.
• Support architecture decisions by translating designs into maintainable engineering patterns.
• Design federated identity architectures with Okta, Entra ID, and ZScaler including SCIM provisioning.
• Harden API gateway controls: authentication, routing, rate limiting, policy enforcement.
• Work across Azure, AWS, or Databricks environments to support platform governance.
• Develop practical code with clear trade-offs and documentation for delivery squads.
• Enablement & integration: deploy vector databases and align with Databricks Unity Catalog governance.

Skills

Federated identity mastery

Platform foundations

API gateways

Cloud & data environments

Pragmatic mindset

AI gateways / model routing

AI observability tools

Policy-as-code / action logging

Okta / Entra ID / ZScaler / Unity Cat.

M365 migration environments

FinOps / token budgeting

Tools

Terraform

GitHub Actions

Databricks AI Search

Unity Catalog

Okta

Entra ID

ZScaler

Job description

Help us make a big green dent in the universe. We're on a mission to drive the global renewable energy revolution, and we need many more people to help us make our vision a reality.

Octopus Energy Generation (OEGEN) is a leader in sourcing, investing in, and managing a wide range of utility-scale renewable energy assets. As we continue to expand our investment base, we are dedicated to building a diverse team that reflects the communities we serve. Currently, we manage a global portfolio of over £8 billion in renewable energy assets.

About the role:

As part of our mission to drive smarter energy decisions, we are building a production-grade, governed platform that empowers our teams to deploy AI-enabled tools, agents, and data products safely and at scale.

Sitting within the Systems and Data / Transformation team, you will turn existing platform architectures, governance specs, and prototype patterns into production-grade infrastructure. You’ll work alongside our Tech and Transformation Lead, data engineers, application builders, and business stakeholders across Finance, Asset Management, and Fund Operations.

This is not a general DevOps or blank-page architecture role. The primary focus is designing and implementing enterprise-grade identity, access, and governance controls across human users, systems, and non-human agents.

Our Tech & Ecosystem: Databricks, Lakebase, Unity Catalog, Entra ID / Azure Identity, Okta, ZScaler, GitHub Actions, MCP patterns, and Terraform.

Platform Engineering & CI/CD

• Build and maintain Infrastructure-as-Code (IaC) using Terraform across development, test, and production environments.
• Strengthen CI/CD practices in GitHub Actions, establishing repeatable deployment patterns, environment promotions, automated rollbacks, and release controls.
• Support architecture decisions by converting designs into maintainable, well-documented engineering patterns.

Identity, Access & Security

• Design and implement federated identity architecture across Okta, Entra ID, and ZScaler.
• Build identity, token exchange, and permission propagation models that work for both human users and non-human AI agents.
• Implement SCIM provisioning/deprovisioning and access models with single-revoke capability and full auditability.

AI Gateway & Governance

• Harden existing AI gateway and agent orchestration layers (built on MCP patterns) into secure, production-ready capabilities.
• Implement API gateway controls: authentication, routing, rate limiting, token budgets, policy-as-code, and action logging.
• Build platform-, tool-, and agent-level kill switches, alongside dry-run/safe-testing modes for agent workflows.

Observability & FinOps

• Implement AI system observability, including prompt logging, output monitoring, quality scoring, and drift detection.
• Establish operational monitoring for gateway usage, latency, error rates, and model routing logic.
• Enforce token budgets, cost alerts, and usage/cost attribution by team and use case.

Enablement & Integration

• Deploy vector database and retrieval infrastructure (e.g., Databricks AI Search) to support Document AI and RAG use cases.
• Align platform controls with Databricks and Unity Catalog governance patterns.
• Provide clear documentation, reusable patterns, and self-service onboarding guidance so delivery squads can deploy tools without bespoke manual intervention.

Required Experience & Skills

• Federated Identity Mastery: Proven experience designing and implementing enterprise-grade federated identity, token exchange, permission propagation, SCIM provisioning, and access lifecycle management.
• Platform Foundations: Strong hands-on experience with IaC (Terraform) and production CI/CD pipelines (environment promotion, rollback, release controls).
• API Gateways: Experience hardening API gateway patterns (authentication, routing, rate limiting, access policy enforcement).
• Cloud & Data Environments: Technical comfort working across Azure, AWS, or Databricks platform ecosystems.
• Pragmatic Mindset: Highly autonomous engineer comfortable in a lean team; skilled at translating architectural specs into practical code and communicating trade-offs clearly in plain English.
• Experience with AI gateways, LLM proxies, or model-routing patterns.
• Exposure to AI observability tools, vector databases (Databricks AI Search), RAG pipelines, or non-deterministic system monitoring.
• Hands-on experience implementing policy-as-code, action logging, or AI kill switches.
• Direct experience with Okta, ZScaler, Entra ID, Unity Catalog, or M365 migration environments.
• Background in regulated or high-governance industries (Financial Services, Energy, Infrastructure Investment, Fintech, or Enterprise SaaS).
• Familiarity with FinOps practices (token budgeting, usage tracking, cost attribution).

Octopus Energy Group is aunique culture. An organisation where people learn, decide, and build quicker. Where people work with autonomy, alongside a wide range of amazing co-owners, on projects that break new ground. We want your hard work to be rewarded with perks you actually care about! We were recently named the UK's top company to work for, and we ranked in the top ten in the Sunday Times Best Places to Work 2024. Our Group CEO, Greg has recorded a podcast about our culture and how we empower our people. We’ve also been placed in the top 10 companies for senior leadership.

Our process usually takes up to 4 weeks, but we’ll always do our best to flex around what works for you. Along the way, you’ll chat with our recruitment team and your Recruiter will help you throughout different stages. Got any burning questions before then? Drop us a message at [e-mail] and we’d love to help!

Are you ready for a career with us? We want to ensure you have all the tools and environment you need to unleash your potential. Need any specific accommodations? Whether you require specific accommodations or have a unique preference, let us know, and we'll do what we can to customise your interview process for comfort and maximum magic!

Studies have shown that some groups of people, like women, are less likely to apply to a role unless they meet 100% of the job requirements. Whoever you are, if you like one of our jobs, we encourage you to apply as you might just be the candidate we hire. Across Octopus, we're looking for genuinely decent people who are honest and empathetic. Our people are our strongest asset and the unique skills and perspectives people bring to the team are the driving force of our success. As an equal opportunity employer, we do not discriminate on the basis of any protected attribute. Our commitment is to provide equal opportunities, an inclusive work environment, and fairness for everyone.