Senior Data & AI Platform Engineer II

StepStone Group; London Borough of Southwark; On-site; GBP 90,000 - 140,000; Full time

Benefits offered by this job

29 days holiday

Private medical

Pension up to 10%

Employee Assistance Programme

Life Assurance

Cycle to work

Dog-friendly office

Job summary

The Stepstone Group is seeking a Senior Software Engineer II for the Data Platform to provide technical leadership and shape architecture across data, platform and infrastructure domains.

You will collaborate with Engineering Managers, Product Managers, and Enterprise Architects to deliver scalable data capabilities that support analytics and AI-driven products while driving observability and engineering excellence.

Qualifications

• Proven experience as a Senior Software Engineer or Senior Data Engineer in an equivalent senior individual contributor role.
• Strong background in data platforms, distributed systems and cloud-native architectures, with experience shaping technical direction across teams and stakeholders.
• Solid expertise in AWS, Terraform, Kafka/MSK, Airflow, Glue or Spark, and building scalable data solutions.
• Experience with modern data ecosystems including data lakes/lakehouse architectures, Iceberg or similar table formats, as well as batch and streaming processing
• Knowledge of data quality, governance, cataloguing and observability tools (e.g. Datadog), with DBT or AI-assisted engineering practices as a plus

Responsibilities

• Define and evolve the technical vision, roadmap and architecture of the Data Platform, driving decisions across data, platform and infrastructure domains.
• Establish engineering standards, best practices and reusable patterns while partnering with Enterprise Architecture and influencing technical direction.
• Drive engineering excellence by improving code quality, testing, observability, reliability and operational practices.
• Support end-to-end delivery by guiding teams through complex technical challenges, improving decision-making, and contributing to planning and risk management.
• Mentor engineers, grow senior technical capabilities and champion AI-assisted development and engineering automation.

Skills

AWS

Terraform

Kafka/MSK

Airflow

Spark

Data lakes

Lakehouse

DBT

Observability

AI-assisted engineering

Tools

Datadog

Job description

The Stepstone Group is seeking a Senior Software Engineer II for the Data Platform to provide technical leadership and shape architecture across data, platform and infrastructure domains.

You will collaborate with Engineering Managers, Product Managers, and Enterprise Architects to deliver scalable data capabilities that support analytics and AI-driven products while driving observability and engineering excellence.