Senior Data Platform Engineer - Python & AWS

Permutable; Greater London; Hybrid; GBP 90,000 - 120,000; Full time

Benefits offered by this job

Equity options

Hybrid flexibility

Job summary

Permutable is hiring an experienced Data Platform Engineer to build real-time data pipelines and own the AWS infrastructure that runs them. You'll design Python services to ingest and transform large data volumes, while scaling S3, RDS/PostgreSQL, Lambda, ECS and EKS for fast, reliable delivery to models and clients.

You'll work in a small, fast-moving team from our London hub, with three days on-site and flexible remote work.

Qualifications

• Strong production Python experience with building scalable data pipelines.
• Hands-on AWS experience: S3, RDS/PostgreSQL, Lambda, ECS or EKS.
• Experience designing and maintaining data platforms and pipelines for real-time workloads.
• Proficiency in SQL, PostgreSQL, data modelling and large dataset processing.
• Familiarity with IaC (Pulumi/Terraform) and CI/CD pipelines.

Responsibilities

• Design and develop Python services and pipelines to ingest, transform and deliver large data volumes.
• Build and scale AWS data platform infrastructure across S3, RDS/PostgreSQL, Lambda, ECS and EKS.
• Develop event-driven services using messaging and queues for reliable workflows.
• Improve data quality with validation, deduplication, retries and recovery.
• Own storage, querying and data serving for research and client apps.
• Collaborate with AI/ML teams to provide data inputs and model outputs.
• Automate infrastructure and deployments with Pulumi and GitHub Actions.
• Monitor, diagnose and optimize pipeline performance and costs.

Skills

Python

AWS

Data pipelines

SQL

PostgreSQL

Kubernetes

Terraform

CI/CD

Linux

Pulumi

Education

Bachelor's degree in computer science or related field

Tools

Pulumi

GitHub Actions

Terraform

Job description

At Permutable, we're building real-time AI systems that turn large volumes of global news, market and proprietary data into market intelligence and systematic signals for financial institutions. We're looking for an experienced Data Platform Engineer with strong Python and AWS skills to build and scale the systems behind our data ingestion, processing, storage and delivery. You'll write production Python, develop reliable data pipelines and own the AWS infrastructure that runs them. As we move towards a more real-time, event-driven architecture, you'll help shape how data flows through our platform and reaches our models and clients. Working directly with our founder and engineering team, you'll have significant responsibility for architecture, implementation and production performance.

Build production data pipelines: Design and develop Python services and pipelines to ingest, validate, transform and deliver large volumes of news, market and proprietary data.

Develop our AWS data platform: Build and scale infrastructure across S3, RDS/PostgreSQL, Lambda, ECS and EKS, choosing the right services for each workload.

Build event-driven systems: Develop services using messaging, queues and asynchronous processing to support faster data updates and reliable distributed workflows.

Improve data quality and reliability: Build validation, deduplication, schema handling, retries and recovery into our pipelines. Make it straightforward to identify missing data, investigate failures and reprocess historical datasets.

Own storage and data access: Improve how we store, query and serve data to support research, production models and client-facing applications.

Support AI and quantitative workloads: Work with AI engineers and researchers to provide reliable data inputs and integrate NLP, LLM and quantitative model outputs into the platform.

Automate infrastructure and deployments: Manage infrastructure through Pulumi and build CI/CD workflows using GitHub Actions.

Monitor and optimise production: Develop monitoring and alerting for pipeline health, data freshness and service performance. Diagnose production issues and improve processing speed, resilience and AWS costs.

Strong production Python experience: You can design, build, test and maintain substantial Python services and data pipelines.

Strong AWS experience: Hands-on experience building and operating production systems on AWS, including services such as S3, RDS/PostgreSQL, Lambda, ECS or EKS.

Experience building data platforms: A substantial track record in data platform engineering, data engineering or backend engineering, with ownership of production data systems.

SQL and database skills: Strong SQL and practical experience with PostgreSQL, data modelling, query performance and processing large datasets.

Event-driven and distributed systems experience: A solid understanding of messaging, queues, asynchronous processing and reliable workflows, including retries and failure recovery.

Infrastructure-as-Code and CI/CD: Experience with Pulumi, Terraform or equivalent, alongside automated testing and deployment pipelines.

Production ownership: Comfortable working with Linux, containers, monitoring and logging, and diagnosing problems across application code, pipelines and infrastructure.

Engineering judgement: Able to make practical architecture decisions and balance performance, reliability, cost and delivery speed.

Startup mindset: Proactive, resourceful and comfortable taking responsibility in a small, fast-moving engineering team.

Relevant technical background: A degree in computer science, software engineering or a related discipline, or equivalent practical experience.

Useful additional experience: Apache Airflow, real-time data processing, Kubernetes, AI/ML infrastructure, LLM workflows, GPU infrastructure or financial-market data.

Real responsibility: Own important parts of our data platform and see your work used in production.

An ambitious, collaborative culture: Work closely with our founder, engineers and researchers, with regular opportunities to contribute ideas and influence technical decisions.

A strong startup culture and office vibe: Join a close-knit team that makes time for regular socials, company offsites, sports events and office treats.

Benefits

• Hybrid flexibility: Spend three or more days each week in our Vauxhall hub, with flexibility to work remotely for the remainder.
• Share in Our Growth: Equity options give you the opportunity to share in Permutable's long-term success as you help build the business.