Senior Production Engineer

Clear Street; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Clear Street is looking for a Production Engineer to own the reliability of our cloud-native platform. You will design monitoring, automate recovery, and partner with engineering and operations to reduce toil while increasing platform resilience.

You’ll balance on-call responsiveness with building scalable tooling and self-healing capabilities. You’ll work across teams to improve MTTR, implement robust observability with Datadog, and contribute to IaC and GitOps practices, ensuring safer

Qualifications

• Hands-on Python experience with automation-focused tooling.
• Experience in SRE/Production Engineering with production ownership.
• Proven automation and diagnostic tooling improving recovery times.
• Deep familiarity with cloud-native tech and failure modes.

Responsibilities

• Design and build monitoring and observability platforms to surface signals and reduce alert fatigue.
• Develop automation and self-healing capabilities to reduce MTTR.
• Analyze incidents to identify trends and prevent recurrence.
• Create runbooks and diagnostic tooling to scale platform capabilities.
• Collaborate with Platform Engineering on CI/CD and deployment safety.
• Contribute to SRE best practices and IaC adoption.

Skills

Python

SRE/Production Eng

Automation tooling

Kubernetes

Observability (Datadog)

Terraform

GitOps (ArgoCD/GitHub Actions)

Java

Go

Kafka

Redis

Snowflake

Postgres

Communication

Tools

Datadog

Terraform

ArgoCD

GitHub Actions

Job description

Clear Street’s mission is to give every sophisticated investor access to every asset, in every market, through a unified platform built for speed, transparency and scale.

We give our clients the technology, tools, and service once reserved for the largest institutions, rebuilt with modern infrastructure. Our single, cloud-native, end-to-end capital markets platform powers investor growth today and is transforming how they can interact with markets tomorrow.

As a Production Engineer, you sit at the intersection of software reliability and operational excellence. You own the health, resilience, and recovery of our production systems—while spending equal energy innovating solutions that eliminate human toil, reduce incident blast radius, and raise the reliability bar across the entire platform. You will partner closely with engineering, operations, and business teams to understand daily pain points and translate them into lasting automated solutions. Half your time is spent in the trenches—supporting production, responding to incidents, and deeply understanding how our systems behave under real conditions. The other half is yours to build: automation, tooling, and observability platforms that make tomorrow's on-call shift meaningfully easier than today's.

You will work on challenges like:

• Design and build comprehensive monitoring and observability platforms that surface the right signal at the right time—eliminating alert fatigue and accelerating root-cause analysis.
• Develop intelligent automation and self-healing capabilities that diagnose issues, trigger recovery workflows, and reduce mean time to recovery (MTTR) without manual intervention.
• Analyze incidents, identify systemic trends, and engineer solutions that prevent entire classes of failures from recurring.
• Build reusable runbooks, diagnostic tooling, and recovery playbooks that turn tribal knowledge into scalable platform capabilities.
• Create golden-path operational workflows—making the safest, most reliable path also the easiest one for engineering teams to follow.
• Partner with Platform Engineering to influence CI/CD pipelines, deployment safety, and infrastructure resilience from a production reliability perspective.
• Champion Infrastructure as Code, GitOps, and SRE best practices while helping teams adopt modern engineering workflows.
• Continuously measure production health through SLIs/SLOs/SLAs, and drive engineering priorities based on reliability data.
• Explore emerging technologies—including AI-assisted diagnostics and developer tooling— that transform how we operate production systems.

The Team

We believe resilient systems are built by engineers who understand them end to end. Our Production Engineering team is the first and last line of defense for our production platform. We treat reliability as a product, with uptime and engineer experience as our north stars. We combine the discipline of SRE with a builder's mindset: when we see a recurring problem, we build a solution—not a workaround.

You will work across every engineering and operations team to understand failure modes, quantify reliability gaps, and build platform capabilities that scale with the organization. Whether it's reducing MTTR from hours to minutes, building self-service diagnostic tools, or designing proactive alerting that catches issues before customers notice, your work will have immediate, measurable impact.

If you're passionate about making production systems invisible to end users—and you get energy from both firefighting and building the systems that make fires less likely—you'll thrive here.

What We're Looking For

We're looking for engineers who combine operational instinct with a builder's discipline.

You should have:

• Strong hands-on Python skills—this is your primary language for automation and tooling.
• Experience in SRE, Production Engineering, Platform Engineering, or a related discipline with direct production ownership.
• Proven track record of building automation and diagnostic tooling that improved recovery times or reduced operational toil.
• Deep familiarity with cloud-native technologies—Kubernetes, containers, distributed systems and how they fail in production.
• Experience with observability platforms such as Datadog, and a strong intuition for what " good" monitoring looks like.
• Exposure to Infrastructure as Code (Terraform) and GitOps-based deployment workflows (ArgoCD, GitHub Actions, or similar).
• Familiarity with the broader technology stack: Java, Go, Kafka, Redis, Snowflake, and Postgres.
• Strong analytical and problem-solving skills—you thrive on ambiguous, high-stakes production problems.
• A product mindset applied to operational tooling: you think about usability, adoption, and documentation when building internal solutions.
• Excellent communication skills and the ability to work fluidly across engineering, operations, and business stakeholders.
• Self-starter mentality—you identify opportunities, take initiative, and deliver with minimal supervision.
• Curiosity and a continuous learning mindset; fintech or financial industry background is a plus.

The Technology You'll Work With

You'll operate and build on a modern cloud-native platform that includes:

What Success Looks Like

Within your first year, you'll have made a measurable impact on production reliability. Success l ooks like:

• Reducing mean time to detection (MTTD) and mean time to recovery (MTTR) across key production systems.
• Building automation that handles a meaningful percentage of incident scenarios without human intervention.
• Becoming a trusted subject matter expert for core platform components and their failure modes.
• Delivering observability and diagnostic tools that other engineers actually use and depend on.
• Establishing SLO baselines and driving engineering investment based on reliability data.
• Spending less of your time—and your teammates time—on repetitive manual toil.

Your impact won't be measured by the number of incidents you respond to—it will be measured by how reliably our systems run and how quickly we recover when they don't.