Site Reliability Engineer

PCI Pal; Greater London; On-site; GBP 70,000 - 90,000; Full time

Job summary

PCI Pal is seeking a Site Reliability Engineer to join our UK-based DevOps team. You will own the observability and reliability framework, primarily DataDog, and partner with development teams to improve monitoring, alerting and resilience across infrastructure and telephony.

This role involves reducing toil, managing capacity and DR readiness, mentoring the team, and guiding progressive delivery practices such as canary deployments and controlled rollouts to protect availability as the system

Qualifications

• Hands-on experience with DataDog (or equivalent like New Relic, Grafana/Prometheus, Dynatrace).
• Experience defining SLIs/SLOs and alerting.
• Experience monitoring networking infrastructure and telemetry.
• Experience with capacity planning and redundancy design.
• Strong incident response and automation skills.

Responsibilities

• Own the observability framework across infrastructure, networking and telephony, and define standards.
• Control observability costs and optimize usage of metrics, logs, and APM.
• Reduce alert noise while ensuring incidents are surfaced.
• Mentor the DevOps team on reliability and observability practices.
• Automate incident response workflows and release processes.
• Drive resiliency improvements including redundancy and DR readiness.
• Lead on incident response and postmortems for lasting fixes.
• Collaborate with development teams to embed resiliency into releases (canary deployments, staged rollouts, rollback).
• Engage with teams to influence monitoring maturity across the engineering function.

Skills

DataDog

Observability

New Relic

Grafana/Prometheus

Jenkins

Jira

Canary deployments

Incident response

SLOs/SLIs

Tools

DataDog

New Relic

Grafana/Prometheus

Dynatrace

Jenkins

Jira

Job description

PCI Pal is a leading provider of SaaS solutions that empower companies to take payments securely, adhere to strict industry governance, and remove their business from the significant risks posed by non‑compliance and data loss. We are integrated and resold by some of the worlds’ leading business communications vendors, as well as major payment service providers.

We are currently looking for a Site Reliability Engineer to join our UK team.

Location:

This role will require monthly travel to our London and Ipswich office

The opportunity:

You'll join PCI Pal's DevOps team as the go-to authority on observability and reliability, owning our DataDog implementation and extending monitoring, alerting and anomaly detection across infrastructure, networking and telephony. Working closely with the senior DevOps engineer and the wider team, you'll drive down manual toil, improve resiliency, and own capacity planning, availability targets and disaster recovery readiness as load grows. You're joining an existing team and will be relied on to raise the reliability and observability bar for the whole group.

You will be responsible for:

• Own the observability framework and standards across infrastructure, networking and telephony, defining what good monitoring and alerting look like, and partnering with development teams to implement against these standards, including a practical monitoring scorecard, reviewing and signing off on their coverage
• Control observability costs - right-sizing usage (hosts, custom metrics, log ingestion/retention, APM) and making deliberate cost-vs-coverage tradeoffs
• Work with teams to reduce alert noise while making sure genuine incidents are still surfaced immediately
• Act as the technical authority on monitoring maturity across the engineering function
• Mentor and upskill the wider DevOps team on reliability and observability practices, working alongside the senior DevOps engineer
• Identify and automate away manual toil, including incident response workflows, compliance evidence collection and release processes
• Drive resiliency improvements across the infrastructure estate - redundancy, failover and capacity planning
• Lead on incident response and resolution, contributing to postmortem practices that turn failures into lasting fixes
• Work with development teams to embed resiliency practices into release processes, e.g. progressive rollouts, canary deployments and fast rollback, so changes can ship quickly without risking availability

Job requirements:

• Extensive, hands-on experience with DataDog (or equivalent tools such as New Relic, Grafana/Prometheus, Dynatrace)
• Experience managing and controlling observability tooling costs at scale
• Solid experience monitoring networking infrastructure (latency, packet loss, device health, anomaly detection)
• Experience with capacity planning, demand forecasting, and redundancy/failover design
• Strong background in incident response practices and building automation around them
• Experience defining SLIs/SLOs and error-budget-based alerting
• Solid software engineering skills, writing maintainable, testable, version-controlled automation and tooling
• Experience implementing progressive delivery practices (canary deployments, staged rollouts, automated rollback)
• Confidence engaging with and influencing development teams
• Experience of PCI compliance or other similar compliance frameworks
• Experience with automated build systems (e.g. Jenkins), work management systems (e.g. Jira), and source control

Nice to haves:

• Experience with telephony/voice infrastructure monitoring
• Experience mentoring or upskilling other engineers