Site Reliability Engineer

PCI Pal; Ipswich; Hybrid; GBP 65,000 - 90,000; Full time

Benefits offered by this job

25 days holiday rising to 28

Medical, dental and optical insurance

Electric Vehicle Scheme incentive

Work from anywhere 2 weeks per year

Option to work in Ipswich office or at

Training and development opportunities

Job summary

PCI Pal is seeking a DevOps reliability-focused engineer to own our observability program, including DataDog implementation and cross‑infrastructure monitoring. You will work with the team to reduce toil, improve resiliency, and own capacity planning as demand grows, shaping the overall reliability bar for the group.

You will collaborate with development teams, define SLIs/SLOs, implement progressive delivery practices, and mentor others while ensuring PCI-like compliance awareness across

Qualifications

• Extensive hands-on DataDog experience.
• Experience monitoring networking infrastructure and latency.
• Experience defining SLIs/SLOs and alerting with error budgets.
• Experience capacity planning and redundancy/failover design.

Responsibilities

• Own the observability framework and standards across infrastructure, networking and telephony, defining what good monitoring and alerting look like, and partnering with development teams to implement against these standards.
• Control observability costs - right-sizing usage of hosts, metrics, logs and APM, with cost-vs-coverage tradeoffs.
• Reduce alert noise while ensuring genuine incidents are surfaced immediately.
• Act as the technical authority on monitoring maturity across the engineering function.
• Mentor and upskill the wider DevOps team on reliability and observability practices.
• Automate away manual toil, including incident response workflows, evidence collection, and release processes.
• Drive resiliency improvements across the infrastructure estate - redundancy, failover and capacity planning.
• Lead on incident response and postmortem practices to derive lasting fixes.
• Embed resiliency into release processes with progressive rollouts, canaries and fast rollback.

Skills

DataDog

Observability tooling cost management

Networking monitoring

Capacity planning

Incident response

SLIs/SLOs

Automation & tooling

Progressive delivery

Influencing development teams

PCI compliance

CI/CD tooling

Jenkins

Jira

Git

Telephony/voice monitoring

Tools

Jenkins

Jira

Git

Job description

You’ll join PCI Pal’s DevOps team as the go-to authority on observability and reliability, owning our DataDog implementation and extending monitoring, alerting and anomaly detection across infrastructure, networking and telephony. Working closely with the senior DevOps engineer and the wider team, you’ll drive down manual toil, improve resiliency, and own capacity planning, availability targets and disaster recovery readiness as load grows. You’re joining an existing team and will be relied on to raise the reliability and observability bar for the whole group.

Job requirements

• Extensive, hands‑on experience with DataDog (or equivalent tools such as New Relic, Grafana/Prometheus, Dynatrace)
• Experience managing and controlling observability tooling costs at scale
• Solid experience monitoring networking infrastructure (latency, packet loss, device health, anomaly detection)
• Experience with capacity planning, demand forecasting, and redundancy/failover design
• Strong background in incident response practices and building automation around them
• Experience defining SLIs/SLOs and error‑budget‑based alerting
• Solid software engineering skills, writing maintainable, testable, version‑controlled automation and tooling
• Experience implementing progressive delivery practices (canary deployments, staged rollouts, automated rollback)
• Confidence engaging with and influencing development teams
• Experience of PCI compliance or other similar compliance frameworks
• Experience with automated build systems (e.g. Jenkins), work management systems (e.g. Jira), and source control

Nice to haves

• Experience with telephony/voice infrastructure monitoring
• Experience mentoring or upskilling other engineers

Job responsibilities

• Own the observability framework and standards across infrastructure, networking and telephony, defining what good monitoring and alerting look like, and partnering with development teams to implement against these standards, including a practical monitoring scorecard, reviewing and signing off on their coverage
• Control observability costs - right‑sizing usage (hosts, custom metrics, log ingestion/retention, APM) and making deliberate cost‑vs‑coverage tradeoffs
• Work with teams to reduce alert noise while making sure genuine incidents are still surfaced immediately
• Act as the technical authority on monitoring maturity across the engineering function
• Mentor and upskill the wider DevOps team on reliability and observability practices, working alongside the senior DevOps engineer
• Identify and automate away manual toil, including incident response workflows, compliance evidence collection and release processes
• Drive resiliency improvements across the infrastructure estate - redundancy, failover and capacity planning
• Lead on incident response and resolution, contributing to postmortem practices that turn failures into lasting fixes
• Work with development teams to embed resiliency practices into release processes, e.g. progressive rollouts, canary deployments and fast rollback, so changes can ship quickly without risking availability
• 25 days holiday, rising to 28 days per annum with length of service
• Medical, dental and optical insurance cover
• Option to either work in our Ipswich office, or from home (or both!)
• An exciting and flexible working environment surrounded by friendly and committed co‑workers
• Electric Vehicle Scheme incentive
• “Work from anywhere” 2 weeks per year policy
• Reward, benefits and wellbeing hub (offering support, discounts, cashback and savings)
• Training and development opportunities
• Ad‑hoc team events, incentives and competitions