Observability & Tooling Engineer

Trust in SODA; United Kingdom; Remote; GBP 68,000 - 113,000; Part time

Job summary

Trust in SODA is seeking an Observability & Tooling Engineer for a fully remote UK role on a 6‑month contract. You will design and build tooling for large-scale infrastructure, focusing on observability, automation and reliability across distributed systems.

You will work with SRE/DevOps and Platform Engineering teams, delivering scalable tooling in Grafana/Prometheus, Ruby/Rails and Go, and maintain CI/CD pipelines with GitHub Actions.

Qualifications

• 6+ years of experience across Infrastructure Engineering, DevOps, SRE or Software Engineering.
• Hands-on experience with production-grade infrastructure tooling, automation or platform solutions.
• Strong experience with Grafana stack (Grafana, Prometheus, Loki, Mimir, Grafana Alloy).
• Proficient programming in Ruby/Rails and Go.
• Experience with Ansible and AWX for automation.
• Kubernetes in production environments.
• Solid understanding of monitoring, alerting and incident management.
• Knowledge of REST APIs and service-to-service communication.
• Experience building CI/CD pipelines, esp. GitHub Actions.
• Familiarity with SNMP and syslog telemetry.
• Certifications CK A and LPIC are a plus.
• Cloud-native observability technologies experience.

Responsibilities

• Design, develop and maintain internal infrastructure tooling and observability platforms across large-scale distributed environments.
• Build solutions transforming logs, metrics and events into actionable operational insights.
• Develop monitoring, alerting and incident response using Grafana stack.
• Automate infrastructure operations including provisioning, onboarding, inventory and lifecycle processes.
• Develop infrastructure automation workflows using Ansible and AWX.
• Create tooling for capacity planning, performance benchmarking and optimisation.
• Develop and integrate services using Ruby/Rails, Go and REST APIs.
• Maintain and improve CI/CD pipelines using GitHub Actions and self-hosted runners.
• Collaborate with SRE and Platform Engineering to deliver scalable, reliable infra.

Skills

Grafana

Prometheus

Loki

Mimir

Grafana Alloy

Ruby/Rails

Go

Ansible

AWX

Kubernetes

CI/CD

GitHub Actions

REST APIs

SRE/DevOps

Linux

Education

CKA

LPIC

Tools

Grafana

Prometheus

Loki

Mimir

Grafana Alloy

Ansible

AWX

Kubernetes

GitHub Actions

Job description

Observability & Tooling Engineer – Outside IR35 – Remote UK

Location: Fully Remote (UK)

Contract: 6 Months

Day Rate: Up to £615 per day (Outside IR35)

Start Date: ASAP

We are looking for an experienced Observability & Tooling Engineer to join a global infrastructure operations team on an initial 6-month contract.

This is an exciting opportunity to work within a large-scale, highly distributed infrastructure environment, developing the tooling, automation and observability solutions that underpin critical operations.

Working closely with SRE, DevOps and Platform Engineering teams, you will design and build scalable, production-ready systems that improve infrastructure reliability, performance and operational efficiency.

Key Responsibilities

• Design, develop and maintain internal infrastructure tooling and observability platforms across large-scale distributed environments.
• Build solutions that transform high-volume logs, metrics and events into actionable operational insights.
• Develop and enhance monitoring, alerting and incident response capabilities using the Grafana observability stack.
• Automate infrastructure operations, including provisioning, cluster onboarding, inventory management and lifecycle processes.
• Develop and maintain infrastructure automation workflows using Ansible and AWX.
• Build tooling to support capacity planning, performance benchmarking and infrastructure optimisation.
• Develop and integrate services using Ruby/Rails, Go and REST APIs.
• Maintain and improve CI/CD pipelines using GitHub Actions and self-hosted runners.
• Collaborate with SRE and Platform Engineering teams to deliver scalable, reliable infrastructure solutions.

Key Skills & Experience

• 6+ years of experience across Infrastructure Engineering, DevOps, SRE or Software Engineering.
• Strong experience developing production-grade infrastructure tooling, automation or platform solutions.
• Hands-on experience with Grafana, Prometheus, Loki, Mimir and Grafana Alloy.
• Strong programming experience in Ruby/Rails, Go or similar languages.
• Experience with Ansible and AWX for infrastructure automation.
• Practical experience working with Kubernetes in production environments.
• Strong understanding of infrastructure monitoring, alerting, incident management and operational reliability.
• Experience working with large-scale, distributed or enterprise infrastructure environments.
• Knowledge of REST APIs, system integrations and service-to-service communication.
• Experience building and maintaining CI/CD pipelines, ideally using GitHub Actions.
• Familiarity with infrastructure telemetry protocols such as SNMP and syslog.
• Experience supporting HPC environments, GPU infrastructure or large-scale compute platforms.
• Kubernetes certifications (CKA).
• Linux certifications (LPIC) or security qualifications.
• Additional experience with cloud-native observability technologies.

Why Apply?

• Up to £615 per day – Outside IR35
• Fully remote working across the UK
• Initial 6-month contract
• Opportunity to work on cutting-edge, large-scale GPU and HPC infrastructure.
• Work with modern observability, automation and platform engineering technologies.