Remote SRE: AI Infra Observability & Auto-Remediation

OpticoreIT; United States; Remote; GBP 70,000 - 110,000; Full time

Job summary

Opticore IT is seeking Site Reliability Engineers to stand up, tune and automate observability platforms for three AI infrastructure clusters. This fully remote role operates inside IR35 and offers exposure to proactive monitoring and automated remediation.

You will own Grafana, Loki, Mimir and VictoriaMetrics, design dashboards, automate health checks, and integrate monitoring with HaloITSM and vendor portals. Experience with GitOps, CI/CD and self-healing workflows is valued.

Qualifications

• Experience with observability stack Grafana, Loki, Mimir and VictoriaMetrics.
• Proficiency in Python or Go scripting and API automation.
• Experience integrating monitoring with HaloITSM, Webhooks or vendor portals.
• Knowledge of GitOps and CI/CD configuration management.
• Experience with proactive incident automation and self-healing.

Responsibilities

• Monitor the health of compute, GPU, storage and network infrastructure across clusters.
• Own and manage Grafana, Loki, Mimir, VictoriaMetrics and supporting telemetry platforms.
• Design and maintain dashboards, alerting rules and automated health checks.
• Analyse trends to identify degradation before it impacts services.
• Produce operational reporting, capacity forecasting and platform health metrics.
• Develop automated workflows that detect faults and trigger remediation actions.
• Integrate monitoring systems with HaloITSM and vendor support portals.
• Build auto ticketing, escalation and notification capabilities, including automated dispatch of field engineers.
• Reduce operational risk through self healing and proactive response mechanisms.
• Manage configuration and deployments through Git and CI/CD pipelines.
• Maintain runbooks and operational procedures, and continually improve tooling.

Skills

Observability stack

Python

Go

SRE automation

Incident response

Tools

Grafana

Loki

Mimir

VictoriaMetrics

HaloITSM

Job description

Opticore IT is seeking Site Reliability Engineers to stand up, tune and automate observability platforms for three AI infrastructure clusters. This fully remote role operates inside IR35 and offers exposure to proactive monitoring and automated remediation.

You will own Grafana, Loki, Mimir and VictoriaMetrics, design dashboards, automate health checks, and integrate monitoring with HaloITSM and vendor portals. Experience with GitOps, CI/CD and self-healing workflows is valued.