Observability Engineer - Assistant Vice President

Citi; Greater London; On-site; GBP 90,000 - 120,000; Full time

Job summary

Citi is seeking an experienced Site Reliability Engineer to lead end-to-end observability and resiliency initiatives in a large-scale environment. You will migrate monitoring tooling to Google Cloud Observability and Grafana, implement OpenTelemetry instrumentation, and author reusable deployment solutions for OpenShift/Kubernetes and VM environments.

The role requires deep SRE knowledge, strong collaboration with application teams, and a focus on reliability, performance, and automation within

Qualifications

• Significant professional experience in software development with a strong focus on Site Reliability Engineering.
• Expertise analyzing complex issues across large-scale systems (apps, DBs, OS, network).
• Strong communication and collaboration across multiple business and technical teams.

Responsibilities

• Foster a culture of transparency, innovation, and technical excellence with automation.
• Partner with development and SRE teams to drive OpenTelemetry and GCO adoption.
• Ensure observability solutions comply with enterprise security and regulatory standards.
• Leverage cloud-native services and resiliency patterns to improve reliability.
• Support advanced recovery testing, including chaos engineering and swing tests.
• Develop automation with Ansible and Terraform to minimize recovery time and streamline deployments.
• Migrate legacy monitoring tools to GCO and Grafana; configure OTEL collectors and dashboards.

Skills

SRE concepts

OpenTelemetry

Kubernetes/OpenShift

Helm charts

IaC (Ansible, Terraform)

Prometheus/Grafana

Cloud platforms (GCP/AWS)

Go/Java/Python

Grafana Alloy/OpenTelemetry collectors

Tools

Google Cloud Platform (GCP)

Grafana

OpenTelemetry

Geneos ITRS

Prometheus

Job description

The Site Reliability Engineer (SRE) - Assistant Vice President is a technical professional responsible for the hands-on execution, technical implementation, and deployment of SRE and observability principles in a complex, critical, and large-scale multi-disciplinary environment. In this role, you will apply a deep understanding of multiple technology domains to enhance the reliability, performance, and efficiency of our Applications and Services.

As a key technical contributor, you will drive the migration of applications from existing monitoring tools (Geneos ITRS, Prometheus, ELK, Splunk, AppDynamics, etc.) to Google Cloud Observability (GCO) and Grafana using OpenTelemetry (OTel) as the instrumentation standard. You will act as a hands-on technical authority, authoring reusable deployment solutions, configuring telemetry collectors, and providing direct technical onboarding support to application teams.

We are seeking a passionate and experienced SRE to join our Production Management team. In this role, you will be instrumental in executing our strategy for end-to-end observability and resiliency, collaborating across the organization to ensure our services are stable, scalable, and fault-tolerant.

Key Responsibilities

• Culture & Strategy Foster Technical Excellence: Foster a culture of transparency, innovation, and technical excellence that encourages continuous improvement and automation.
• Collaborative Enablement: Partner with development and SRE teams to drive the adoption of OpenTelemetry (OTel) and Google Cloud Observability (GCO) and Grafana standards.
• Regulatory Compliance: Operate effectively within a highly regulated environment, ensuring all observability and deployment solutions comply with relevant enterprise standards and security requirements.

Resiliency & Recovery

• Resiliency Patterns: Partner with development teams to leverage cloud-native services and implement established resiliency patterns to enhance application reliability.
• Recovery Testing Support: Support and participate in advanced recovery testing, including Production Swing Tests, Data Recovery Tests, and chaos engineering practices.
• Automation Drive: Drive the adoption and development of automation solutions (such as Ansible playbooks and Terraform) to minimize recovery time and streamline infrastructure deployments.

Observability & Migration Execution Legacy Tool Migration:

• Actively execute the migration of applications from legacy monitoring tools (Geneos ITRS, GEM, ELK, Splunk, AppDynamics) to GCO and Grafana.
• Telemetry Configuration: Develop, test, and maintain OpenTelemetry Collector (or Grafana Alloy) configurations for metrics, logs, and traces.
• Reusable Deployment Solutions: Author and maintain reusable deployment solutions, including Helm charts and Ansible playbooks for deploying observability agents across OpenShift/Kubernetes and VM environments.
• Application Instrumentation: Partner with application development teams to effectively instrument applications using OpenTelemetry (OTel) standards, providing deep insights into system health and performance.
• GCO Feature Implementation: Aid with the design of GCO dashboards, log-based metrics, alerts, and SLO/SLI tracking to provide comprehensive visibility.

Essential Skills

• 5+ Years of Experience: Deep understanding of SRE concepts, including SLOs, SLIs, error budgets, and toil reduction.
• Observability Instrumentation: Hands-on experience with OpenTelemetry (or Grafana Alloy) collectors.
• Container Orchestration: Proficiency in deploying, managing, and troubleshooting applications on OpenShift/Kubernetes.
• Helm Chart Management: Experience creating, modifying, and managing Helm charts for application deployment.
• Infrastructure as Code (IaC): Experience with IaC, configuration management, and automation tools (e.g., Ansible, Terraform).
• Modern Observability Tools: Hands-on experience with modern observability tools (e.g., Prometheus, Grafana, Loki, Mimir, Tempo, AppDynamics, Splunk, or Google Cloud Observability).

Desired Skills

• Cloud Platforms: Experience with major public cloud providers, such as Google Cloud Platform (GCP) and Amazon Web Services (AWS).
• Legacy Migration Experience: Proven experience migrating applications from legacy monitoring tools (Geneos ITRS, GEM, ELK, Splunk, AppDynamics).
• Programming Proficiency: Experience writing or maintaining code in Java, Python, Go, or similar languages.
• Agile Frameworks: Proven experience delivering software and infrastructure using Agile frameworks.
• Collaboration & Communication: Strong communication and collaboration skills to work effectively with application teams, SRE leads, and stakeholders.

Qualifications Significant professional experience in software development, or an equivalent field, with a strong focus on Site Reliability Engineering and Observability. Expertise in analyzing complex application, database, network, and OS issues within large-scale, customer-facing systems. A service-oriented attitude combined with excellent problem-solving and strategic thinking skills. Strong communication and diplomacy skills, with a proven ability to work effectively across multiple business and technical teams

Job Family Group:

Technology

Job Family:

Applications Support

Time Type:

Full time

Most Relevant Skills:

Please see the requirements listed above.

Other Relevant Skills:

For complementary skills, please see above and/or contact the recruiter.

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View Citi’s EEO Policy Statement and the Know Your Rights poster.