Observability Engineer - Assistant Vice President

Citibank (Switzerland) AG; Greater London; Hybrid; Confidential; Full time

Benefits offered by this job

Annual leave 27d

Discretionary bonus

Medical & life insurance

Employee assistance program

Pension plan

Parental leave

Employee discounts

Learning resources

Job summary

Citi is seeking a Site Reliability Engineer - Assistant Vice President in London to drive end‑to‑end observability, migrate legacy monitoring to Google Cloud Observability and Grafana, and implement OpenTelemetry instrumentation across OpenShift/Kubernetes environments.

The role emphasizes hands‑on deployment, automation (Ansible/Terraform), and collaboration with application teams to ensure resilience and performance in a highly regulated environment.

Qualifications

• Significant professional experience in software development with a strong focus on Site Reliability Engineering and Observability.

Responsibilities

• Culture & Strategy: Foster transparency, innovation, and technical excellence to drive automation.

Skills

SRE concepts

Observability

Container orchestration

Helm charts

IaC

Agile methodologies

Cloud platforms

Programming proficiency

Collaboration

Tools

OpenTelemetry

Grafana

Prometheus

Kubernetes

OpenShift

Terraform

Ansible

Job description

The Site Reliability Engineer (SRE) - Assistant Vice President is a technical professional responsible for the hands‑on execution, technical implementation, and deployment of SRE and observability principles in a complex, critical, and large-scale multi-disciplinary environment. In this role, you will apply a deep understanding of multiple technology domains to enhance the reliability, performance, and efficiency of our Applications and Services. As a key technical contributor, you will drive the migration of applications from existing monitoring tools (Geneos ITRS, Prometheus, ELK, Splunk, AppDynamics, etc.) to Google Cloud Observability (GCO) and Grafana using OpenTelemetry (OTel) as the instrumentation standard. You will act as a hands‑on technical authority, authoring reusable deployment solutions, configuring telemetry collectors, and providing direct technical onboarding support to application teams. We are seeking a passionate and experienced SRE to join our Production Management team. In this role, you will be instrumental in executing our strategy for end‑to‑end observability and resiliency, collaborating across the organization to ensure our services are stable, scalable, and fault‑tolerant.

Key Responsibilities

• Culture & Strategy Foster Technical Excellence: Foster a culture of transparency, innovation, and technical excellence that encourages continuous improvement and automation.
• Collaborative Enablement: Partner with development and SRE teams to drive the adoption of OpenTelemetry (OTel) and Google Cloud Observability (GCO) and Grafana standards.
• Regulatory Compliance: Operate effectively within a highly regulated environment, ensuring all observability and deployment solutions comply with relevant enterprise standards and security requirements.
• Resiliency & Recovery Resiliency Patterns: Partner with development teams to leverage cloud‑native services and implement established resiliency patterns to enhance application reliability.
• Recovery Testing Support: Support and participate in advanced recovery testing, including Production Swing Tests, Data Recovery Tests, and chaos engineering practices.
• Automation Drive: Drive the adoption and development of automation solutions (such as Ansible playbooks and Terraform) to minimize recovery time and streamline infrastructure deployments.
• Observability & Migration Execution Legacy Tool Migration: Actively execute the migration of applications from legacy monitoring tools (Geneos ITRS, GEM, ELK, Splunk, AppDynamics) to GCO and Grafana.
• Telemetry Configuration: Develop, test, and maintain OpenTelemetry Collector (or Grafana Alloy) configurations for metrics, logs, and traces.
• Reusable Deployment Solutions: Author and maintain reusable deployment solutions, including Helm charts and Ansible playbooks for deploying observability agents across OpenShift/Kubernetes and VM environments.
• Application Instrumentation: Partner with application development teams to effectively instrument applications using OpenTelemetry (OTel) standards, providing deep insights into system health and performance.
• GCO Feature Implementation: Aid with the design of GCO dashboards, log‑based metrics, alerts, and SLO/SLI tracking to provide comprehensive visibility.

Essential Skills

• Strong understanding of SRE concepts, including SLOs, SLIs, error budgets, and toil reduction.
• Observability Instrumentation: Hands‑on experience with OpenTelemetry (or Grafana Alloy) collectors.
• Container Orchestration: Proficiency in deploying, managing, and troubleshooting applications on OpenShift/Kubernetes.
• Helm Chart Management: Experience creating, modifying, and managing Helm charts for application deployment.
• Infrastructure as Code (IaC): Experience with IaC, configuration management, and automation tools (e.g., Ansible, Terraform).
• Modern Observability Tools: Hands‑on experience with modern observability tools (e.g., Prometheus, Grafana, Loki, Mimir, Tempo, AppDynamics, Splunk, or Google Cloud Observability).

Desired Skills

• Cloud Platforms: Experience with major public cloud providers, such as Google Cloud Platform (GCP) and Amazon Web Services (AWS).
• Legacy Migration Experience: Proven experience migrating applications from legacy monitoring tools (Geneos ITRS, GEM, ELK, Splunk, AppDynamics).
• Programming Proficiency: Experience writing or maintaining code in Java, Python, Go, or similar languages.
• Agile Frameworks: Proven experience delivering software and infrastructure using Agile frameworks.
• Collaboration & Communication: Strong communication and collaboration skills to work effectively with application teams, SRE leads, and stakeholders.

Qualifications Significant professional experience in software development, or an equivalent field, with a strong focus on Site Reliability Engineering and Observability. Expertise in analyzing complex application, database, network, and OS issues within large-scale, customer‑facing systems. A service‑oriented attitude combined with excellent problem‑solving and strategic thinking skills. Strong communication and diplomacy skills, with a proven ability to work effectively across multiple business and technical teams.

What we’ll provide you:

• 27 days annual leave (plus bank holidays)
• A discretional annual performance related bonus
• Private Medical Care & Life Insurance
• Employee Assistance Program
• Pension Plan
• Paid Parental Leave
• Special discounts for employees, family, and friends
• Access to an array of learning and development resources

Alongside these benefits Citi is committed to ensuring our workplace is where everyone feels comfortable coming to work as their whole self, every day. We want the best talent around the world to be energized to join us, motivated to stay and empowered to thrive.

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law. If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi. View Citi’s EEO Policy Statement and the Know Your Rights poster.

#LI-BH3

Job Family Group: Technology

Job Family: Applications Support

Time Type: Full time

Most Relevant Skills Please see the requirements listed above.

Other Relevant Skills For complementary skills, please see above and/or contact the recruiter.