GenAI L3 Production Reliability Engineer

United Overseas Bank Limited (UOB); Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

United Overseas Bank Limited (UOB) is recruiting an incident-focused Software Engineer/SRE in Singapore to own L3 GenAI production incident response, debug LangChain and Elasticsearch issues, implement safe hotfixes, and collaborate with business units to ensure reliable services. You will enhance observability, runbooks, and post-incident reviews while working with OpenShift, GPUs, and NVIDIA AI services.

The role emphasizes security, data handling, and proactive reliability improvements across

Qualifications

• 3 to 4 years of software engineering or SRE with at least one year supporting AI or data intensive services in production.
• Strong Python. Comfortable reading and fixing LangChain code, FastAPI or Flask services, async patterns and task queues.
• Hands on with LangChain in real projects. Chains, tools, retrievers, memory and callback handlers.
• Elasticsearch proficiency. Query DSL, relevance tuning, index lifecycle, scaling, snapshots and Kibana for triage.
• Redis proficiency. Caching patterns, pub or sub, streams, eviction, persistence and troubleshooting latency or timeouts.
• Safety and guardrails experience. Guardrails AI configuration, schema and validator design, LlamaGuard or similar classifiers in the loo
• CI or CD with Jenkins. Declarative pipelines, approvals, artifacts, environment promotion and rollback strategy
• Observability depth. Prometheus metrics, Grafana dashboards, OpenTelemetry traces and logs.
• Able to decide which signals matter and set actionable alerts
• Solid Git. Branching, pull requests, code reviews and release tagging. Comfortable with feature flags and canary or blue green patterns
• Working knowledge of Kubernetes and OpenShift as a strong plus. Debugging pods, logs, events and basic resource tuning
• Familiarity with LLM serving stacks such as vLLM or NVIDIA Enterprise AI is a plus. Know how to read model server logs, timeouts and token throughput
• Data handling discipline. Understanding of PII, masking, prompt redaction and safe logging
• Clear communicator who can write crisp incident updates, RCAs and user facing notes

Responsibilities

• Own L3 incident response for GenAI use cases in production. Triage, deep debug, fix or work with the vendor to resolve defects and improve reliability
• Review, reproduce and root cause complex bugs across LangChain flows, Elasticsearch queries, Redis caches and safety layers such as Guardrails AI and LlamaGuard
• Implement safe changes and hotfixes. Build, test and deploy code and config updates through Jenkins with proper approvals and rollbacks
• Partner with business users to understand issue impact and edge cases. Translate vague problem reports into actionable steps and clear acceptance criteria
• Harden use case pipelines. Add input validation, timeouts, retries, circuit breakers and fallback strategies to reduce customer facing impact
• Improve RAG quality for stability. Tune chunking, retrieval parameters, query rewriting and caching to reduce latency and errors
• Maintain and evolve runbooks, playbooks and knowledge articles. Keep them current and prove they work with regular game days
• Drive observability for the use cases. Define golden signals, add OpenTelemetry traces, wire up Prometheus metrics and curate Grafana dashboards
• Manage production hygiene. Track error budgets, SLOs and SLAs. Push for defect burn down and change quality
• Coordinate with platform and infra teams when incidents touch OpenShift, GPUs, vLLM or NVIDIA Enterprise AI services
• Perform safe data operations. Validate indices, manage Redis eviction strategies and handle backfills and reindex tasks with minimal risk
• Champion secure by default practices. Enforce secrets handling, PII redaction and prompt or output guardrails
• Lead post incident reviews with blameless RCAs and concrete corrective actions. Close the loop with business stakeholders
• Proactively surface reliability risks and propose code or architecture changes that remove recurring failure modes

Skills

Python

LangChain

FastAPI/Flask

Async patterns

Task queues

Git

Kubernetes

OpenShift

Observability

PII handling

LLM serving

Data safety

Incident updates

Tools

Elasticsearch

Redis

Grafana

Prometheus

OpenTelemetry

Job description

United Overseas Bank Limited (UOB) is recruiting an incident-focused Software Engineer/SRE in Singapore to own L3 GenAI production incident response, debug LangChain and Elasticsearch issues, implement safe hotfixes, and collaborate with business units to ensure reliable services. You will enhance observability, runbooks, and post-incident reviews while working with OpenShift, GPUs, and NVIDIA AI services.

The role emphasizes security, data handling, and proactive reliability improvements across