Senior AI Observability & Guardrail Engineer

Bank of America; Charlotte (NC); On-site; USD 122,000 - 200,000; Full time

Job summary

Bank of America is seeking an experienced software engineer to own the end-to-end delivery of critical vendor applications on an on-prem OpenShift platform. You will build APIs and integrations, and establish guardrails for enterprise AI Agent solutions.

You will implement model evaluation methods, develop observability dashboards, and lead cross-functional teams to ensure reliable, compliant, and scalable AI services.

Qualifications

• Bachelor's or Master's degree in Computer Science, Data Science, MIS, or related field.
• 13+ years of software engineering experience designing, building, and scaling production applications.
• Strong Python proficiency and experience with OpenShift or Kubernetes.
• Experience taking LLM- or RAG-based applications into production at scale.
• Experience building APIs and integrating third-party platforms.
• Experience with AI Agents development, including OpenAI APIs and LangChain.
• Knowledge of guardrails, observability, and model-based evaluation for AI apps.
• Experience delivering in Agile or Agile-at-scale environments.

Responsibilities

• Own delivery of critical vendor applications from build through production support in on-prem OpenShift.
• Build and maintain APIs and integrations to enterprise AI Agent solutions.
• Establish workflows to monitor performance, safety, latency, throughput, and guardrail effectiveness.
• Implement and maintain model evaluation frameworks using LLMs and specialized models.
• Develop observability dashboards and alerts with Grafana or similar tools.
• Troubleshoot production issues and improve scalability and resilience.
• Contribute to evaluation methodologies and guardrails for AI systems.
• Lead and mentor engineers, set development standards, and drive process improvements.

Skills

Automation

Influence

Result Orientation

Stakeholder Management

Technical Strategy Development

Application Development

Architecture

Business Acumen

Risk Management

Solution Design

Agile Practices

Analytical Thinking

Collaboration

Data Management

Solution Delivery Process

Education

Bachelor's or Master's degree in Computer Science, Data Science, MIS, a related field, or equivalent practical experience

Tools

OpenShift

Kubernetes

LangChain

OpenAI APIs

LangSmith

Splunk

LlamaIndex

Haystack

Job description

Bank of America is seeking an experienced software engineer to own the end-to-end delivery of critical vendor applications on an on-prem OpenShift platform. You will build APIs and integrations, and establish guardrails for enterprise AI Agent solutions.

You will implement model evaluation methods, develop observability dashboards, and lead cross-functional teams to ensure reliable, compliant, and scalable AI services.