Senior AI Observability & Guardrails Engineer

Bank of America

Jersey City (NJ)

On-site

USD 180,000 - 240,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Bank of America is seeking an experienced engineer to own delivery of vendor AI monitoring and governance apps on‑prem OpenShift. You will build, deploy, and operate containerized services while ensuring audit‑ready reliability for risk‑sensitive workloads.

This role demands strong Python expertise, LLM/AI guardrails, and collaboration across teams to meet milestones. You will mentor engineers, define standards, and contribute to budgeting in a regulated enterprise setting.

Qualifications

Bachelor's or Master's degree in Computer Science, Data Science, MIS, or related field, or equivalent practical experience.
13+ years of software engineering experience designing, building, and scaling production applications.
Strong proficiency in Python, including common data and ML libraries; hands-on experience with OpenShift or Kubernetes.
Proven experience taking LLM- or RAG-based applications into production and maintaining them at scale.
Experience building APIs and integrating third-party platforms.
Experience with AI Agents development, including OpenAI APIs and LangChain.
Understanding of inference performance optimization; experience with guardrails, observability, and model-based evaluation.
Experience delivering within Agile and Agile-at-scale environments against firm timelines.
Proven ability to influence outcomes and meet delivery commitments in a matrixed organization.

Responsibilities

Own delivery of critical vendor applications from build through production support in a fully on-premises OpenShift environment, meeting milestones and quality expectations.
Build and maintain APIs and integrations connecting vendor guardrail and observability platforms to enterprise AI Agent solutions.
Establish and manage workflows that monitor performance, safety, latency, throughput, quality, and guardrail effectiveness for AI risk and control objectives.
Implement and maintain model evaluation frameworks using LLM-as-a-Judge and specialized language models.
Develop observability dashboards and alerting using Grafana or similar tools with audit-friendly evidence and traceability.
Troubleshoot production issues and improve scalability, resilience, and operational efficiency for critical services.
Contribute to evaluation methodologies and guardrails for agentic AI systems.
Lead and mentor engineers, set development standards, and drive process improvements across cross-functional delivery.
Provide input into financial planning, budgeting, forecasting, and management of supported applications.