Senior AI Engineer

Amalgamated Bank of NY; New York (NY); Hybrid; USD 170,000 - 200,000; Full time

Job summary

Amalgamated Bank seeks an experienced Senior AI Engineer to lead and build governed AI agents on the Databricks platform within the Data Governance Team. You will translate business needs into scalable AI capabilities, deliver production-grade agents, and ensure auditable, compliant deployments in a regulated banking environment.

You will design, test, deploy, and operate agents using retrieval-augmented generation, tool calling, and multi-agent patterns; collaborate with security, data

Responsibilities

• Act as hands-on lead engineer writing production code and delivering agents and LLM apps within the Bank's AI architecture on Databricks.
• Design, build, test, deploy, and operate production agents using retrieval-augmented generation, tool calling, multi-agent, supervisory, and human-in-the-loop patterns.
• Lead prompt and context engineering, model selection, evaluation, token usage optimization, and cost-performance trade-offs.
• Configure and operate Databricks AI Gateway including routing, access policies, logging, governance, and guardrails.
• Partner with business stakeholders across credit, lending, relationship banking to translate requirements into agent scope and acceptance criteria.
• Collaborate with Information Security on AI observability and telemetry, SIEM signals, and anomaly detection.
• Build CI/CD, automated testing, monitoring for AI workloads, mentor junior engineers, evaluate vendors, and support governance artifacts.

Skills

Python

SQL

Databricks

AI/LLM

Agent-based design

Prompt engineering

Cost optimization

Security/compliance

Tiered governance

Education

Bachelor's in CS/Engineering/Data Science or related

Master's degree preferred

Tools

LangGraph

LlamaIndex

Salesforce Data Cloud

Agentforce

MuleSoft

Copilot Studio

MLflow tracing

Job description

If you are unable to complete this application due to a disability, contact this employer to ask for an accommodation or an alternative application process.

New York, NY, US

2 days ago Requisition ID: 2282

Salary Range: $170,000.00 To $200,000.00 Annually

Purpose of Position:

We are seeking an experienced Senior AI Engineer to serve as a hands-on technical lead and primary builder within the Data Governance Team. This individual will design, build, deploy, and operate governed, auditable agentic solutions on the Bank's Databricks foundation. The role combines production engineering and architecture judgment, with direct responsibility for translating business needs into scalable AI capabilities that deliver measurable value in a regulated environment.

Essential Job Functions:

• Act as a hands-on lead engineer and primary builder, writing production code and delivering agents and LLM applications while shaping the Bank's AI architecture on Databricks.
• Design, build, test, deploy, and operate production agents using retrieval-augmented generation, tool calling, multi-agent, supervisory, and human-in-the-loop patterns.
• Build and integrate Model Context Protocol (MCP) clients and servers; author secure custom tools and APIs; and establish reusable patterns for tool discovery, permissions, execution, and error handling.
• Lead prompt and context engineering, model selection, evaluation, and token-usage optimization to improve quality, latency, reliability, and cost efficiency.
• Configure and operate Databricks AI Gateway as the control point for model and agent access, including routing, access policies, rate limits, usage logging, cost controls, and guardrails.
• Establish agent identity and authorization patterns, including on-behalf-of and delegated identity approaches, so agent actions are properly attributed, authorized, and auditable.
• Partner with business stakeholders across credit, lending, relationship banking, and other functions to understand context and requirements and translate them into agent scope, delivery plans, and acceptance criteria.
• Manage backlog priorities and trade-off decisions across business value, risk, cost, complexity, and delivery timing; drive adoption and measure and communicate business KPI impact.
• Collaborate with BI engineers to architect dashboards that monitor agent cost, accuracy, usage, performance, traceability, and business outcomes.
• Partner with Information Security on AI observability by identifying relevant control-plane and agent telemetry, defining signals for enterprise SIEM platforms such as Google SecOps or Splunk, and recognizing anomalous patterns that warrant investigation.
• Partner with data stewards and data engineers to certify data sources and enforce classification, lineage, and entitlements through Unity Catalog.
• Integrate agents across platforms using Salesforce Data Cloud and Agentforce, Microsoft Copilot Studio, and MuleSoft.
• Collaborate with Information Security, Model Risk, Compliance, Legal, and Internal Audit to develop governance standards, documentation, control evidence, and testing artifacts.
• Build CI/CD, automated testing, evaluation, and operational monitoring for AI workloads; mentor junior engineers; evaluate vendors and implementation partners; and perform other duties as directed.

Knowledge, Skills and Experience Requirements:

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related field, or equivalent experience; Master's degree preferred.
• 7+ years of experience in software, data, or machine learning engineering, including 2+ years delivering LLM or agent-based solutions to production.
• Demonstrated experience delivering agentic capabilities in a regulated industry that produced measurable business value and scaled beyond proof of concept; financial services experience preferred.
• Strong Python and SQL skills with hands-on production experience using Databricks, Spark, Delta Lake, Unity Catalog, and Databricks AI Gateway or a comparable LLM gateway.
• Experience building and integrating MCP clients and servers, developing custom tools, implementing tool-calling workflows, and securing agent access to enterprise systems and data.
• Strong prompt and context engineering skills, including model selection, evaluation design, token-usage optimization, and cost/performance trade-off management.
• Knowledge of agent identity and authorization patterns, including on-behalf-of and delegated identity models.
• Experience implementing guardrails for cost, accuracy, safety, observability, and end-to-end traceability of LLM and agent workloads.
• Experience working directly with non-technical business teams to define scope, manage prioritization trade-offs, drive adoption, and connect AI delivery to measurable business outcomes.
• Understanding of AI and data governance, model risk management, privacy, and data security in a regulated environment.
• Excellent problem-solving and communication skills, with the ability to influence technical, business, risk, and executive stakeholders.

Preferred Qualifications:

• Production experience with LangGraph, LlamaIndex, or an equivalent agentic orchestration framework.
• Experience with Salesforce Data Cloud and Agentforce, MuleSoft, or Microsoft Copilot Studio.
• Experience with MLflow tracing, inference tables, or comparable LLM observability tooling.
• Familiarity with Google SecOps, Splunk, or comparable SIEM platforms and AI telemetry integration.
• Familiarity with SR 11-7 model risk guidance and emerging AI regulatory expectations.
• Databricks or Azure AI certifications.

.

Our job titles may span more than one career level. The starting base salary for this role is between $170,000 – $200,000. The actual base pay is dependent upon many factors, such as: training, transferrable skills, work experience, business needs and market demands. The base pay range is subject to change and may be modified in the future.

Hybrid Work Model

Effective February 18, 2025, employees in office-based positions will be working a Hybrid work schedule consisting of three days or more, on-site per week, Monday - Thursday, although the specific days may vary by site or organization, with Friday designated as a remote-working day, unless business critical tasks require an on-site presence. This Hybrid work model does not apply to, and daily in-person attendance is required for, the contact center, branch service roles, and general services where the work to be performed is located at a Company site; positions covered by a collective-bargaining agreement (unless the agreement provides for hybrid work); or any other position for which the Company has determined the job requirements cannot be reasonably met working remotely. Please note, this Hybrid work model guidance does not apply to roles that have been designated as “remote”.

Search Firm Representatives- Please Read Carefully

Amalgamated Bank does not accept unsolicited assistance from search firms for employment opportunities. All CVs / resumes submitted by search firms to any employee at our company without a valid written search agreement in place for the position will be deemed the sole property of our company. No fee will be paid in the event a candidate is hired by our company as a result of an agency referral where no pre-existing agreement is in place. Where agency agreements are in place, introductions are position specific. Please, no phone calls or emails.