Forward Deployed AI Engineer - Assistant Vice President

Citigroup Inc.; New York (NY); On-site; USD 121,000 - 182,000; Full time

Benefits offered by this job

Medical, dental & vision coverage

401(k)

Life, accident & disability insurance

Wellness programs

Paid time off

Job summary

Citigroup Inc. is seeking an Applications Development Senior Programmer Analyst — Forward Deployed Engineer (C12) in New York, NY.

You will lead the creation of automated evaluation harnesses, CI/CD pipelines, and robust testing suites for production AI agents, partnering with business and technology teams to deliver reliable, secure solutions. The role emphasizes hands-on engineering, architecture, and enterprise integration, with a focus on safe, scalable AI systems and observability.

Qualifications

• 5+ years of relevant software engineering or systems analysis experience.
• Solid Python programming proficiency, with practical experience integrating REST/gRPC APIs, Docker containers, and Git workflows.
• Strong working knowledge of AI evaluation frameworks, benchmarking methodologies (BLEU, ROUGE, semantic similarity, LLM-as-a-judge), and automated testing suites.
• Experience in managing and implementing automated test pipelines in CI/CD environments (Lightspeed, Tekton, Jenkins).
• Ability to work effectively under pressure, meeting tight sprint deadlines and adapting to evolving requirements.
• Good written and verbal communication skills with a methodical, detail-oriented approach to software quality and data integrity.

Responsibilities

• Build and run automated evaluation test suites with Agentflow Studio, measuring task completion, tool calls, latency, and hallucination.
• Curate and maintain golden datasets, synthetic benchmarks, and regression testbeds with domain experts.
• Optimize prompts, instructions, few-shot examples, and JSON parsers (Pydantic/JSON Schema) for deterministic behavior.
• Develop and maintain automated test and deployment pipelines in Lightspeed CI/CD, enforcing gates before staging and production.
• Analyze agent errors and performance; recommend coding and prompt improvements.
• Implement security measures and Lakera Guardrail policies to ensure safe usage.
• Monitor test cycles, provide metrics and diagnostic reporting to stakeholders and leads.
• Operate with limited supervision, providing independent judgment as an evaluation SME.
• Assess risk and ensure compliance with laws and policy; escalate and report control issues transparently.

Skills

Python

REST APIs

Docker

Git workflows

AI evaluation frameworks

CI/CD

Education

Bachelor’s degree in Computer Science or related field

Tools

Agentflow Studio

Lightspeed CI/CD

Tekton

Jenkins

Job description

The Applied AI and Agents Factory brings together ambitious engineers to turn advanced AI capabilities into production-ready solutions. Working directly with business and technology teams, you’ll tackle complex engineering challenges, build intelligent agents and take ideas from initial concept through to live deployment.

This is an opportunity to work at the cutting edge of AI engineering, developing systems that go beyond conversational tools to reason, plan and execute multi-step workflows. You’ll take ownership of the full engineering lifecycle, from agent architecture and orchestration to enterprise integration, security and observability.

With a strong focus on hands‑on engineering, collaboration and rapid delivery, the Factory gives you the opportunity to solve challenging technical problems, influence architectural decisions and help shape reusable tools and engineering standards adopted across Citi.

The Applications Development Senior Programmer Analyst — Forward Deployed Engineer (C12) is an intermediate‑level software engineering position responsible for constructing automated evaluation harnesses, curating regression testbeds, optimizing prompts and output schemas, and automating testing pipelines within embedded LOB delivery pods. The overall objective of this role is to ensure the reliability, accuracy, and safety of production AI agents by building rigorous evaluation suites in Agentflow Studio and driving automated CI/CD test execution.

Responsibilities:

• Conducts tasks related to the construction, calibration, and execution of automated evaluation test suites using Agentflow Studio, measuring agent task completion, tool call correctness, latency, and hallucination metrics.
• Curates and maintains domain‑specific golden datasets, synthetic benchmarks, and regression testbeds in collaboration with LOB domain experts and senior engineers.
• Optimizes prompt templates, system instructions, few‑shot examples, and structured JSON output parsers (Pydantic/JSON Schema) to maximize deterministic agent behavior.
• Authors and maintains automated test execution and deployment pipelines within Lightspeed CI/CD, enforcing continuous evaluation gates prior to staging and production rollout.
• Utilizes in‑depth specialty knowledge of Python applications development to analyze complex agent errors, evaluate system performance, and recommend advanced coding and prompt solutions.
• Recommends and implements security measures and Lakera Guardrail policies in post‑implementation analysis of agent usage to ensure safe functionality.
• Monitors and controls evaluation test cycles, providing operational support, performance metrics, and prompt diagnostic reporting to business stakeholders and squad leads.
• Has the ability to operate with a limited level of direct supervision, exercising independence of judgement and autonomy while acting as an evaluation SME to pod team members.
• Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm’s reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency.

Qualifications:

• 5+ years of relevant software engineering or systems analysis experience (or 2+ years with specialized focus in Python and Generative AI application development).
• Solid Python programming proficiency, with practical experience integrating REST/gRPC APIs, Docker containers, and Git workflows.
• Strong working knowledge of AI evaluation frameworks, benchmarking methodologies (BLEU, ROUGE, semantic similarity, LLM-as-a-judge), and automated testing suites.
• Experience in managing and implementing automated test pipelines in CI/CD environments (Lightspeed, Tekton, Jenkins).
• Ability to work effectively under pressure, managing tight sprint deadlines and unexpected changes in model outputs or LOB requirements.
• Good written and verbal communication skills with a methodical, detail‑oriented approach to software quality and data integrity.

Education:

• Bachelor’s/University degree or equivalent experience in Computer Science, Data Science, or related quantitative discipline.

Job Family Group:

Technology

Job Family:

Applications Development

Primary Location:

New York New York United States

Primary Location Full Time Salary Range:

$121,200.00 - $181,800.00

In addition to salary, Citi’s offerings may also include, for eligible employees, discretionary and formulaic incentive and retention awards. Citi offers competitive employee benefits, including: medical, dental & vision coverage; 401(k); life, accident, and disability insurance; and wellness programs. Citi also offers paid time off packages, including planned time off (vacation), unplanned time off (sick leave), and paid holidays. For additional information regarding Citi employee benefits, please visit citibenefits.com. Available offerings may vary by jurisdiction, job level, and date of hire.

Most Relevant Skills:

Please see the requirements listed above.

Automated Processing and AI

We use automated processing, including artificial intelligence, for our legitimate business interests (or our reasonable and appropriate business purposes) to identify and align the candidate's skills and abilities with a specific job opening. Additionally, if you so choose, or consent, we can match your skills and abilities to other suitable roles at Citi.Importantly, all our hiring processes and decisions, including determining your suitability for a role, are conducted, checked, and decided by individuals. Our automated processing and AI do not involve relying on automatic or autonomous decision‑making. Please refer to any Jurisdictional Considerations, with specific provisions for your country (where relevant) for further details.Illinois residents – AI Notice and Right

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View Citi’s EEO Policy Statement and the Know Your Rights poster.