Forward Deployed Engineer - Assistant Vice President

Citigroup; New York (NY); On-site; USD 121,000 - 182,000; Full time

Job summary

Citigroup seeks an Applications Development Senior Programmer Analyst - Forward Deployed Engineer in New York to build automated evaluation harnesses for AI agents, curate regression testbeds, and optimize prompts and output schemas.

The role concentrates on ensuring reliability, accuracy, and safety of production AI agents through rigorous evaluation suites in Agentflow Studio, and driving automated CI/CD test execution.

Qualifications

• Experience with Python and AI evaluation
• Hands-on with automated testing and CI/CD pipelines
• Strong attention to data integrity and software quality

Responsibilities

• Constructs automated evaluation test suites and calibration of metrics
• Maintains golden datasets and regression testbeds
• Optimizes prompts, system instructions, and JSON output parsers
• Authors automated test pipelines and enforces gates before staging/production
• Analyzes agent errors and recommends prompts and code fixes
• Implements security measures and safety policies in post-implementation analysis
• Monitors evaluation cycles and reports performance to stakeholders
• Operates with independence as an evaluation SME

Skills

Python programming

AI evaluation frameworks

CI/CD workflows

Communication skills

Education

Bachelor's/University degree in Computer Science, Data Science, or related quantitative discipline

Tools

Docker

Git

REST/gRPC APIs

Job description

The Applications Development Senior Programmer Analyst - Forward Deployed Engineer (C12) is an intermediate-level software engineering position responsible for constructing automated evaluation harnesses, curating regression testbeds, optimizing prompts and output schemas, and automating testing pipelines within embedded LOB delivery pods. The overall objective of this role is to ensure the reliability, accuracy, and safety of production AI agents by building rigorous evaluation suites in Agentflow Studio and driving automated CI/CD test execution.

Responsibilities

• Conducts tasks related to the construction, calibration, and execution of automated evaluation test suites using Agentflow Studio , measuring agent task completion, tool call correctness, latency, and hallucination metrics.
• Curates and maintains domain-specific golden datasets, synthetic benchmarks, and regression testbeds in collaboration with LOB domain experts and senior engineers.
• Optimizes prompt templates, system instructions, few-shot examples, and structured JSON output parsers (Pydantic/JSON Schema) to maximize deterministic agent behavior.
• Authors and maintains automated test execution and deployment pipelines within Lightspeed CI/CD , enforcing continuous evaluation gates prior to staging and production rollout.
• Utilizes in-depth specialty knowledge of Python applications development to analyze complex agent errors, evaluate system performance, and recommend advanced coding and prompt solutions.
• Recommends and implements security measures and Lakera Guardrail policies in post-implementation analysis of agent usage to ensure safe functionality.
• Monitors and controls evaluation test cycles, providing operational support, performance metrics, and prompt diagnostic reporting to business stakeholders and squad leads.
• Has the ability to operate with a limited level of direct supervision, exercising independence of judgement and autonomy while acting as an evaluation SME to pod team members.
• Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency.

Qualifications

• 5+ years of relevant software engineering or systems analysis experience (or 2+ years with specialized focus in Python and Generative AI application development).
• Solid Python programming proficiency, with practical experience integrating REST/gRPC APIs, Docker containers, and Git workflows.
• Strong working knowledge of AI evaluation frameworks, benchmarking methodologies (BLEU, ROUGE, semantic similarity, LLM-as-a-judge), and automated testing suites.
• Experience in managing and implementing automated test pipelines in CI/CD environments (Lightspeed, Tekton, Jenkins).
• Ability to work effectively under pressure, managing tight sprint deadlines and unexpected changes in model outputs or LOB requirements.
• Good written and verbal communication skills with a methodical, detail-oriented approach to software quality and data integrity.

Education

• Bachelor's/University degree or equivalent experience in Computer Science, Data Science, or related quantitative discipline.

Job Family Group

Technology

Job Family

Applications Development

Primary Location

New York New York United States

Full Time Salary Range

$121,200.00 - $181,800.00 In addition to salary, Citi's offerings may also include, for eligible employees, discretionary and formulaic incentive and retention awards. Citi offers competitive employee benefits, including: medical, dental & vision coverage; 401(k); life, accident, and disability insurance; and wellness programs. Citi also offers paid time off packages, including planned time off (vacation), unplanned time off (sick leave), and paid holidays. For additional information regarding Citi employee benefits, please visit citibenefits.com. Available offerings may vary by jurisdiction, job level, and date of hire.

Automated Processing and AI

We use automated processing, including artificial intelligence, for our

Most Relevant Skills

Please see the requirements listed above.