Forward-Deployed AI Evaluation Engineer AVP

Citigroup; New York (NY); On-site; USD 121,000 - 182,000; Full time

Job summary

Citigroup seeks an Applications Development Senior Programmer Analyst - Forward Deployed Engineer in New York to build automated evaluation harnesses for AI agents, curate regression testbeds, and optimize prompts and output schemas.

The role concentrates on ensuring reliability, accuracy, and safety of production AI agents through rigorous evaluation suites in Agentflow Studio, and driving automated CI/CD test execution.

Qualifications

• Experience with Python and AI evaluation
• Hands-on with automated testing and CI/CD pipelines
• Strong attention to data integrity and software quality

Responsibilities

• Constructs automated evaluation test suites and calibration of metrics
• Maintains golden datasets and regression testbeds
• Optimizes prompts, system instructions, and JSON output parsers
• Authors automated test pipelines and enforces gates before staging/production
• Analyzes agent errors and recommends prompts and code fixes
• Implements security measures and safety policies in post-implementation analysis
• Monitors evaluation cycles and reports performance to stakeholders
• Operates with independence as an evaluation SME

Skills

Python programming

AI evaluation frameworks

CI/CD workflows

Communication skills

Education

Bachelor's/University degree in Computer Science, Data Science, or related quantitative discipline

Tools

Docker

Git

REST/gRPC APIs

Job description

Citigroup seeks an Applications Development Senior Programmer Analyst - Forward Deployed Engineer in New York to build automated evaluation harnesses for AI agents, curate regression testbeds, and optimize prompts and output schemas.

The role concentrates on ensuring reliability, accuracy, and safety of production AI agents through rigorous evaluation suites in Agentflow Studio, and driving automated CI/CD test execution.