Forward-Deployed AI Engineer – AVP

Citigroup Inc.; New York (NY); On-site; USD 121,000 - 182,000; Full time

Benefits offered by this job

Medical, dental & vision coverage

401(k)

Life, accident & disability insurance

Wellness programs

Paid time off

Job summary

Citigroup Inc. is seeking an Applications Development Senior Programmer Analyst — Forward Deployed Engineer (C12) in New York, NY.

You will lead the creation of automated evaluation harnesses, CI/CD pipelines, and robust testing suites for production AI agents, partnering with business and technology teams to deliver reliable, secure solutions. The role emphasizes hands-on engineering, architecture, and enterprise integration, with a focus on safe, scalable AI systems and observability.

Qualifications

• 5+ years of relevant software engineering or systems analysis experience.
• Solid Python programming proficiency, with practical experience integrating REST/gRPC APIs, Docker containers, and Git workflows.
• Strong working knowledge of AI evaluation frameworks, benchmarking methodologies (BLEU, ROUGE, semantic similarity, LLM-as-a-judge), and automated testing suites.
• Experience in managing and implementing automated test pipelines in CI/CD environments (Lightspeed, Tekton, Jenkins).
• Ability to work effectively under pressure, meeting tight sprint deadlines and adapting to evolving requirements.
• Good written and verbal communication skills with a methodical, detail-oriented approach to software quality and data integrity.

Responsibilities

• Build and run automated evaluation test suites with Agentflow Studio, measuring task completion, tool calls, latency, and hallucination.
• Curate and maintain golden datasets, synthetic benchmarks, and regression testbeds with domain experts.
• Optimize prompts, instructions, few-shot examples, and JSON parsers (Pydantic/JSON Schema) for deterministic behavior.
• Develop and maintain automated test and deployment pipelines in Lightspeed CI/CD, enforcing gates before staging and production.
• Analyze agent errors and performance; recommend coding and prompt improvements.
• Implement security measures and Lakera Guardrail policies to ensure safe usage.
• Monitor test cycles, provide metrics and diagnostic reporting to stakeholders and leads.
• Operate with limited supervision, providing independent judgment as an evaluation SME.
• Assess risk and ensure compliance with laws and policy; escalate and report control issues transparently.

Skills

Python

REST APIs

Docker

Git workflows

AI evaluation frameworks

CI/CD

Education

Bachelor’s degree in Computer Science or related field

Tools

Agentflow Studio

Lightspeed CI/CD

Tekton

Jenkins

Job description

Citigroup Inc. is seeking an Applications Development Senior Programmer Analyst — Forward Deployed Engineer (C12) in New York, NY.

You will lead the creation of automated evaluation harnesses, CI/CD pipelines, and robust testing suites for production AI agents, partnering with business and technology teams to deliver reliable, secure solutions. The role emphasizes hands-on engineering, architecture, and enterprise integration, with a focus on safe, scalable AI systems and observability.