Senior Forward-Deployed AI Engineer & QA Automation

Citigroup Inc.; New York (NY); On-site; USD 121,000 - 182,000; Full time

Job summary

Citigroup Inc. in New York seeks an Applications Development Senior Programmer Analyst — Forward Deployed Engineer to build automated evaluation harnesses, curate regression testbeds, and ensure reliable AI agent performance.

You will optimize prompts, system instructions, and JSON output parsers, and drive automated CI/CD testing pipelines with Lightspeed CI/CD, Python apps, and tool integrations to deliver safe, accurate production AI solutions.

Qualifications

• 5+ years of relevant software engineering or systems analysis experience (or 2+ years with specialized focus in Python and Generative AI application development).
• Solid Python programming proficiency with experience integrating REST/gRPC APIs, Docker containers, and Git workflows.
• Strong knowledge of AI evaluation frameworks, benchmarking methodologies, and automated testing suites.

Responsibilities

• Constructs automated evaluation test suites and calibration of testbeds.
• Maintains domain-specific golden datasets and regression testbeds in collaboration with engineers.
• Optimizes prompt templates, system instructions, and JSON output parsers to maximize deterministic agent behavior.
• Authors and maintains automated test execution and deployment pipelines within Lightspeed CI/CD, enforcing gates prior to staging and production.
• Uses Python knowledge to analyze agent errors, evaluate performance, and propose coding and prompt solutions.
• Recommends security measures and guardrail policies to ensure safe functionality.
• Monitors evaluation cycles, provides metrics and diagnostics to stakeholders and squad leads.
• Operates with minimal supervision as an evaluation SME to pod team members.

Skills

Python programming

APIs integration

CI/CD experience

AI evaluation frameworks

communication under pressure

Education

Bachelor’s degree in Computer Science or related field

Tools

Docker

Git workflows

Lightspeed

Tekton

Jenkins

Job description

Citigroup Inc. in New York seeks an Applications Development Senior Programmer Analyst — Forward Deployed Engineer to build automated evaluation harnesses, curate regression testbeds, and ensure reliable AI agent performance.

You will optimize prompts, system instructions, and JSON output parsers, and drive automated CI/CD testing pipelines with Lightspeed CI/CD, Python apps, and tool integrations to deliver safe, accurate production AI solutions.