AI Quality Automation AVP — Scale AI Validation & ML Ops

Citi; New York (NY); Hybrid; USD 31,000 - 63,000; Full time

Job summary

Citi seeks an AVP-level AI Quality Automation Engineer to design and maintain automated testing for AI/ML systems in a hybrid setup. You will validate model outputs, data integrity, and software integrations, collaborating with data scientists, ML engineers, and engineers across teams.

The role emphasizes building robust validation services, monitoring production performance, and upholding ethical AI guidelines within Citi's enterprise-scale quality framework.

Qualifications

• 5-8 years of professional experience implementing data-intensive solutions or test automation frameworks.
• 2-3 years of direct experience validating AI/ML, Generative AI, or data-intensive systems.
• Experience with agile methodologies.
• Strong programming in Python and Shell scripting.
• Experience with CI/CD and ML lifecycle tooling.

Responsibilities

• Design, build, and maintain scalable automated testing frameworks for AI/ML systems.
• Validate model accuracy, data pipelines, and software integrations.
• Bridge between data science, software engineering, and QA.
• Lead adversarial testing to identify vulnerabilities and ensure compliance.
• Implement monitoring/observability for model drift, latency, and performance in production.
• Collaborate with cross-functional stakeholders to align on quality benchmarks.

Skills

Python

Shell scripting

Jupyter notebook scripting

Java

AI/ML validation

Education

Bachelor's degree or equivalent experience

Tools

MLFlow

Dagster

Vertex AI

SageMaker

HuggingFace Transformers

RAGAs

DsPy

Job description

Citi seeks an AVP-level AI Quality Automation Engineer to design and maintain automated testing for AI/ML systems in a hybrid setup. You will validate model outputs, data integrity, and software integrations, collaborating with data scientists, ML engineers, and engineers across teams.

The role emphasizes building robust validation services, monitoring production performance, and upholding ethical AI guidelines within Citi's enterprise-scale quality framework.