AI Quality Automation Engineer - Assistant Vice President

Citi; New York (NY); Hybrid; USD 31,000 - 63,000; Full time

Job summary

Citi seeks an AVP-level AI Quality Automation Engineer to design and maintain automated testing for AI/ML systems in a hybrid setup. You will validate model outputs, data integrity, and software integrations, collaborating with data scientists, ML engineers, and engineers across teams.

The role emphasizes building robust validation services, monitoring production performance, and upholding ethical AI guidelines within Citi's enterprise-scale quality framework.

Qualifications

• 5-8 years of professional experience implementing data-intensive solutions or test automation frameworks.
• 2-3 years of direct experience validating AI/ML, Generative AI, or data-intensive systems.
• Experience with agile methodologies.
• Strong programming in Python and Shell scripting.
• Experience with CI/CD and ML lifecycle tooling.

Responsibilities

• Design, build, and maintain scalable automated testing frameworks for AI/ML systems.
• Validate model accuracy, data pipelines, and software integrations.
• Bridge between data science, software engineering, and QA.
• Lead adversarial testing to identify vulnerabilities and ensure compliance.
• Implement monitoring/observability for model drift, latency, and performance in production.
• Collaborate with cross-functional stakeholders to align on quality benchmarks.

Skills

Python

Shell scripting

Jupyter notebook scripting

Java

AI/ML validation

Education

Bachelor's degree or equivalent experience

Tools

MLFlow

Dagster

Vertex AI

SageMaker

HuggingFace Transformers

RAGAs

DsPy

Job description

AI Quality Automation Engineer - Assistant Vice President

Location(s): Pune, Maharashtra, India

Job Type: Hybrid

Posted: Aug. 24, 2026

Discover your future at Citi

Working at Citi is far more than just a job. A career with us means joining a team of approximately 219,000 dedicated people from around the globe. At Citi, you’ll have the opportunity to grow your career, give back to your community and make a real impact.

Job Overview

In this AVP-level role, you will be responsible for designing, implementing, and maintaining automated testing frameworks for AI systems. You will validate model accuracy, data pipelines, and software integrations to ensure high performance, reliability, and ethical compliance across the development lifecycle.

You will act as a crucial bridge between data science, software engineering, and quality assurance, working closely with data scientists, ML systems engineers, front-end engineers, and business leaders. If you are passionate about operationalizing complex models, creating highly reliable automated validation services, and defining the quality lifecycle for data science workflows at an enterprise scale, this role is a perfect fit.

Key Responsibilities & Competencies

• Quality Framework Architecture: Design, build, and maintain scalable automated testing frameworks specifically tailored for AI/ML systems, including LLMs, RAG pipelines, and traditional ML models.
• Model & Output Validation: Design and execute automated evaluation pipelines to measure critical metrics such as hallucination rates, context retention, retrieval precision, semantic similarity, and answer relevance (using tools like RAGAs, DsPy).
• Data Pipeline & ETL Auditing: Validate the quality, integrity, and schema compliance of data pipelines and ETL processes (PySpark, Hive, Kafka, Parquet, Iceberg) to ensure clean data flows into AI models.
• CI/CD Integration: Integrate automated AI quality gates into enterprise CI/CD pipelines (using Tekton, Harness, UDeploy, Jenkins) to enable continuous testing and prevent regressions.
• Risk Stewardship & Compliance: Lead adversarial testing (Red Teaming) to identify vulnerabilities like prompt injection, jailbreaking, and data leakage. Validate real-time safety guardrails to ensure compliance with ethical AI guidelines, corporate policies, and regulatory standards.
• Operational Excellence & Observability: Implement monitoring and observability solutions (using MLFlow, Dagster, Vertex AI, SageMaker) to detect model drift, performance degradation, and latency bottlenecks in production.
• Stakeholder Engagement: Demonstrate exceptional communication and diplomacy skills, effectively engaging with cross-functional stakeholders to articulate quality risks, negotiate solutions, and align on quality benchmarks.

Required Skills & Technical Qualifications

• Experience: 5-8 years of professional experience implementing data-intensive solutions or test automation frameworks using agile methodologies, with at least 2-3 years of direct experience validating AI/ML, Generative AI, or data-intensive systems.
• Programming Languages:
  • Primary: Python, Shell Scripting, Jupyter Notebook Scripting
  • Secondary: Java
• Machine Learning & AI Evaluation Frameworks:
  • Core ML: Scikit-learn, Keras, TensorFlow, PyTorch, spaCy, NLTK
  • Advanced NLP & GenAI Evaluation: RAGAs, DsPy, Hugging Face Transformers, RASA, Large Language Models (LLM)
• MLOps & ML Lifecycle Management:
  • Platforms: MLFlow, Dagster, Google Cloud Vertex AI, Amazon SageMaker
• Database Management Systems:
  • Relational: Oracle, MySQL, PostgreSQL
  • NoSQL: MongoDB
• Infrastructure & DevOps:
  • CI/CD & DevOps Tooling: UDeploy, Harness, Tekton, Jenkins, Maven
  • Version Control: Git, Subversion (SVN)
  • Containerization & Orchestration: Docker, OpenShift, Amazon ECS
• Data Engineering & Processing:
  • Messaging & Asynchronous Communication: Kafka, RabbitMQ
  • ETL & Data Warehousing: Sqoop, PySpark, Hive, Hadoop, HDFS
  • File Formats: Avro, Parquet, Iceberg
  • Job Scheduling & Orchestration: Autosys, Cron
• Web Technologies:
  • Web Frameworks: FastAPI, Flask
  • Web Servers & Reverse Proxies: NGINX

Education:

• Bachelor's/University degree or equivalent experience

Job Family Group: Technology

Job Family: Technology Quality

Time Type: Full time

Most Relevant Skills

Please see the requirements listed above.

Other Relevant Skills

For complementary skills, please see above and/or contact the recruiter.

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View Citi's EEO Policy Statement and the Know Your Rights poster.