Senior QA Automation Engineer (Python, Selenium & AI Testing) - Assistant Vice President

Citi; New York (NY); Hybrid; USD 26,000 - 47,000; Full time

Job summary

Citi is seeking a Senior QA Automation Engineer to design and implement comprehensive test strategies for AI/ML and Generative AI applications. You will build scalable test automation with Python and Selenium, perform rigorous data validation with SQL, and collaborate with data scientists, engineers, and product managers to ensure high-quality software releases.

The role emphasizes mentoring, adherence to SDLC, and driving QA practices across teams, with a hybrid work model across Citi locations.

Qualifications

• 7+ years in Quality Engineering or Software/Data QA.
• Strong Python scripting for test automation.
• Hands-on experience with web automation using Selenium.
• SQL experience for data validation.
• Experience testing AI/ML systems; Generative AI knowledge desired.
• Solid QE fundamentals and SDLC understanding.
• Ability to design evaluation frameworks for GenAI, LLM, and RAG.
• Excellent analytical and communication skills.

Responsibilities

• Lead UAT processes and product rollout testing.
• Conduct complex process monitoring and audits; report issues and propose solutions.
• Ensure SDLC standards and documentation are followed; improve docs.
• Monitor conformance to standards; identify risks and mitigations.
• Document findings and perform root-cause analyses; propose improvements.
• Produce quality metrics reports and ensure traceability.
• Mentor junior team members and share knowledge.
• Collaborate with data scientists, software engineers, and product managers.
• Develop and execute test strategies for AI/ML and Generative AI applications.

Skills

Python scripting for test automation

Selenium web automation

AI/ML testing

Quality Engineering fundamentals

SDLC knowledge

Communication skills

Education

Bachelor’s degree or equivalent experience

Tools

Selenium

Langfuse

Arize AI

Fiddler AI

RAGAS

Promptfoo

DeepEval

TruLens

Job description

Senior QA Automation Engineer (Python, Selenium & AI Testing) - Assistant Vice President

Location(s):

Pune, Maharashtra, India, Chennai, Tamil Nadu, India

Job Type:

Hybrid

Time Type:

Full time

Discover your future at Citi

Working at Citi is far more than just a job. A career with us means joining a team of approximately 219,000 dedicated people from around the globe. At Citi, you’ll have the opportunity to grow your career, give back to your community and make a real impact.

Job Overview

The IT Quality Sr Analyst is a seasoned professional role. Applies in-depth disciplinary knowledge, contributing to the development of new techniques and the improvement of processes and work‑flow for the area or function. Integrates subject matter and industry expertise within a defined area. Requires in-depth understanding of how areas collectively integrate within the sub‑function as well as coordinate and contribute to the objectives of the function and overall business. Evaluates moderately complex and variable issues with substantial potential impact, where development of an approach/taking of an action involves weighing various alternatives and balancing potentially conflicting situations using multiple sources of information. Requires good analytical skills in order to filter, prioritize and validate potentially complex and dynamic material from multiple sources. Strong communication and diplomacy skills are required. Regularly assumes informal/formal leadership role within teams. Involved in coaching and training of new recruits Significant impact in terms of project size, geography, etc. by influencing decisions through advice, counsel and/or facilitating services to others in area of specialization. Work and performance of all teams in the area are directly affected by the performance of the individual.

Responsibilities

• Responsible for initiatives related to User Acceptance Testing (UAT) process and product rollout into production, making recommendations of solutions and improvements. Responsibilities may also include software quality assurance testing.
• Conducts complex process monitoring, product evaluation and audit assignments; reports issues and makes recommendations for solutions.
• Ensures project standards and procedures exist, are fully / clearly documented and followed throughout the software development life cycle; makes recommendations for enhanced documentation.
• Monitors products for conformance to standards; monitors processes for conformance to procedures; identifies risks and recommends solutions; ensures annual organizational reviews of standards and procedures are conducted.
• Documents findings and performs root cause analyses to provide recommended improvements. Independently recommends solutions and improvements to existing standards and procedures; collaborates with other areas in technology to develop new standards and procedures.
• Ensures all corrective actions are assigned, documented and completed. Verifies traceability in project work products. Gathers, maintains and creates reports on quality metrics.
• Exhibits sound understanding of concepts and principles in own technical area and a basic knowledge of these elements in other areas.
• Requires in-depth understanding of how own area integrates within IT Quality and has basic commercial awareness.
• Acts as advisor and mentor for junior members of the team.
• Makes evaluative judgments based on analysis of factual information in complicated and novel situations and provides solutions to problems.
• Has direct impact on the team and closely related teams by ensuring the quality of the tasks/services/information provided by self and others.
• Requires sound and comprehensive communication and diplomacy skills to exchange complex information.
• Performs other duties and functions as assigned.
• Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm’s reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency.
• Design, develop, and execute comprehensive test strategies, plans, and test cases for AI/ML and Generative AI applications.
• Build and maintain robust, scalable test automation frameworks using Python and Selenium.
• Perform rigorous data validation and integrity testing using advanced SQL queries to ensure data quality.
• Evaluate AI model performance, accuracy, fairness, and robustness to ensure they meet the highest quality standards.
• Collaborate closely with data scientists, software engineers, and product managers to integrate testing seamlessly into the development lifecycle.
• Identify, document, and track defects and performance issues from discovery through to resolution.
• Champion quality engineering best practices and contribute to the continuous improvement of our testing processes and methodologies.

Recommended Qualifications

• 7+ years of professional experience in Quality Engineering or Software/Data Quality Assurance.
• Strong proficiency in Python scripting for test automation and framework development.
• Hands‑on experience with web automation tools, particularly Selenium.
• Solid expertise in SQL for writing complex queries and performing backend data validation.
• Proven experience in testing AI/Machine Learning systems; familiarity with Generative AI concepts is highly desirable.
• Deep understanding of QE fundamentals, test methodologies, and the software development lifecycle (SDLC).
• Design and execute comprehensive evaluation frameworks for Generative AI, Large Language Model (LLM), and Retrieval‑Augmented Generation (RAG) applications.
• Assess and benchmark AI models (e.g., GPT, Gemini, and other foundation models) against defined business and technical requirements, providing recommendations on model selection and suitability.
• Implement and manage AI observability and monitoring solutions using platforms such as Langfuse, Arize AI, and Fiddler AI.
• Implement automated AI quality evaluation frameworks such as RAGAS, Promptfoo, DeepEval, and TruLens to measure key quality metrics including accuracy, relevance, precision, faithfulness, context recall, and response quality.
• Establish AI guardrails and governance controls to detect and mitigate risks related to bias, toxicity, hallucinations, prompt injection, jailbreak attempts, and unsafe outputs.
• Excellent analytical and problem‑solving abilities with a meticulous eye for detail.
• Strong communication and interpersonal skills, with a proven ability to work effectively in a collaborative team environment.

Education

• Bachelor’s/University degree or equivalent experience

Job Family Group:

Technology

Job Family:

Technology Quality

Time Type:

Full time

Most Relevant Skills

Please see the requirements listed above.

Other Relevant Skills

For complementary skills, please see above and/or contact the recruiter.

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi. View Citi’s EEO Policy Statement and the Know Your Rights poster.