Senior QA Automation Engineer Python Selenium And AI Testing - Assistant Vice President

Citi; Pune District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Citi in Pune, India seeks an IT Quality Sr Analyst to lead UAT initiatives, design robust test strategies for AI/ML applications, and advance QA practices across the SDLC. The role emphasizes Python-based automation, SQL data validation, and collaboration with data scientists and engineers.

You will mentor junior staff, develop test plans for Generative AI and RAG systems, and drive quality improvements through rigorous evaluation and monitoring using industry tools.

Qualifications

• 7+ years of professional experience in Quality Engineering or Software/Data Quality Assurance.
• Strong proficiency in Python scripting for test automation and framework development.
• Hands-on experience with web automation tools, particularly Selenium.
• SQL for writing complex queries and backend data validation.
• Experience testing AI/ML systems; familiarity with Generative AI concepts.
• Deep understanding of QE fundamentals, test methodologies, and the SDLC.
• Design and execute evaluation frameworks for Generative AI, LLM, and RAG applications.
• Knowledge of AI observability/monitoring platforms such as Langfuse, Arize AI, and Fiddler.
• Implement AI quality metrics and guardrails for bias, toxicity, hallucinations, and prompt injection.
• Excellent analytical and problem-solving abilities with strong communication.

Responsibilities

• Lead UAT process initiatives and guide product rollout into production; may include QA testing.
• Conduct complex process monitoring, product evaluation, and audit assignments.
• Ensure project standards and documentation are aligned with the SDLC; improve where needed.
• Monitor conformance to standards; identify risks and propose solutions; annual reviews.
• Document findings, perform root cause analyses, and recommend improvements.
• Provide coaching and mentoring to junior team members; influence decisions across teams.
• Collaborate with data scientists, software engineers, and product managers to integrate testing.
• Identify, document, and track defects and performance issues from discovery to resolution.

Skills

Python scripting

Test automation

AI/ML testing

QE fundamentals

SDLC

Analytical thinking

Communication

Education

Bachelor’s degree or equivalent

Tools

Selenium

Langfuse

Arize AI

Fiddler AI

Promptfoo

DeepEval

TruLens

Job description

Job Description:

The IT Quality Sr Analyst is a seasoned professional role. Applies in-depth disciplinary knowledge, contributing to the development of new techniques and the improvement of processes and work-flow for the area or function. Integrates subject matter and industry expertise within a defined area. Requires in-depth understanding of how areas collectively integrate within the sub-function as well as coordinate and contribute to the objectives of the function and overall business. Evaluates moderately complex and variable issues with substantial potential impact, where development of an approach/taking of an action involves weighing various alternatives and balancing potentially conflicting situations using multiple sources of information. Requires good analytical skills in order to filter, prioritize and validate potentially complex and dynamic material from multiple sources. Strong communication and diplomacy skills are required. Regularly assumes informal/formal leadership role within teams. Involved in coaching and training of new recruits Significant impact in terms of project size, geography, etc. by influencing decisions through advice, counsel and/or facilitating services to others in area of specialization. Work and performance of all teams in the area are directly affected by the performance of the individual.

Responsibilities:

• Responsible for initiatives related to User Acceptance Testing (UAT) process and product rollout into production, making recommendations of solutions and improvements. Responsibilities may also include software quality assurance testing.
• Conducts complex process monitoring, product evaluation and audit assignments; reports issues and makes recommendations for solutions.
• Ensures project standards and procedures exist, are fully / clearly documented and followed throughout the software development life cycle; makes recommendations for enhanced documentation.
• Monitors products for conformance to standards; monitors processes for conformance to procedures; identifies risks and recommends solutions; ensures annual organizational reviews of standards and procedures are conducted.
• Documents findings and performs root cause analyses to provide recommended improvements. Independently recommends solutions and improvements to existing standards and procedures; collaborates with other areas in technology to develop new standards and procedures.
• Ensures all corrective actions are assigned, documented and completed. Verifies traceability in project work products. Gathers, maintains and creates reports on quality metrics.
• Exhibits sound understanding of concepts and principles in own technical area and a basic knowledge of these elements in other areas.
• Requires in-depth understanding of how own area integrates within IT Quality and has basic commercial awareness.
• Acts as advisor and mentor for junior members of the team.
• Makes evaluative judgments based on analysis of factual information in complicated and novel situations and provides solutions to problems.
• Has direct impact on the team and closely related teams by ensuring the quality of the tasks/services/information provided by self and others.
• Requires sound and comprehensive communication and diplomacy skills to exchange complex information.
• Performs other duties and functions as assigned.
• Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency.
• Design, develop, and execute comprehensive test strategies, plans, and test cases for AI/ML and Generative AI applications.
• Build and maintain robust, scalable test automation frameworks using Python and Selenium.
• Perform rigorous data validation and integrity testing using advanced SQL queries to ensure data quality.
• Evaluate AI model performance, accuracy, fairness, and robustness to ensure they meet the highest quality standards.
• Collaborate closely with data scientists, software engineers, and product managers to integrate testing seamlessly into the development lifecycle.
• Identify, document, and track defects and performance issues from discovery through to resolution.
• Champion quality engineering best practices and contribute to the continuous improvement of our testing processes and methodologies.

Recommended Qualifications:

• 7+ years of professional experience in Quality Engineering or Software/Data Quality Assurance.
• Strong proficiency in Python scripting for test automation and framework development.
• Hands-on experience with web automation tools, particularly Selenium.
• Solid expertise in SQL for writing complex queries and performing backend data validation.
• Proven experience in testing AI/Machine Learning systems; familiarity with Generative AI concepts is highly desirable.
• Deep understanding of QE fundamentals, test methodologies, and the software development lifecycle (SDLC).
• Design and execute comprehensive evaluation frameworks for Generative AI, Large Language Model (LLM), and Retrieval-Augmented Generation (RAG) applications.
• Assess and benchmark AI models (e.g., GPT, Gemini, and other foundation models) against defined business and technical requirements, providing recommendations on model selection and suitability.
• Implement and manage AI observability and monitoring solutions using platforms such as Langfuse, Arize AI, and Fiddler AI.
• Implement automated AI quality evaluation frameworks such as RAGAS, Promptfoo, DeepEval, and TruLens to measure key quality metrics including accuracy, relevance, precision, faithfulness, context recall, and response quality.
• Establish AI guardrails and governance controls to detect and mitigate risks related to bias, toxicity, hallucinations, prompt injection, jailbreak attempts, and unsafe outputs.
• Excellent analytical and problem-solving abilities with a meticulous eye for detail.
• Strong communication and interpersonal skills, with a proven ability to work effectively in a collaborative team environment.

Education:

• Bachelor’s/University degree or equivalent experience

------------------------------------------------------

Job Family Group:

Technology

------------------------------------------------------

Job Family:

Technology Quality

------------------------------------------------------

Time Type:

Full time

Most Relevant Skills

Please see the requirements listed above.

------------------------------------------------------

Other Relevant Skills

For complementary skills, please see above and/or contact the recruiter.

------------------------------------------------------

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View Citi’s EEO Policy Statement and the Know Your Rights poster.