Senior QA Engineer, AI

ICE; Hyderabad; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

ICE’s AI Center of Excellence is seeking a Senior Software Test Engineer to own QA for AI-powered applications in Hyderabad, India. You will collaborate with Systems Analysts, Development, and QA to translate AI/ML requirements into robust test scenarios, data validation, and release gates.

The role emphasizes model/output validation, bias/hallucination testing, and safeguarding against adversarial prompts, with hands-on automation using Python and UI testing frameworks.

Qualifications

• 8+ years of QA/test engineering experience, including 3+ years testing AI/ML or data-driven applications.
• 3+ years of experience with AI/LLM evaluation frameworks: prompt regression, hallucination detection, bias/fairness testing, and drift detection.
• 5+ years of Python experience for test automation, data analysis, and evaluation scripting.
• 3+ years of experience with UI automation frameworks (Selenium, Playwright, or Cypress) for AI powered features.
• Strong SQL and data validation skills for testing data pipelines feeding AI models.
• Working knowledge of ML fundamentals: evaluation metrics, train/validation/test splits, model versioning.
• Experience testing conversational AI/chatbots: multi-turn dialogue, intent/context handling, guardrail validation.
• Experience testing REST/GraphQL APIs: functional, contract, schema validation, auth testing, negative/boundary testing, load testing.
• Experience testing AI applications for infosec: prompt injection resistance, data leakage/PII exposure prevention, access control, secure endpoint authentication.
• Ability to validate AI applications against organizational compliance requirements and frameworks (SOC 2, ISO 27001, NIST AI RMF, GDPR).

Responsibilities

• Lead test strategy for AI/ML features: model output validation, regression testing across model/prompt versions, and edge case discovery.
• Build evaluation frameworks for testing AI outputs for accuracy, consistency, bias, hallucination, and safety/guardrail compliance.
• Define golden datasets, ground truth labels, and release acceptance criteria with data science and ML engineering.
• Automate prompt-based test suites and model evaluation metrics into CI/CD release gates.
• Lead and mentor a QA team, setting AI testing standards across projects.
• Translate AI feature requirements into test scenarios and quality gates.
• Monitor production of AI behavior and evolve test coverage based on real-world failures.

Skills

QA/testing

AI/ML evaluation

Python scripting

UI automation

SQL/data validation

ML concepts & evaluation

REST/GraphQL testing

InfoSec/compliance awareness

Conversational AI testing

Mentor/leadership

Tools

Git

LangSmith

Ragas

DeepEval

Selenium

Playwright

Cypress

CI/CD

Job description

ICE's AI Center of Excellence is looking for a Senior Software Test Engineer to own QA for our AI powered applications. This role works with Systems Analysts, Development, and QA to understand business/product and system requirements and define test scenarios for AI applications, combining core QA fundamentals, Python scripting, and API validation with specialized AI/LLM evaluation skills such as detecting hallucinations, bias, and drift.

Responsibilities

• Lead test strategy for AI/ML features: model output validation, regression testing across model/prompt versions, and edge case discovery.
• Build evaluation frameworks for testing AI outputs for accuracy, consistency, bias, hallucination, and safety/guardrail compliance.
• Define golden datasets, ground truth labels, and release acceptance criteria with data science and ML engineering.
• Automate prompt based test suites and model evaluation metrics into CI/CD release gates.
• Lead and mentor a QA team, setting AI testing standards across projects.
• Translate AI feature requirements into test scenarios and quality gates.
• Monitor production of AI behavior and evolve test coverage based on real world failures.

Knowledge And Experience

• 8+ years of QA/test engineering experience, including 3+ years testing AI, ML, or data driven applications.
• 3+ years of experience with AI/LLM evaluation frameworks: prompt regression, hallucination detection, bias/fairness testing, and model/data drift detection.
• 5+ years of Python experience for test automation, data analysis, and evaluation scripting.
• 3+ years of experience with UI automation frameworks (Selenium, Playwright, or Cypress) for AI powered features.
• Strong SQL and data validation skills for testing the pipelines that feed AI models.
• Working knowledge of ML fundamentals: evaluation of metrics, train/validation/test splits, model versioning.
• Experience testing conversational AI/chatbots: multi turn dialogue, intent/context handling, and guardrail validation against harmful or adversarial prompts.
• Experience testing REST/GraphQL APIs: functional, contract, and schema validation, auth testing, negative/boundary testing, and load testing.
• Experience testing AI applications for infosec: prompt injection resistance, data leakage/PII exposure prevention, access control, and secure endpoint authentication, per organizational security policy.
• Ability to validate AI applications against organizational compliance requirements and frameworks (SOC 2, ISO 27001, NIST AI RMF, GDPR), coordinating with InfoSec and Compliance.

Preferred Knowledge And Experience

• Experience testing RAG systems or autonomous agents.
• Familiarity with responsible AI principles: fairness, transparency, safety guardrails.
• Experience with LLM eval tools such as LangSmith, Ragas, or DeepEval.
• Exposure cloud AI/ML platforms (AWS SageMaker, Azure ML, or GCP Vertex AI).
• Experience in financial services, Mortgage or fintech applications.
• Experience with Git or other version control systems.