AI QA Engineer

NewVision Software

Pune District

Hybrid

INR 1,200,000 - 1,800,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

New Vision Softcom & Consultancy Pvt. Ltd is seeking an AI Test Engineer to validate quality, reliability and safety of AI systems, including LLMs, RAG pipelines and AI agents. You will design evaluation datasets, prompts and benchmarks, and drive automated testing within CI/CD pipelines.

The role focuses on prompt behavior, risk management and governance, with hands-on work in Python, Pytest, Java, Selenium and Playwright, plus Azure AI Services. Hybrid Pune-based office preferred.

Qualifications

  • 3-6 years of experience in AI Test Automation.
  • Proven experience in testing AI/ML/LLM-based systems.
  • Strong understanding of prompt behavior, model evaluation, dataset curation and risk management.
  • Hands-on experience with Python, Pytest, Pyrit, Java; test automation tools (Selenium, Playwright).
  • AI evaluation tools and frameworks.
  • Azure AI Services / Azure OpenAI, CI/CD pipelines and GitHub Actions.

Responsibilities

  • Define AI test strategy for LLMs, AI agents, and orchestration workflows leveraging tools & framework like DeepEval, Arize.
  • Design and maintain golden evaluation datasets, prompt test cases, and benchmarking frameworks for LLMs, RAG systems.
  • Perform regression testing across prompt updates, agent logic changes, multi-step agent workflows and model upgrades.
  • Measure and analyse accuracy, consistency, latency, throughput, and cost efficiency of AI services.
  • Integrate AI test suites into CI/CD pipelines with automated quality gates.
  • Conduct red-team testing to identify safety, compliance, security, and prompt-injection vulnerabilities.
  • Provide actionable insights through evaluation reports, metrics, and release readiness assessments.

Skills

AI test automation
Python
Pytest
Java
Selenium
Playwright
Azure AI Services
CI/CD pipelines

Tools

DeepEval
Arize
Selenium
Playwright
GitHub Actions

Job description

Company Name New Vision Softcom & Consultancy Pvt. Ltd

Function Business Units (BU)

Department/Practice Digital Assurance

Organization Unit Technology Assurance / Automation

Region APAC

Country India

Base office location Pune

Working model Hybrid

Weekly Off Pune office Standard

State Maharashtra

Job Description

The AI Test Engineer is responsible for validating the quality, reliability, safety, performance, and governance of AI systems, including LLMs, RAG pipelines, and AI agents. Unlike traditional QA role, this position focuses on LLM evaluation, behavioral testing, hallucination, bias detection, AI test automation, and alignment with Responsible AI governance standards.
Key Responsibilities:

  • Define AI test strategy for LLMs, AI agents, and orchestration workflows leveraging Tools & framework like DeepEval, Arize
  • Design and maintain golden evaluation datasets, prompt test cases, and benchmarking frameworks for LLMs, RAG systems
  • Perform regression testing across prompt updates, agent logic changes, multi-step agent workflows and model upgrades
  • Measure and analyse accuracy, consistency, latency, throughput, and cost efficiency of AI services
  • Integrate AI test suites into CI/CD pipelines with automated quality gates
  • Conduct red-team testing to identify safety, compliance, security, and prompt-injection vulnerabilities
  • Provide actionable insights through evaluation reports, metrics, and release readiness assessments

Qualifications:

  • 3-6 years of experience in AI Test Automation
  • Proven experience in testing AI/ML/LLM-based systems
  • Strong understanding of:
    Prompt behavior, model evaluation, dataset curation and risk management
  • Hands-on experience with:
    Python, Pytest,Pyrit, Java
    Test automation tools (Selenium, Playwright)
  • AI evaluation tools and frameworks
  • Azure AI Services / Azure OpenAI, CI/CD pipelines and GitHub Actions
  • Test automation integration into DevOps workflows
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior QA AI Testing Engineer
Senior QA AI Testing Engineer

NewVision • Hyderabad

On-site
INR 2,500,000 - 5,000,000
Senior AI QA Engineer (Pune )
Senior AI QA Engineer (Pune )

Money Forward India • Pune District

On-site
INR 2,800,000 - 4,000,000
AI QA Engineer
AI QA Engineer

Konstant Infosolutions Pvt. Ltd. • Jaipur

On-site
INR 900,000 - 1,500,000
AI QA Engineer
AI QA Engineer

Mployee.me • Delhi

On-site
INR 1,500,000 - 2,800,000
Qa Test Engineer
Qa Test Engineer

Pwc India • Gurugram District

Hybrid
INR 800,000 - 1,600,000
AI or ML Engineer - Python, QA Testing
AI or ML Engineer - Python, QA Testing

Optum • Hyderabad

On-site
INR 1,800,000 - 3,200,000
Senior QA Engineer, AI
Senior QA Engineer, AI

Intercontinental Exchange Holdings, Inc. • Hyderabad

On-site
INR 2,000,000 - 4,000,000
Gen AI QA Engineer
Gen AI QA Engineer

Deqode • Bengaluru

On-site
INR 1,200,000 - 2,100,000
Senior QA Engineer, AI
Senior QA Engineer, AI

ICE • Hyderabad

On-site
INR 1,800,000 - 3,000,000
QA Engineer
QA Engineer

Weekday AI • India

Remote
INR 1,000,000 - 2,500,000