AI Data & Quality Analytics Tester(GenAI / Agent Testing Engineer)

PwC; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

PwC seeks a GenAI / LLM Agent Test Engineer to assure quality and responsible behavior of LLM-based and RAG-based agents. The role emphasizes prompt engineering, hallucination detection, agent reasoning validation, and automated evaluation using modern LLM testing frameworks.

Responsibilities include functional, regression, integration, and system testing for AI-driven workflows, testing edge cases, hallucination scenarios, and adversarial prompts, with CI/CD automated validation and

Qualifications

• Bachelor's or master's degree in computer science, engineering, data science or equivalent experience.
• 6+ years of experience in QA automation or software engineering.
• 3+ years of experience with AI-driven automation or intelligent testing frameworks.
• Minimum 2 years of hands-on experience building or testing AI-based tools or platforms.

Responsibilities

• Perform data validation and reconciliation across source-to-target pipelines using SQL and automated checks.
• Design and execute test strategies for data quality pipelines, including test data preparation, labeling verification, and schema validation.
• Validate predictive quality metrics (trend forecasts, risk scoring, defect leakage signals) for correctness and stability.
• Execute defect trend analysis and risk-based testing using telemetry and historical patterns.
• Implement automated data validation in CI/CD pipelines and ensure continuous quality gates.
• Conduct prompt testing and A/B comparisons for AI-assisted analytics interactions where applicable.
• Leverage LLMs for intelligent test data generation, automated data labeling validation, and anomaly detection in data quality pipelines.

Skills

QA automation

AI testing

Data validation

Python scripting

SQL

Prompt engineering

Test planning

Education

Bachelor's or master's in CS/Engineering/Data Science

Tools

SQL

Pandas/NumPy

Python

Promptfoo

Test data profiling

OpenAI/Azure OpenAI APIs

Cursor AI / Copilot

PydanticAI

Job description

Role Name

GenAI / Agent Testing Engineer

Position - Senior Associate

Location- (Bangalore)

Job Description Summary

Job Summary:

We are looking for a GenAI / LLM Agent Test Engineer to assure quality, reliability, and responsible behavior of LLMbased and RAGbased agents, including singleagent and multiagent workflows. The role focuses on prompt engineering, hallucination detection, agent reasoning validation, and automated evaluation of GenAI outputs using modern LLM testing frameworks.

Perform functional, regression, integration, and system testing for AI-driven workflows.
Test edge cases, hallucination scenarios, and adversarial prompts.

Key Responsibilities:

• Perform data validation and reconciliation across source-to-target pipelines using SQL and automated checks
• Design and execute test strategies for data quality pipelines, including test data preparation, labeling verification, and schema validation
• Validate predictive quality metrics (trend forecasts, risk scoring, defect leakage signals) for correctness and stability
• Execute defect trend analysis and riskbased testing using telemetry and historical patterns
• Implement automated data validation in CI/CD pipelines and ensure continuous quality gates
• Conduct prompt testing and A/B comparisons for AIassisted analytics interactions (where applicable)
• Leverage LLMs for:
• Intelligent test data generation
• Automated data labeling validation
• AIassisted anomaly detection in data quality pipelines
• Produce actionable insights via statistical thinking and analytical reasoning; communicate quality risks clearly to stakeholders

Qualifications and Skills:

• Bachelors or master’s degree in computer science, Engineering, Data Science, or equivalent experience.
• 6+ years of experience in QA automation or software engineering.
• 3+ years of experience working with AIdriven automation or intelligent testing frameworks.
• Minimum 2 years of handson experience in building or testing AIbased tools or platforms.

Must Have Tools:

• SQL (Postgres / Snowflake)
• Pandas / NumPy
• Python (data validation)
• Promptfoo (A/B prompt tests)
• Test data profiling tools
• OpenAI / Azure OpenAI APIs
• Cursor AI / GitHub Copilot / Claude CLI
• PydanticAI

Good To Have Tools:

• Power BI / Tableau
• Great Expectations
• Evidently AI
• MLflow
• CI data checks (dbt tests)
• Apache Airflow
• dbt (Data Build Tool)

Nice To Have Tools:

• AI bias detection tools
• WhyLabs
• Fairlearn
• Data drift simulators
• Feature store exposure

Communication:

• Strong verbal and written communication skills, with the ability to clearly articulate defects, risks, and quality insights.
• AI & GenAI Testing Skills:
• Handson experience in testing AI/LLMbased systems, including promptdriven workflows, RAG pipelines, and agentbased solutions.
• Strong exposure to prompt engineering, prompt validation, and response evaluation techniques.

Additional Information:

• Travel Requirements: Travel to client locations may be required as per project requirements.
• Shift Requirements: Required to work on shift as per project requirements.
• Line of Service: Advisory
• Industry: Enterprise Testing Managed Services
• Designation: Senior Associate
• Location: Bangalore, India (ONLY)

Position - Senior Associate

Location- (Bangalore)