AI Data & Quality Analytics Tester

PwC; Bengaluru; On-site; INR 2,600,000 - 4,800,000; Full time

Job summary

PwC in Bangalore is seeking a GenAI / LLM Agent Test Engineer to assure quality and responsible behavior of LLM-based and RAG-based agents across single and multi-agent workflows.

The role emphasizes prompt engineering, hallucination detection, agent reasoning validation, and automated evaluation using modern LLM testing frameworks.

Qualifications

• Bachelors or masters in CS/Engineering/Data Science or equivalent experience.
• 5+ years in QA automation or software engineering with data pipelines.
• 3+ years in AI-driven automation or intelligent testing frameworks.
• 2+ years building or testing AI-based tools or platforms.

Responsibilities

• Perform data validation and reconciliation across pipelines using SQL and automated checks.
• Design and execute test strategies for data quality pipelines and schema validation.
• Validate predictive quality metrics and defect leakage signals for correctness.
• Conduct defect trend analysis and risk-based testing using telemetry.
• Integrate automated data validation into CI/CD and perform prompt testing / A/B comparisons for AI-assisted analytics.

Education

Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or equivalent

Tools

SQL (Postgres / Snowflake)

Pandas / NumPy

Python (data validation)

Promptfoo

Test data profiling tools

OpenAI / Azure OpenAI APIs

Cursor AI

GitHub Copilot / Claude CLI

PydanticAI

Job description

GenAI / LLM Agent Test Engineer

Experience : 5 to 9 years

Location : Bangalore

Job Summary:

We are looking for a GenAI / LLM Agent Test Engineer to assure quality, reliability, and responsible behavior of LLMbased and RAGbased agents, including singleagent and multiagent workflows. The role focuses on prompt engineering, hallucination detection, agent reasoning validation, and automated evaluation of GenAI outputs using modern LLM testing frameworks.

Key Responsibilities:

• Perform data validation and reconciliation across source-to-target pipelines using SQL and automated checks
• Design and execute test strategies for data quality pipelines, including test data preparation, labeling verification, and schema validation
• Validate predictive quality metrics (trend forecasts, risk scoring, defect leakage signals) for correctness and stability
• Execute defect trend analysis and riskbased testing using telemetry and historical patterns
• Implement automated data validation in CI/CD pipelines and ensure continuous quality gatesConduct prompt testing and A/B comparisons for AIassisted analytics interactions (where applicable)
• Leverage LLMs for:
• Intelligent test data generation
• Automated data labeling validation
• AIassisted anomaly detection in data quality pipelines
• Produce actionable insights via statistical thinking and analytical reasoning; communicate quality risks clearly to stakeholders

Qualifications and Skills:

• Bachelors or master’s degree in computer science, Engineering, Data Science, or equivalent experience.
• 6+ years of experience in QA automation or software engineering.
• 3+ years of experience working with AIdriven automation or intelligent testing frameworks.
• Minimum 2 years of handson experience in building or testing AIbased tools or platforms.

Must Have Tools

• SQL (Postgres / Snowflake)
• Pandas / NumPy
• Python (data validation)
• Promptfoo (A/B prompt tests)
• Test data profiling tools
• OpenAI / Azure OpenAI APIs
• Cursor AI / GitHub Copilot / Claude CLI
• PydanticAI