QA Engineer – Generative AI & Agentic AI

UST; Thiruvananthapuram; In loco; INR 1.500.000 - 2.300.000; Tempo pieno

Descrizione del lavoro

UST is seeking an experienced QA Engineer for Generative AI & Agentic AI to lead quality assurance across GenAI solutions. The role covers multi-step agent workflows, Text-to-SQL, and RAG pipelines, ensuring production-ready AI outputs and reliable AC acceptance criteria.

You will design test strategies, implement evaluation metrics, and own end-to-end QA for production releases. Collaboration with Data Science, Platform Engineering, and Product teams is essential to define quality gates and

Competenze

• GenAI/LLM application testing
• Agentic AI and multi-step agent workflows
• Agent orchestration and tool/function-call validation
• Text-to-SQL testing
• Retrieval-Augmented Generation (RAG) testing
• LLM-as-a-Judge evaluation
• Prompt and LLM behavior validation
• Hallucination and response-grounding assessment
• Context-window and context-handling concepts
• AI output-quality evaluation beyond traditional pass/fail testing

Mansioni

• Design, develop, and execute test strategies for GenAI and Agentic AI applications.
• Test multi-step agentic workflows validating orchestration, tool calls, and outputs across hops.
• Evaluate AI-generated outputs for correctness, relevance, coherence, and reliability.
• Test Text-to-SQL workflows: NL->SQL accuracy, syntax, schema alignment, query execution, results.
• Validate RAG pipelines: retrieval relevance, grounding, chunking, context use.
• Apply GenAI evaluation metrics: embedding similarity, ROUGE, BLEU, LLM-as-a-Judge.
• Develop evaluation frameworks and scoring rubrics for automated assessment.
• Create golden datasets, regression suites, and reusable testing frameworks.
• Own QA activities for end-to-end production releases and post-production monitoring.
• Collaborate with DS, Platform Eng, and Product to set acceptance criteria and gates.
• Identify LLM-specific edge cases and failure modes including prompt injection risks.
• Document defects with evidence and maintain traceability of coverage and results.

Conoscenze

Generative AI

Agentic AI

Multi-step workflows

Text-to-SQL testing

RAG testing

LLM-as-a-Judge

Prompt validation

Hallucination assessment

Context handling

AI output quality evaluation

Test Automation

BLEU

RAG

Snowflake

Dataiku

Strumenti

Snowflake

Dataiku

OpenAI workspace

Leena AI

Descrizione del lavoro

Role Description

We are looking for an experienced QA Engineer - Generative AI & Agentic AI to lead quality assurance activities for GenAI and Agentic AI solutions.

This role goes beyond traditional functional testing. The successful candidate will be responsible for validating the correctness, reliability, quality, and production-readiness of AI-generated outputs, including multi-step agent workflows, Text-to-SQL pipelines, Retrieval-Augmented Generation (RAG) systems, and LLM-based applications.

The ideal candidate will have hands-on, production-grade experience testing GenAI/Agentic AI applications end-to-end and should have successfully supported at least one production release of a GenAI or Agentic AI solution.

Key Responsibilities

• Design, develop, and execute comprehensive test strategies for GenAI and Agentic AI applications, covering functional, non-functional, integration, and AI output-quality testing.
• Test multi-step agentic workflows, validating agent orchestration, tool/function calls, task completion, chained decision-making, reasoning consistency, and outputs across individual agent hops.
• Evaluate AI-generated outputs for correctness, relevance, coherence, hallucination, consistency, and reliability.
• Test Text-to-SQL workflows, including validation of:
  • Natural language to SQL generation accuracy
  • SQL syntax and logic
  • Schema/table/column alignment
  • Query execution
  • Result accuracy against Snowflake data models
• Validate RAG pipelines, including:
  • Retrieval relevance
  • Context grounding
  • Chunking effectiveness
  • Context utilization
  • Faithfulness of generated responses to retrieved information
• Apply GenAI evaluation metrics and methodologies such as:
  • Embedding similarity
  • ROUGE
  • BLEU
  • LLM-as-a-Judge
  • Other relevant GenAI quality and accuracy metrics
• Design and maintain LLM-as-a-Judge evaluation frameworks and scoring rubrics for scalable and automated assessment of GenAI outputs.
• Create and maintain golden datasets, evaluation datasets, regression suites, and reusable AI testing frameworks.
• Develop test harnesses/scripts for batch evaluation and automated quality validation of LLM-generated outputs.
• Own QA activities for end-to-end production releases, including test planning, execution, regression testing, release quality assessment, sign-off, and post-production monitoring.
• Partner closely with Data Science, Platform Engineering, Product, and other stakeholders to establish acceptance criteria, quality gates, and go/no-go release benchmarks.
• Identify LLM-specific edge cases and failure modes, including hallucinations, prompt injection risks, bias, inconsistent responses, and agent/tool failures.
• Document and track defects through resolution and provide clear evidence of AI application quality.
• Maintain traceability of test coverage, evaluation results, quality metrics, and release reports for stakeholders and audit requirements.

Mandatory Skills & Experience

GenAI / Agentic AI Testing

• Generative AI / LLM application testing
• Agentic AI and multi-step agent workflows
• Agent orchestration and tool/function-call validation
• Text-to-SQL testing
• Retrieval-Augmented Generation (RAG) testing
• LLM-as-a-Judge evaluation
• Prompt and LLM behavior validation
• Hallucination and response‑grounding assessment
• Context-window and context-handling concepts
• AI output-quality evaluation beyond traditional pass/fail testing

Platforms

• Dataiku: Strong hands-on experience is highly preferred and considered the first preference
• Exposure to platforms such as OpenAI workspace, Leena AI, or similar GenAI/Agentic AI platforms

Data / Database

• Strong hands-on working experience with Snowflake
• Ability to understand and validate data models, schemas, generated SQL queries, and query results

AI Evaluation

Practical Experience With At Least Some Of The Following

• Embedding Similarity
• RO

Provide your feedback on BizChat

Skills

• Generative AI
• Test Automation
• Agentic AI
• BLEU
• LLM-as-a-Judge
• RAG
• Snowflake
• Text-to-SQL