AI/LLM QA Engineer - Agentic and Multi-Agent System Testing

Crew Kraftorz LLP; Hyderabad; On-site; INR 1,500,000 - 2,200,000; Full time

Job summary

Crew Kraftorz LLP is looking for a highly skilled AI/LLM QA Engineer in Hyderabad to join their team. You will validate the performance, accuracy, and robustness of next-generation AI systems, focusing on orchestration-driven workflows and autonomous agents.

The ideal candidate will have over 4 years of experience in software QA/testing and a minimum of 2 years in AI/ML systems. Responsibilities include developing test strategies and collaborating with AI/ML engineers to ensure reliable systems.

Qualifications

• 4+ years of experience in Software QA/Testing.
• 2+ years in AI/ML or LLM-based systems.
• Hands-on experience with LLM evaluation metrics.

Responsibilities

• Design and execute test strategies for AI systems.
• Perform evaluation using metrics like BLEU, ROUGE, and BERTScore.
• Collaborate with AI/ML engineers to define test cases.

Skills

Agentic & Multi-Agent Testing

LLM evaluation frameworks

Prompt engineering validation

Software QA/Testing

AI/ML or LLM-based systems

Tools

Jira

GitLab

Grafana

Jenkins

Job description

AI/LLM QA Engineer - Agentic and Multi-Agent System Testing

We are seeking a highly skilled AI/LLM QA Engineer with expertise in Agentic and Multi-Agent system testing to join our team. The ideal candidate will have a strong background in software QA/testing combined with hands‑on experience in AI/ML or LLM‑based applications. You will be responsible for validating the performance, accuracy, and robustness of next‑generation AI systems, including orchestration‑driven workflows and autonomous agents.

Requirements

• Experience: 4+ years
• Work Mode: Hybrid
• Employment Type: Full-time
• Notice Period: 15 days

Key responsibilities

• Design and execute test strategies for Agentic and Multi-Agent systems, ensuring reliability and scalability.
• Perform LLM evaluation using metrics such as Exact match / Soft match, BLEU, ROUGE, BERTScore, and semantic similarity using embeddings.
• Develop and maintain prompt testing frameworks and validate output consistency across use cases.
• Implement and evaluate guardrails for responsible AI behavior (hallucinations, toxicity, bias).
• Build automated test pipelines for LLM-driven workflows and orchestration frameworks.
• Collaborate with AI/ML engineers and product teams to define test cases, benchmarks, and acceptance criteria.
• Identify defects, performance issues, and edge cases in AI systems.

Required skills and experience

• Hands‑on experience in Agentic & Multi-Agent Testing.
• Expertise in LLM evaluation frameworks and methodologies.
• Strong understanding of prompt engineering validation, guardrails and safety mechanisms, and output evaluation using embeddings and NLP metrics.
• 3+ years in Software QA/Testing.
• Minimum 2+ years of experience in AI/ML or LLM‑based systems.

Good to have

• Experience with orchestration frameworks such as LangChain, LangGraph, LlamaIndex, DSPy, or OpenAI Assistants/Actions.
• Familiarity with CI/CD tools like GitLab and Jenkins.
• Knowledge of monitoring tools like Grafana and test management/tracking tools like Jira and X-Ray.
• Exposure to AI Quality Engineering practices.

About the Company

Our client is a forward‑thinking organization specializing in innovative AI solutions and technologies.