AI Quality Engineer

Tekskills Inc.; Delhi; On-site; INR 1,400,000 - 2,200,000; Full time

Job summary

Tekskills Inc. is seeking an AI Quality Engineer (Tester) to lead testing and evaluation of AI applications, focusing on LLMs, RAG, and agent workflows in Pune. The role covers design of test cases, performance testing, and security validations to ensure quality and business alignment.

You will work with cross-functional teams to implement automated tests in CI/CD pipelines, monitor AI system performance, and support release readiness with an emphasis on reliability and safety.

Qualifications

• 3+ years in QA/testing (API + performance testing).
• Experience with automation tools such as Selenium and Playwright.
• Programming knowledge in Python or TypeScript.
• CI/CD, version control, and cloud basics (AWS/Azure/GCP).

Responsibilities

• Design and execute functional and regression tests (SIT/UAT).
• Evaluate LLMs using frameworks like RAGAS and LangSmith.
• Test RAG systems for retrieval accuracy, hallucination, grounding.
• Validate agent workflows, multi-step logic and tool usage.
• Perform API, performance, and load testing.
• Conduct security testing for prompts, data privacy and guardrails.
• Integrate automated tests into CI/CD pipelines.
• Monitor and optimize AI system performance.
• Collaborate with cross-functional teams for release readiness.

Skills

QA testing

Automation tools

Python/TypeScript

Tools

Selenium

Playwright

Postman

REST Assured

k6

JMeter

CI/CD tools

AWS

Azure

GCP

Job description

Role: AI Quality Engineer (Tester)
Experience: 3-4 years
Location: Pune

Purpose

Lead testing and evaluation of AI applications (LLMs, RAG, agent workflows) to ensure quality, safety, performance, and business alignment.

Key Responsibilities

• Design & execute functional/regression test cases (SIT/UAT)
• Evaluate LLMs using frameworks (e.g., RAGAS, PromptFoo)
• Test RAG systems: retrieval accuracy, hallucination, grounding
• Validate agent workflows: multi-step logic, tool usage, structured outputs
• Perform API, performance, and load testing (latency, throughput, cost)
• Conduct security testing (prompt injection, data privacy, guardrails)
• Integrate automated tests into CI/CD pipelines
• Monitor, report, and optimize AI system performance
• Collaborate with cross-functional teams and support release readiness

Required Skills

• 3+ years in QA/testing (API + performance testing)
• Experience with automation tools (Selenium, Playwright)
• Programming knowledge (Python/TypeScript)
• CI/CD, version control, and cloud basics (AWS/Azure/GCP)

Preferred

• AI/ML and MLOps understanding
• Experience with LLM evaluation tools (LangSmith, RAGAS, TruLens, etc.)
• Knowledge of AI security testing and observability tools

Tools

Playwright, Selenium, Postman, REST Assured, k6, JMeter, CI/CD tools

Traits

Strong communication, problem-solving, and quality-focused mindset