AI QA Engineer

Konstant Infosolutions Pvt. Ltd.; Jaipur; Presencial; INR 900 000 - 1 500 000; Tempo integral

Resumo da oferta

Konstantinfo is seeking an AI QA Engineer to ensure reliability, safety, and quality of intelligent applications powered by LLMs, RAG pipelines, and AI agents. You will design and execute tests for accuracy, hallucinations, groundedness, bias, security, prompt injection, and workflow correctness.

You will own evaluation strategy, automation, golden datasets, and go/no-go recommendations for AI releases, collaborating with engineers and security teams to improve observability and governance.

Qualificações

• 3+ years of experience in QA engineering, test automation, ML QA, AI evaluation, or MLOps.
• Hands-on experience testing LLM applications, RAG pipelines, or AI agents in production or pre-production environments.
• Understanding of AI failure modes: hallucinations, grounding errors, prompt sensitivity, context limits, and tool misuse.
• Familiarity with AI evaluation tools and platforms such as MLflow, Galileo, PyRIT, LangSmith, or equivalent.
• Knowledge of security testing concepts including prompt injection, jailbreaking, and data leakage risks.
• Experience working with CI/CD, containers, cloud environments, and observability tools.
• Strong analytical skills to interpret evaluation results and communicate risks clearly.
• Experience with multi-agent systems, autonomous workflows, or long-horizon task evaluation.
• Knowledge of responsible AI, fairness testing, and regulatory compliance for AI systems.

Responsabilidades

• Design test strategies for AI applications including LLMs, RAG systems, conversational AI, and agentic workflows.
• Build automated evaluation pipelines for accuracy, groundedness, context relevance, reasoning, tool-use correctness, and latency.
• Create and maintain golden datasets, test scenarios, and evaluation metrics for regression testing.
• Implement LLM-as-a-Judge, schema validation, tracing, and calibrated scoring for multi-turn and long-horizon tasks.
• Test for hallucinations, bias, harmful content, policy violations, PII leaks, and brand drift using filters and guardrails.
• Conduct red-team testing for prompt injection, jailbreaking, data exfiltration, and adversarial inputs.
• Embed evaluation harnesses into CI/CD pipelines and monitor production telemetry for behavioral drift and failures.
• Analyze model and integration failures to identify root causes, controls, and mitigations.
• Translate business risks into measurable quality criteria, thresholds, and release recommendations.
• Collaborate with engineers, product managers, and security teams on testability, observability, and governance.

Conhecimentos

Python

Pytest

Test automation

QA engineering

ML QA

AI evaluation

CI/CD

Observability

multi-agent systems

Ferramentas

MLflow

Galileo

PyRIT

LangSmith

Descrição da oferta de emprego

Telemetry Red-Teaming Prompt Injection LLM-as-a-Judge Frameworks PyRIT Galileo Jenkins CI/CD Pytest GitHub Actions Observability Platforms LangSmith MLflow GCP Kubernetes Vector Databases FastAPI Docker Azure AWS Python

Job Description:

We are looking for an AI QA Engineer to ensure the reliability, safety, and quality of intelligent applications powered by LLMs, RAG pipelines, and AI agents. You will design and execute functional, non-functional, and AI-specific tests covering accuracy, hallucinations, groundedness, bias, security, prompt injection, and workflow correctness. AI QA engineers own evaluation strategy, test automation, golden datasets, and go/no-go recommendations for AI releases.

Requirements and Duties:

Design test strategies for AI applications including LLMs, RAG systems, conversational AI, and agentic workflows.Build automated evaluation pipelines for accuracy, groundedness, context relevance, reasoning, tool-use correctness, and latency.Create and maintain golden datasets, test scenarios, and evaluation metrics for regression testing.Implement LLM-as-a-Judge, schema validation, tracing, and calibrated scoring for multi-turn and long-horizon tasks.Test for hallucinations, bias, harmful content, policy violations, PII leaks, and brand drift using filters and guardrails.Conduct red-team testing for prompt injection, jailbreaking, data exfiltration, and adversarial inputs.Embed evaluation harnesses into CI/CD pipelines and monitor production telemetry for behavioral drift and failures.Analyze model and integration failures to identify root causes, controls, and mitigations.Translate business risks into measurable quality criteria, thresholds, and release recommendations.Collaborate with engineers, product managers, and security teams on testability, observability, and governance.

Skill and Ability:

3+ years of experience in QA engineering, test automation, ML QA, AI evaluation, or MLOps.Strong proficiency in Python and test frameworks such as Pytest, with experience building automated test suites.Hands-on experience testing LLM applications, RAG pipelines, or AI agents in production or pre-production environments.Understanding of AI failure modes: hallucinations, grounding errors, prompt sensitivity, context limits, and tool misuse.Familiarity with AI evaluation tools and platforms such as MLflow, Galileo, PyRIT, LangSmith, or equivalent.Knowledge of security testing concepts including prompt injection, jailbreaking, and data leakage risks.Experience working with CI/CD, containers, cloud environments, and observability tools.Strong analytical skills to interpret evaluation results and communicate risks clearly.Experience with multi-agent systems, autonomous workflows, or long-horizon task evaluation.Knowledge of responsible AI, fairness testing, and regulatory compliance for AI systems.Background in software development, data engineering, or ML engineering.Contributions to AI testing frameworks, open-source evaluation tools, or internal QA standards.

Current Openings

We consider each team member as a part of our family. Our work culture is rooted in transparency, meaningful collaboration, and inclusivity. Explore career opportunities and find the role that fits your skills.

We are hunting expert web and app developers, ingenious graphic designers and solution-oriented project managers to work with us on the various challenging and innovative projects of our clients.