Senior AI Quality Engineer

Dataart Technologies; Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Dataart Technologies in Bengaluru is seeking an experienced QA Engineer to design AI evaluation pipelines, automate testing, and build agent-based workflows using LangChain, Semantic Kernel, and related tools. You will implement Python pytest automation, validate models, and integrate evaluations into CI/CD with GitHub Actions, Allure, and TestRail while ensuring data quality and security considerations.

Experience in RAG, vector databases, and evaluating LLMs; collaborate with domain experts

Qualifications

• Proficient in Python with pytest for test automation and evaluation pipelines.
• Hands-on experience building AI agents and agentic workflows with LangChain, LangGraph, or Semantic Kernel.
• Experience designing AI evaluation strategies, regression benchmarks, and calibration for LLMS and agents.
• Experience validating deep-learning, reinforcement-learning, and predictive models with LangChain.
• Experience integrating AI evaluation workflows with CI/CD using GitHub Actions, Allure, and TestRail.
• Familiarity with RAG, embeddings, vector databases, and retrieval pipelines.

Responsibilities

• Analyze requirements and define evaluation strategies, golden datasets, regression benchmarks, baselines, thresholds, and blocking metrics with domain-expert review.
• Design, build, and orchestrate AI agents, persona-based workflows, and reusable skills for the STLC using LangChain, LangGraph, or Semantic Kernel.
• Build automated evaluations and benchmarks for LLMs, agents, RAG, retrieval, tool trajectories, and contracts using Ragas, DeepEval, and promptfoo.
• Calibrate LLM-as-judge against human grading, manage non-determinism through repeated runs, and detect regression and drift.
• Write and maintain Python and pytest automation for AI, ML, API, data, and integration workflows.
• Use Cursor and agentic tools to analyze requirements, generate and review tests, and maintain evaluation suites.
• Validate deep-learning, reinforcement-learning, and predictive models using LangChain.
• Build AI-assisted STLC workflows for requirements analysis, test design, script generation, execution, self-healing, failure analysis, defect triage, plus intelligent test selection and defect prediction.
• Integrate LLMs, APIs, embeddings, vector databases, retrieval pipelines, and enterprise data sources into QE frameworks using Python.
• Test REST and streaming APIs, agent tools, RAG and MCP integrations, including authentication, errors, retries, and recovery.
• Test data and retrieval pipelines for schema integrity, lineage, idempotency, fallback, promotion gates, and rollback.
• Integrate evaluations, benchmarks, and agent workflows into QE frameworks and CI/CD on GitHub Actions, review Allure results, and keep TestRail aligned with automated coverage.
• Test prompt injection, jailbreaks, unsafe tool use, data exfiltration, access control, PII controls, and guardrails.
• Measure accuracy, relevance, groundedness, hallucination, retrieval quality, agent reliability, latency, throughput, token use, and cost, and diagnose distributed failures.
• Implement tracing, monitoring, and production evaluation using Langfuse, LangSmith, or a comparable AI observability platform.
• Deploy and operate QE agents on Amazon Bedrock AgentCore or a comparable agent platform, with governance and human oversight.
• Deploy and validate AI and ML solutions on AWS or Azure, including secrets, database services, and cloud-native test infrastructure.
• Report, track, and verify defects in Jira, and work with domain experts, product, and engineering on traceable coverage and untested scope.
• Contribute to release readiness for the assigned product area without owning sign-off for the whole team.

Skills

Python

pytest

LangChain

LangGraph

Semantic Kernel

AI agent orchestration

Test automation

Tools

Ragas

DeepEval

promptfoo

Cursor

Allure

TestRail

GitHub Actions

Jira

Job description

Responsibilities

• Analyze requirements and define evaluation strategies, golden datasets, regression benchmarks, baselines, thresholds, and blocking metrics with domain-expert review.
• Design, build, and orchestrate AI agents, persona-based workflows, and reusable skills for the STLC using LangChain, LangGraph, or Semantic Kernel.
• Build automated evaluations and benchmarks for LLMs, agents, RAG, retrieval, tool trajectories, and contracts using Ragas, DeepEval, and promptfoo.
• Calibrate LLM-as-judge against human grading, manage non-determinism through repeated runs, and detect regression and drift.
• Write and maintain Python and pytest automation for AI, ML, API, data, and integration workflows.
• Use Cursor and agentic tools to analyze requirements, generate and review tests, and maintain evaluation suites.
• Validate deep-learning, reinforcement-learning, and predictive models using LangChain.
• Build AI-assisted STLC workflows for requirements analysis, test design, script generation, execution, self-healing, failure analysis, defect triage, plus intelligent test selection and defect prediction.
• Integrate LLMs, APIs, embeddings, vector databases, retrieval pipelines, and enterprise data sources into QE frameworks using Python.
• Test REST and streaming APIs, agent tools, RAG and MCP integrations, including authentication, errors, retries, and recovery.
• Test data and retrieval pipelines for schema integrity, lineage, idempotency, fallback, promotion gates, and rollback.
• Integrate evaluations, benchmarks, and agent workflows into QE frameworks and CI/CD on GitHub Actions, review Allure results, and keep TestRail aligned with automated coverage.
• Test prompt injection, jailbreaks, unsafe tool use, data exfiltration, access control, PII controls, and guardrails.
• Measure accuracy, relevance, groundedness, hallucination, retrieval quality, agent reliability, latency, throughput, token use, and cost, and diagnose distributed failures.
• Implement tracing, monitoring, and production evaluation using Langfuse, LangSmith, or a comparable AI observability platform.
• Deploy and operate QE agents on Amazon Bedrock AgentCore or a comparable agent platform, with governance and human oversight.
• Deploy and validate AI and ML solutions on AWS or Azure, including secrets, database services, and cloud-native test infrastructure.
• Report, track, and verify defects in Jira, and work with domain experts, product, and engineering on traceable coverage and untested scope.
• Contribute to release readiness for the assigned product area without owning sign-off for the whole team.

Requirements

• Strong Python proficiency with pytest for test and evaluation automation.
• Hands-on experience building and orchestrating AI agents and agentic workflows with LangChain, LangGraph, Semantic Kernel, or comparable frameworks.
• Hands-on experience building AI evaluations and benchmarks with Ragas, DeepEval, promptfoo, or equivalent tools.
• Hands-on use of Cursor or agentic development tools for test generation and maintenance.
• Experience defining AI evaluation strategies, golden datasets, regression benchmarks, and LLM-as-judge calibration, including non-determinism and drift.
• Experience validating deep-learning, reinforcement-learning, and predictive models with LangChain.
• Experience building AI-assisted STLC workflows for requirements analysis, test design, script generation, execution, self-healing, failure analysis, defect triage, intelligent test selection, and defect prediction.
• Experience integrating LLMs, APIs, embeddings, vector databases, and retrieval pipelines with automation frameworks in Python.
• Experience testing REST and streaming APIs, agent tools, RAG workflows, and MCP integrations.
• Experience testing data and retrieval pipelines for correctness, lineage, and safe promotion.
• Experience integrating AI evaluation and agent workflows with automation frameworks and CI/CD using GitHub Actions, Allure, TestRail, or equivalent tools.
• Experience testing AI security, privacy, access control, and guardrails.
• Experience measuring accuracy, relevance, groundedness, hallucination, retrieval quality, agent reliability, latency, throughput, token use, and cost, and diagnosing distributed failures.
• Experience with AI tracing, monitoring, and production evaluation using Langfuse, LangSmith, or comparable tools.
• Familiarity with AWS, Azure, or GCP cloud AI services and agent platforms for deploying and validating AI and ML solutions.
• Experience deploying and operating agents on Amazon Bedrock AgentCore or a comparable agent platform, with governance and human oversight.
• Experience managing defects through resolution in Jira or an equivalent tool.
• Ability to work with domain experts, product, and engineering on ground truth and coverage gaps.
• Experience contributing to release readiness for an assigned product area.

Nice to have

• Playwright and TypeScript experience for UI and API automation.
• Real-device and cross-browser execution using LambdaTest, Sauce Labs, or an equivalent platform.
• Knowledge of AI-driven test optimization and coverage analytics using SeaLights or an equivalent platform.
• Experience contributing reusable components to shared automation frameworks and reviewing automation code through Git and GitHub workflows.
• Experience managing test data and troubleshooting test environments needed for reliable execution.
• Experience triaging security findings and validating operational readiness, monitoring, rollback, and recovery controls.
• Experience producing audit-ready evaluation reports and improving shared evaluation patterns, automation libraries, and datasets.