AI Quality Engineer

Allegis Global Solutions; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Allegis Global Solutions in Bengaluru is seeking an AI Quality Engineer to design how we test intelligent agents and foundation layer capabilities. This role emphasizes automated testing, evaluation pipelines, and code-driven QA in CI/CD, not manual testing.

You will work across architectures, define quality metrics for groundedness, factuality, and task completion, and implement scalable testing frameworks to keep AI systems reliable in production.

Qualifications

• Define testing strategies for AI agents and workflows.
• Design non-deterministic output tests and evaluation criteria.
• Build automated pipelines and CI/CD integration for QA.
• Assess groundedness, factuality, relevance, and task completion.
• Work across architecture patterns like RAG and autonomous workflows.

Responsibilities

• Define testing strategies for AI agents, conversational interfaces, and agentic workflows.
• Design behavioral test suites for non-deterministic outputs - where "correct" isn't binary.
• Build evaluation frameworks that measure groundedness, factuality, relevance, and task completion.
• Identify failure modes specific to AI systems: hallucinations, prompt injection, context window limitations, drift.
• Develop testing approaches for each architecture pattern: RAG, function calling, human-in-the-loop, autonomous workflows.

Skills

AI testing

Automation

Python

CI/CD

Evaluation pipelines

Tools

Python

Jenkins

CI/CD

Job description

Working at Allegis Global Solutions (AGS) is more than just a job. It's a career. It's a community of people who invest in your development and empower you to blaze your own trail. Each of us is here to create real, measurable impact that moves needles. We operate beyond "roles" or "jobs" to realize the opportunity to make meaningful contributions to a bigger idea. Because we believe that when you build a workforce that’s designed to harness human enterprise, you design a workforce that’s built for impact.

At AGS, we help companies all over the world transform their people into a competitive advantage. It's not about filling seats. It's about designing workforces to meet missions and unleash the most transformative power in business today: The power of human enterprise.

With services around the globe, we have a point of view on the future of work that enables us to be a transformative partner in the way work gets done for our clients’ organizations. Meeting clients where they are, we design a plan and guide them along a transformational journey, applying bold actions and diverse minds to solve the most complex challenges - from permanent and extended workforce management to services procurement, consulting, direct sourcing and our Universal Workforce Model.

We also represent over 100 countries and speak dozens of languages. So as you're building relationships and doing your job, you'll be exposed to other cultures and advancement opportunities while expanding your knowledge of global markets and strategies.

As a workplace, we focus on relationships - with each other, our clients and our candidates - in fact serving others is one of our core values. We support open communication and recognize that giving constructive criticism can be even harder than receiving it. We appreciate the fearless and the passionate, who force us to be better. Everything we do sits on a pillar of diversity - diverse perspectives, backgrounds and ideas drive innovation and make us successful.

See what it's like to work at AGS by searching #LifeAtAGS on any social network.

Job Description

About the Role

Testing AI systems is a fundamentally different problem than testing traditional software. Outputs are non-deterministic. "Correct" is often a spectrum. And the failure modes - hallucinations, drift, prompt injection - don't show up in unit tests. We need an engineer who understands this and can build the testing strategies, evaluation frameworks, and quality infrastructure to keep our agents reliable in production.

As an AI Quality Engineer, you'll design how we test intelligent agents, agentic workflows, and Foundation Layer capabilities. This is not a manual QA role—you'll write code, build evaluation pipelines, and create automated testing frameworks that run in CI/CD. You'll define what "quality" means for AI systems at AGS and build the systems to measure it.

You'll work across every solution the team builds, which means you'll have broad visibility into the architecture and deep understanding of how our agents behave in the real world. If you're an engineer who cares about quality and wants to solve testing problems that most teams haven't figured out yet, this is the role.

Responsibilities

• Define testing strategies for AI agents, conversational interfaces, and agentic workflows
• Design behavioral test suites for non-deterministic outputs - where "correct" isn't binary
• Build evaluation frameworks that measure groundedness, factuality, relevance, and task completion
• Identify failure modes specific to AI systems: hallucinations, prompt injection, context window limitations, drift
• Develop testing approaches for each architecture pattern: RAG, function calling, human-in-the-loop, autonomous workflows

Test Automation & Infrastructure

• Build automated evaluation pipelines that run as part of