AI Test Engineer - VLabs

Vialto Partners

Bengaluru

Hybrid

INR 900,000 - 1,350,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Vialto Labs (VLabs) Bangalore seeks a senior AI Test Engineering professional to validate AI-enabled solutions across LLMs, OCR pipelines, and agentic workflows. You will design evaluation pipelines, build reusable frameworks, and embed QA checks into production deliveries.

The role requires strong Python skills, experience with AI evaluation tools, data quality, drift detection, and API testing, with a focus on responsible AI and governance.

Qualifications

  • 7+ years in software testing, incl. 2–3 years AI/ML-enabled systems in production
  • Experience designing AI evaluation frameworks and quality strategies
  • Track record building ground truth datasets and drift detection systems
  • Experience testing multi-step agentic workflows and AI-driven automation
  • Experience in fast-paced, iterative delivery environments
  • Regulated/compliance-focused background preferred

Responsibilities

  • Translate AI testing strategy into executable test scenarios across LLM outputs and document AI workflows
  • Build reusable evaluation frameworks and parameterized test scripts
  • Embed evaluation frameworks into CI/CD pipelines for continuous testing
  • Design and operate drift-detection and data-quality checks
  • Validate AI outputs for structure, accuracy, and production readiness
  • Coordinate with engineering, product, and delivery teams to ensure model readiness and risk assessment
  • Communicate findings to stakeholders with actionable insights
  • Support governance and responsible AI initiatives across VLabs

Skills

Python
AI evaluation
QA automation
SQL
Postman/Newman
LangChain
LlamaIndex
Azure AI Foundry
AWS Bedrock

Education

Bachelor’s degree
Advanced degree in CS/Data Science

Tools

deepeval
RAGAS
promptfoo
HTTP clients (requests/httpx)

Job description

Company Description

Join the leader in global work. Vialto helps leading organizations identify, develop, and execute their workforce priorities with confidence. Our practice areas include global tax, immigration, rewards, advisory and Vialto private and our team is the gold standard in navigating the complexities of cross-border compliance, and workforce strategies. We look for people our clients and assignees can count on to thoughtfully deliver clear solutions to complex problems.

To be the best in the business, we cultivate a dynamic, highly collaborative culture that fuels the growth of our people. We invite you to inquire boldly, learn deeply, and grow endlessly with us. You’ll work alongside recognized experts, shaping people-first, technology-driven solutions that drive the future of work. We help our clients achieve their goals and imagine new possibilities. And we can do the same for you. Whatever your aspirations, Vialto can help you find your way.

About Vialto Labs (VLabs)

Vialto Labs (VLabs)is responsible forredesigning how work is delivered in the tax and immigration service lines, as well as driving operational efficiency acrossVialto’sfunctional areas using AI. The team builds and deploys novel AI-enabled solutions that directly improve productivity and increase delivery quality for our clients.VLabsis accountable for rapidly turning innovative experiments into production-ready deliverables at scale and embedding them into day-to-day operations. This team focuses on the highest-impact workflows, creating standardized, repeatable capabilities that can be deployed globally. Operating with a mandate for speed and measurable outcomes,VLabsworks alongside serviceline,product, and platform leaders.

About the Role

AI TestEngineeringis a hands-on role withinVLabsQuality Engineering, responsible forvalidatingthe performance, reliability, and integrity of AI-enabled solutions in production environments.This roleoperatesat the intersection of AI engineering and quality assurance, ensuring that outputs from LLMs, OCR pipelines, document classification models, and agentic workflows perform as expected at scale and meet defined business performance thresholds.Working closely with theProgrammeTest Manager and partnering with engineering, product, and delivery teams, this role translates AI testing strategy into executable frameworks, evaluation pipelines, and reusable assets embedded into the delivery lifecycle.

Success requires independent execution, strong technical depth, and the ability to proactivelyidentifyrisks, patterns, and performance gaps while enabling rapid, production-grade deployment of AI capabilities.

Key Responsibilities
AI Evaluation & Test Design
  • Translate AI testing strategy into executable test scenarios across LLM outputs, document classification, extraction accuracy, agent workflows, and edge cases
  • Design adversarial and boundary test inputs to expose hallucination, misclassification, and failure modes
  • Validate AI outputs for structure, consistency, accuracy, and production readiness against defined performance thresholds
Evaluation Engineering & Automation
  • Build reusable Python-based evaluation frameworks, including output validation, hallucination detection, and scoring mechanisms
  • Develop parameterized test scripts reusable across features, models, and releases
  • Implement AI-as-Judge frameworks, including prompt design, scoring logic, and calibration of evaluation reliability
  • Embed evaluation frameworks into CI/CD pipelines to support continuous testing and deployment
Drift Detection & Quality Monitoring
  • Designandoperatedriftdetection frameworks using fixed baseline datasets and scheduled re-evaluation
  • Establish thresholds to distinguish acceptable variation from performance degradation
  • Enable release gatingbyidentifyingregressionsprior to production deployment
Ground Truth & Data Quality
  • Buildandmaintaingroundtruth datasets in partnership with subject matter experts
  • Define standards for classification, extraction accuracy, and acceptable outputcharacteristics
  • Continuously update datasets to reflect evolving business requirements and use cases
Workflow & Integration Testing
  • Test end-to-end agenticworkflows,validatingdataintegrity, error propagation, and fallback behavior
  • Perform API-level testing of AI pipeline endpoints using Python and Postman/Newman
  • Validate data persistence and integrity across system layers using SQL
  • Partner with engineering teams to ensure testability, observability, and system reliability
Standardization & Scaling
  • Define and scale standardized AI evaluation patterns and reusable quality frameworksacrossVLabs
  • Contribute to enterprise AI quality standards andreferencearchitectures
Governance & Responsible AI
  • Ensure adherence to Responsible AI, data privacy, and governance requirements
  • Support auditability, traceability, and transparency of AI outputs and evaluation processes
Stakeholder Enablement
  • Translate evaluation results into actionable insights for engineering, product, and business stakeholders
  • Support decision-making on model readiness, release risk, and performance trade-offs
  • Proactivelyidentifyrisks,patterns, and systemic issues and upscale appropriately
Qualifications & Experience
Professional Experience
  • 7+ years in software testing, including 2–3 years focused on AI/ML-enabled systems in production environments
  • Proven experience designing and executing AI evaluation frameworks and quality strategies
  • Strongtrack recordbuilding ground truth datasets, drift detection systems, and scalable evaluation pipelines
  • Experience testing multi-step agentic workflows and AI-driven automation systems
  • Experienceoperatinginfast-paced, iterative delivery environments
  • Background in regulated or compliance-driven environments preferred
Technical Expertise
  • Advanced Python programming for evaluation frameworks, batch processing, and data analysis
  • Experience with LLM evaluation tools such asdeepeval,RAGAS,promptfoo,or similar
  • Strong capabilities in:
  • AI output validation, hallucination detection, and grounding checks
  • Drift detection frameworks and statistical evaluation methods
  • OCR, VLM, and document AI testing (classification, extraction, edge cases)
  • API testing using Python (requests/httpx) and Postman/Newman
  • SQL for data validation and pipeline integrity checks
  • FamiliaritywithLangChain,LlamaIndex,or similar frameworks
  • Experience with cloud AI platforms such as Azure AI Foundry or AWS Bedrock preferred
Operating Capabilities
  • Abilitytooperateindependentlyin fast-moving, ambiguous environments
  • Strong analytical mindset with attention to detail and quality rigor
  • Ability to balance speed and rigor in AI evaluation and delivery cycles
  • Proactive communicatorwhoidentifiesrisksand drives resolution
  • Ability to translate technical findings into business-relevant insights
Education
  • Bachelor’s degreerequired
  • Advanced degree in Computer Science, Data Science, or related field preferred
Additional Information:
  • This job is based in our Bangalore office with the possibility of hybrid work mode
  • We are an equal opportunity employer that does not discriminate on the basis of any legally protected status
  • Please note, AI is used as part of the application process
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Test Engineer - VLabs
AI Test Engineer - VLabs

Vialto UK & Ireland • Bengaluru

Hybrid
INR 3,000,000 - 4,500,000
Hybrid work mode
Senior Associate - AI Engineer
Senior Associate - AI Engineer

Vialto UK & Ireland • Bengaluru

On-site
INR 700,000 - 1,100,000
Senior Associate - AI Engineer
Senior Associate - AI Engineer

Vialto Partners • Bengaluru

Remote
INR 4,000,000 - 7,000,000
Manager- AI Engineer
Manager- AI Engineer

Mployee.me • Bengaluru

On-site
INR 2,500,000 - 4,500,000
Software Engineering - Senior Manager
Software Engineering - Senior Manager

Vialto Partners • Bengaluru

Hybrid
INR 1,800,000 - 3,000,000
AI Test Engineer
AI Test Engineer

AXA Global Business Services (AXA GBS) • Pune District

On-site
INR 900,000 - 1,500,000
AI/ML TESTER
AI/ML TESTER

Vrinda International • Bengaluru

On-site
INR 900,000 - 1,100,000
Quality Assurance Engineer – AI Agentic Applications, AVP
Quality Assurance Engineer – AI Agentic Applications, AVP

Deutsche Bank AG • Pune District

On-site
INR 2,400,000 - 3,800,000
Best in class leave policy
Gender neutral parental leaves
Childcare assistance reimbursement
+5
AI - Software Development Engineer in Test (AI - SDET)
AI - Software Development Engineer in Test (AI - SDET)

Mployee.me • Mumbai

On-site
INR 1,200,000 - 1,800,000
Quality Assurance Engineer – AI Agentic Applications, AVP
Quality Assurance Engineer – AI Agentic Applications, AVP

Deutsche Bank AG • Maharashtra

On-site
INR 1,500,000 - 2,000,000
Leave policy
Parental leaves (gender neutral)
Childcare assistance
+5