Lead Quality Engineer – R01573517

Brillio

Bengaluru

On-site

INR 3,000,000 - 5,400,000

Full time

7 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Brillio is seeking a Lead Quality Engineer in Bengaluru to lead QA for AI-powered product applications and microservices. The role focuses on building robust evaluation frameworks, synthetic testing environments, and safety-focused assessments across AI agents.

You will implement adversarial testing, integrate pipelines with CI/CD, and collaborate with policy teams to ensure ethical and compliant practices. Strong Python, PyTest, and AI testing toolkit skills are required.

Qualifications

  • Experience 4–6 years in quality engineering for AI-powered products and microservices.
  • Hands-on testing and evaluation of AI agents and systems.
  • Ability to design evaluation frameworks with safety metrics (bias, toxicity, alignment).
  • Experience integrating tests into CI/CD pipelines.

Responsibilities

  • Develop evaluation frameworks for AI agents focusing on safety metrics.
  • Design synthetic testing environments and stress-test strategies.
  • Implement adversarial testing and red-teaming methods.
  • Integrate evaluation pipelines into CI/CD workflows.
  • Collaborate with policy and legal teams for regulatory compliance.
  • Analyze results and provide actionable risk insights to teams.
  • Configure automated test suites using Python and AI testing libraries.
  • Manage bug detection, documentation, and Jira dashboards.

Skills

Python
PyTest
DeepEval
Ragas
Giskard
Langsmith
ADK
Vertex AI
API testing
Pandas
SQL
Jira
Prompt injection
Adversarial testing
CI/CD

Education

Bachelor's degree in Computer Science / IT / AI
ISTQB Foundation Level
AI/ML testing certification (preferred)

Tools

Giskard
DeepEval
Ragas
Gemini models
Hugging Face
Google ADK
Vertex AI

Job description

Lead Quality Engineer - R01573517

Bangalore, India

AI & Data Engineering

Lead Quality Engineer

Primary Skills
  • Test Data Management, Jenkins, Regression Testing, Postman, Selenium, Functional Testing, BDD – Behavior Driven Development, SoapUI, System Integration testing, Application Testing, Pixel Testing, Cross Browser Testing, Zephyr, TDD – Test Driven Development, Accessibility Testing (JAWS), Visual Testing, Cucumber
Specialization
  • Quality Engineering – Product Applications & Microservices: Lead Quality Engineer
Job requirements
  • Experience Range: with at least 4 to 6 years of experience in quality engineering, including hands-on testing and evaluation of AI-powered product applications and microservices Key Responsibilities: 1. Develop and execute robust evaluation frameworks for AI agents, focusing on Responsible AI and safety metrics such as bias, toxicity, and alignment failures 2. Design synthetic testing environments and simulation strategies to stress-test agents under diverse real-world conditions 3. Implement adversarial testing, red-teaming, and prompt injection methodologies to identify vulnerabilities and ensure agent robustness 4. Integrate evaluation pipelines into CI/CD workflows for continuous quality assurance and rapid iteration 5. Collaborate with policy and legal teams to ensure agent evaluations meet regulatory compliance and ethical standards 6. Analyze evaluation results and provide actionable insights to development teams, prioritizing safety and risk mitigation 7. Configure and execute automated test suites using Python and relevant AI/ML testing libraries (e.g., PyTest, DeepEval, Ragas, Giskard) 8. Manage bug detection, documentation, and tracking using Jira, focusing on performance degradations and alignment failures Required Skills: 1. Python programming 2. PyTest 3. DeepEval 4. Ragas 5. Giskard 6. Langsmith 7. Google Agent Development Kit (ADK) 8. Vertex AI 9. Safety evaluations (red teaming, adversarial testing) 10. Prompt injection testing 11. API testing for AI/ML systems 12. Data analysis with Pandas and SQL Preferred Skills: 1. Gemini models 2. Hugging Face 3. Garbage-in/Garbage-out (GIGO) testing tools 4. Data poisoning techniques 5. AI ethics principles and policy background 6. Visualization tools for evaluation results Desired Qualifications: 1. Bachelor’s degree in Computer Science, Artificial Intelligence, Information Technology, or Software Engineering 2. ISTQB Foundation Level certification or Certified Software Tester (CST) 3. Certification in AI/ML testing or Responsible AI (e.g., Google Cloud Certified – Professional Machine Learning Engineer)
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI or ML Engineer - Python, QA Testing
AI or ML Engineer - Python, QA Testing

Optum • Hyderabad

On-site
INR 1,800,000 - 3,200,000
AI Senior QA
AI Senior QA

Hiver, Inc. • Bengaluru

On-site
INR 2,200,000 - 3,400,000
Senior AI Testing Engineer (Generative AI)
Senior AI Testing Engineer (Generative AI)

Wfnen • Bengaluru

Hybrid
INR 1,200,000 - 1,600,000
Exciting Opportunity - AI Evaluation Testing Lead / Expert - Bengalaru
Exciting Opportunity - AI Evaluation Testing Lead / Expert - Bengalaru

QualityKiosk • Navi Mumbai, Mumbai, Bengaluru

On-site
INR 4,000,000 - 8,000,000
Lead Software Development Engineer in Test - AI
Lead Software Development Engineer in Test - AI

Jobgether • India

Remote
INR 4,000,000 - 7,000,000
Fully remote (India)
Competitive salary
Leadership opportunities
+2
Associate Principal - Quality Engineering (AI-Driven Test Automation)
Associate Principal - Quality Engineering (AI-Driven Test Automation)

Vivantify Technology Solutions India Private Limited • Bengaluru

Hybrid
INR 3,500,000 - 6,000,000
Competitive salary
Professional growth opportunities
Collaborative work environment
+2
Senior AI QA Engineer (Pune )
Senior AI QA Engineer (Pune )

Money Forward India • Pune District

On-site
INR 2,800,000 - 4,000,000
Lead Quality Engineer
Lead Quality Engineer

The Hartford India • Telangana

On-site
INR 500,000 - 900,000
Advanced AI tech
Collaborative environment
Competitive compensation
+1
AI Senior QA
AI Senior QA

Hiver • Bengaluru

On-site
INR 1,500,000 - 2,200,000
AI Data & Quality Analytics Tester(Bangalore only)
AI Data & Quality Analytics Tester(Bangalore only)

PwC • Bengaluru

On-site
INR 900,000 - 1,300,000