Quality Assurance Engineer

Straive; Bengaluru; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Straive is seeking an QA engineer specialized in AI/ML quality assurance to design comprehensive test strategies for AI/LLM systems and data pipelines. The role emphasizes Python-based automation, evaluating RAG outputs for accuracy and bias, building robust evaluation datasets, and collaborating with Data Scientists to define acceptance criteria and performance thresholds.

You will monitor production models and ensure data quality throughout the ML lifecycle.

Qualifications

• 3 to 6 years of core QA experience, with 1 to 2 years specialized in AI/ML quality assurance and LLM testing.
• Bachelor's or Master's degree in Computer Science, Engineering, or a closely related technical field.
• Strong hands-on coding in Python for data analysis and test automation.
• Familiarity with LLM evaluation frameworks (RAGAS, DeepEval, Promptfoo, LangSmith) alongside traditional QA tools (Pytest, Selenium, Postman).

Responsibilities

• Design and execute test strategies for AI/ML models, LLM-based applications, and data pipelines to ensure system accuracy and reliability.
• Develop and maintain automated test frameworks for model validation, regression testing, and performance benchmarking using Pytest.
• Evaluate LLM and RAG outputs for structural correctness, factual consistency, relevance, hallucination risks, and demographic bias.
• Build, update, and manage robust evaluation datasets, golden/ground truth sets, and adversarial edge‑case test suites.
• Monitor production models to track performance metrics, identify data/model drift, and catch anomalous behaviors over time.
• Validate incoming data streams, data pipelines, and feature stores to confirm completeness and accuracy of training/inference data.
• Collaborate directly with Data Scientists and Machine Learning Engineers to define explicit acceptance criteria, quality thresholds, and prompt regression standards.
• Conduct ethical and compliance assessments, including bias audits and model explainability checks.
• Document distinct defects, edge cases, and failure patterns specific to non-deterministic AI behaviors.

Skills

AI/ML QA

Python

Test automation

Data quality

Education

Bachelor's or Master's in CS/Engineering

Tools

RAGAS

DeepEval

Promptfoo

LangSmith

Pytest

Selenium

Postman

Job description

Role & responsibilities

• Design and execute comprehensive test strategies tailored for AI/ML models, LLM-based applications, and data pipelines to ensure system accuracy and reliability.
• Develop and maintain automated test frameworks for model validation, regression testing, and performance benchmarking using Pytest.
• Evaluate LLM and RAG outputs for structural correctness, factual consistency, relevance, hallucination risks, and demographic bias.
• Build, update, and manage robust evaluation datasets, golden/ground truth sets, and adversarial edge‑case test suites.
• Monitor production models to track performance metrics, identify data/model drift, and catch anomalous behaviors over time.
• Validate incoming data streams, data pipelines, and feature stores to confirm completeness and accuracy of training/inference data.
• Collaborate directly with Data Scientists and Machine Learning Engineers to define explicit acceptance criteria, quality thresholds, and prompt regression standards.
• Conduct ethical and compliance assessments, including bias audits and model explainability checks.
• Document distinct defects, edge cases, and failure patterns specific to non-deterministic AI behaviors.

Preferred profile

Experience: 3 to 6 years of core QA experience, with a mandatory 1 to 2 years specialized in AI/ML quality assurance and LLM testing.

Education: Bachelor's or Master's degree in Computer Science, Engineering, or a closely related technical field.

Technical Expertise: Strong hands‑on coding in Python for data analysis and test automation.

Tools Knowledge: Proven familiarity with LLM evaluation frameworks (RAGAS, DeepEval, Promptfoo, LangSmith) alongside traditional QA tools (Pytest, Selenium, Postman).

Data Skills: Working knowledge of data quality frameworks such as Great Expectations or dbt test structures.

Domain Context: Solid understanding of the Machine Learning lifecycle (training, validation, deployment, monitoring) and RAG system architectures.

Straive is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All employment decisions are based on business needs, job requirements, and individual qualifications, without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran, or disability status.