QA Analyst

Winjit Technologies; Mumbai, Hyderabad; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Winjit Technologies in Mumbai/Hyderabad is seeking a Quality Analyst - AI/ML (Level 1) with 1 to 3 years of hands-on experience. The role focuses on evaluating GenAI applications, building test cases, and ensuring robust performance across AI/ML workflows.

You will design automated evaluation pipelines, assess outputs for bias and safety, and work with REST APIs, vector databases, and knowledge retrieval systems. Immediate joiners preferred; office-based work from Mumbai or Hyderabad.

Qualifications

• Must have hands-on experience with Generative AI and GenAI evaluation.
• Experience in evaluating AI systems using established frameworks.
• Proficiency in Python and AI/ML development libraries is required.

Responsibilities

• Design, execute, and automate evaluation strategies for Agentic AI applications.
• Develop evaluation datasets and test cases for AI systems.
• Analyze outputs for hallucinations, bias, safety risks and provide actionable recommendations.

Skills

Generative AI

LLMs

Agentic AI

Evaluation frameworks

Python libraries

Prompt engineering

Automation testing

REST APIs

Vector databases

Knowledge retrieval

Testing & optimization

Safety & bias testing

Agent reasoning

Tool calling

Python testing

Tools

RAGAS

DeepEval

TruLens

LangSmith

Promptfoo

Job description

Job description

Role

Quality Analyst - AI/ML (Level 1)

Experience

1 to 3 years

Location

Mumbai / Hyderabad (Work from Office )

IMMEDIATE JOINERS ONLY

Must-have skills

• Generative AI, LLMs, and Agentic AI Systems
• AI Evaluation Frameworks (RAGAS, DeepEval, TruLens, LangSmith, Promptfoo, etc.)
• Python and AI/ML Development Libraries
• Prompt Engineering, Testing, and Optimization
• Automated Evaluation Pipelines and Benchmarking Frameworks
• AI Safety, Guardrails, Bias Testing, and Responsible AI Practices
• REST APIs, JSON, Vector Databases, and Knowledge Retrieval Systems
• Manual Testing of AI/GenAI Applications (Functional, Exploratory, UAT, Regression, E2E)
• Agent Reasoning, Tool Calling, Workflow Execution, and Response Validation
• Automation Testing using Python Frameworks

Good to Have Skills

• A/B Testing
• Human-in-the-Loop Evaluation
• Red Teaming
• Evaluation Reporting & Dashboarding
• Exposure to RAG-based Applications

Top 3 Responsibilities

1. Design, execute, and automate evaluation strategies for Agentic AI applications.
2. Develop evaluation datasets, test cases, in .. etc..
3. Analyze outputs for hallucinations, bias, safety risks, and provide actionable recommendations.