Quality Analyst

Winjit Technologies; Mumbai, Hyderabad; On-site; INR 600,000 - 900,000; Full time

Job summary

Winjit Technologies is seeking a Quality Analyst - AI/ML (Level 1) to design and run automated evaluation strategies for agentic AI applications, ensuring robust performance and safety across models. The role focuses on dataset creation, test case development, and bias detection to improve reliability.

The ideal candidate has 1–3 years of experience, strong Python skills, and hands-on work with AI evaluation frameworks and testing pipelines in a work-from-office setup in Mumbai/Hyderabad.

Qualifications

• Must have hands-on exp with Generative AI, LLMs, and AI systems.
• Strong experience in Python and AI/ML development libraries.
• Proficiency in prompt engineering, testing, and optimization.
• Experience building automated evaluation pipelines and benchmarking.
• Knowledge of AI safety, guardrails, bias testing, and responsible AI practices.

Responsibilities

• Design, execute, and automate evaluation strategies for Agentic AI applications.
• Develop evaluation datasets, test cases, and benchmark suites to measure agent performance.
• Analyze outputs for hallucinations, bias, safety risks, and provide actionable recommendations.

Skills

Generative AI

LLMs

Agentic AI

AI Evaluation

Python

Python ML Libs

Prompt Engineering

Evaluation Pipelines

AI Safety

REST APIs

Vector Databases

Knowledge Retrieval

Agent Reasoning

Tool Calling

Workflow Execution

Automation Testing

Tools

RAGAS

DeepEval

TruLens

LangSmith

Promptfoo

VectorDBs

Job description

Role -Quality Analyst - AI/ML (Level 1)

Experience - 1 to 3 years

Location - Mumbai / Hyderabad (Work from Office )

Must-have skills -

• Generative AI, LLMs, and Agentic AI Systems
• AI Evaluation Frameworks (RAGAS, DeepEval, TruLens, LangSmith, Promptfoo, etc.)
• Python and AI/ML Development Libraries
• Prompt Engineering, Testing, and Optimization
• Automated Evaluation Pipelines and Benchmarking Frameworks
• AI Safety, Guardrails, Bias Testing, and Responsible AI Practices
• REST APIs, JSON, Vector Databases, and Knowledge Retrieval Systems
• Manual Testing of AI/GenAI Applications (Functional, Exploratory, UAT, Regression, E2E)
• Agent Reasoning, Tool Calling, Workflow Execution, and Response Validation
• Automation Testing using Python Frameworks

Good to Have Skills-

• A/B Testing
• Human-in-the-Loop Evaluation
• Red Teaming
• Evaluation Reporting & Dashboarding
• Exposure to RAG-based Applications

Top 3 Responsibilities -

1. 1. Design, execute, and automate evaluation strategies for Agentic AI applications.
2. 2. Develop evaluation datasets, test cases, and benchmark suites to measure agent performance.
3. 3. Analyze outputs for hallucinations, bias, safety risks, and provide actionable recommendations.