AI Quality Assurance Engineer

Tekskills; Pune District; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Tekskills is seeking a detail-oriented AI QA Engineer to join our AI Metadata Augmentation team. You will design evaluation strategies for LLM outputs and build automated tests that ensure accuracy and reliability on AWS-hosted systems.

You will collaborate with AI engineers to debug prompts, implement regression testing, and maintain Python-based testing pipelines for data validation and metadata quality.

Qualifications

• Experience in QA Automation / Data Quality / Software Testing.
• Understanding of AI lifecycle (Training vs Inference, Prompt Engineering).
• Proficiency in Python for test automation and data validation.
• Hands-on experience with JSON parsing and validation.
• Basic knowledge of AWS (CloudWatch, S3).

Responsibilities

• Design and execute LLM evaluation & benchmarking strategies (relevancy, consistency, metadata quality).
• Build and maintain automated testing frameworks using Python.
• Perform prompt regression testing to track impact of changes over time.
• Collaborate with AI engineers to debug edge cases and improve prompts/logic.

Skills

QA Automation

Data Quality

Software Testing

AI lifecycle understanding

Python for test automation

JSON parsing

AWS basics

Tools

Python Testing Frameworks

Job description

We are looking for a detail-oriented AI QA Engineer to join our AI Metadata Augmentation team. If you are passionate about ensuring the quality, accuracy, and reliability of AI/LLM outputs, this is a great opportunity to work on cutting-edge AI systems deployed on AWS.

Key Responsibilities

• Design and execute LLM evaluation & benchmarking strategies (relevancy, consistency, metadata quality)
• Build and maintain automated testing frameworks using Python
• Perform prompt regression testing to track impact of changes over time
• Collaborate with AI engineers to debug edge cases and improve prompts/logic

Required Skills & Experience

• Strong experience in QA Automation / Data Quality / Software Testing
• Understanding of AI lifecycle (Training vs Inference, Prompt Engineering)
• Proficiency in Python for test automation and data validation
• Hands-on experience with JSON parsing and validation
• Basic knowledge of AWS (CloudWatch, S3)