LLM/NLP QA Engineer — Up to $9,000 + AWS

SCIENTEC CONSULTING PTE. LTD.; Singapore; On-site; SGD 60,000 - 80,000; Full time

Job summary

A consulting firm in Singapore is seeking an AI Quality Engineer to ensure the accuracy and performance of Large Language Models (LLMs) in GenAI applications. The role involves identifying inaccuracies, developing automated test scripts, and collaborating with AI teams to enhance model behaviour. Candidates must possess strong experience in LLM testing, proficiency in Python, and familiarity with NLP methodologies. The position offers dynamic working hours and a remuneration of up to $9,000 plus AWS.

Qualifications

• Experience testing LLMs for chatbots and AI.
• Strong Python skills for writing test scripts.
• Ability to document and track issues.

Responsibilities

• Design and execute test cases for LLM accuracy.
• Detect and analyse hallucinations in outputs.
• Develop automated test scripts to streamline regression testing.
• Conduct functional and non-functional testing.
• Evaluate model output quality using NLP metrics.
• Collaborate with AI engineers to improve model behaviour.
• Perform regression testing after updates.
• Maintain structured documentation for testing.
• Use issue tracking tools to report bugs.
• Apply LLMs and NLP knowledge for QA coverage.

Skills

Testing LLMs (e.g., GPT, BERT)

Test automation (PyTest, custom AI frameworks)

Accuracy evaluation methods for NLP

AI/NLP testing methodologies

Python programming

Issue tracking using Jira

Problem-solving skills

Tools

Jira

AWS

GCP

Azure

Job description

A consulting firm in Singapore is seeking an AI Quality Engineer to ensure the accuracy and performance of Large Language Models (LLMs) in GenAI applications. The role involves identifying inaccuracies, developing automated test scripts, and collaborating with AI teams to enhance model behaviour. Candidates must possess strong experience in LLM testing, proficiency in Python, and familiarity with NLP methodologies. The position offers dynamic working hours and a remuneration of up to $9,000 plus AWS.