Gen AI Evaluators

Monster.com SG Pte Limited; Singapore; On-site; SGD 60,000 - 80,000; Full time

Job summary

A leading company is seeking a professional to enhance Generative AI accuracy. The role involves maintaining a knowledge base, evaluating AI-generated responses, and collaborating with product owners. Candidates should demonstrate attention to detail, possess analytical skills, and have a basic understanding of data processing in AI systems. Proficiency in MS Office is required.

Qualifications

• Meticulous and show great attention to detail.
• Possess an analytical mindset to evaluate performance metrics.
• Basic understanding of prompt engineering and data techniques is a plus.

Responsibilities

• Build and maintain a knowledge base for Generative AI.
• Evaluate AI responses and collaborate with development teams.
• Establish safety scenarios to mitigate risks in AI interactions.

Skills

Attention to Detail

Analytical Mindset

Prompt Engineering Understanding

Data Pre-processing Skills

Education

Experience in AI-related fields

Tools

MS Office

Job description

Enhance Generative AI Accuracy:

1) Build and curate a comprehensive knowledge base to serve as a foundation for the Generative AI model.

2) Evaluate and analyse AI-generated responses to identify areas of improvement and optimize accuracy.

Response Analysis and Evaluation:

1) Identify, analyse, and assess the quality of AI-generated responses, ensuring they align with company standards and customer expectations.

2) Collaborate with the AI Development team to implement necessary adjustments to improve response quality.

Safety Scenario Establishment:

1) Develop and implement safety scenarios to mitigate the risk of inappropriate or sensitive AI responses, maintaining ethical and responsible AI interactions.

Data Set Maintenance:

1) Deconflict and curate data sets across various platforms to ensure a diverse and clean input for the Generative AI model.

2) Regularly update and refine data sets to reflect evolving language trends and user preferences.

Knowledge Base Updates:

1) Perform daily updates to the knowledge base, integrating the latest product information, industry updates, and relevant content.

2) Ensure accuracy and relevance of information provided by collaborating with product owners and subject matter experts.

Demand Analysis:

1) Engage in call listening activities to identify emerging demand topics and customer preferences.

2) Provide insights and recommendations to the AI Project Manager for enhancing AI-generated responses in response to these demands.

Collaboration with Product Owners:

1) Establish effective communication channels with product owners to continuously enhance the knowledge base and refine AI responses.

2) Actively gather feedback and insights from product owners to improve the accuracy and relevance of AI interactions.

Product Knowledge & Call Handling Trained

1) To be trained in Banking Related product knowledge and undergo training to learn how a customer service officer handles a customer over a call.

2) Standby support to be activated on adhoc basis by Workforce Management team to service calls/emails or any other functions deemed to be necessary to maintain Customer Centre's service levels.

These are the experience and skillsets that we are generally looking for:

1. Meticulous and show great attention to detail. (Important)

2. Possess an analytical mindset to identify key performance metrics and evaluate the success of Gen AI initiatives. (Important)

3. Basic understanding of prompt engineering, data pre-processing, and data ingestion techniques in the context of Generative AI systems. (Good to have but not essential)

4. Proficient in MS Office (Word, Excel, Project and PowerPoint)