LLM Evaluation Engineer – Multilingual AI Metrics

Nanyang Technological University; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

NTU Singapore’s AISG invites a capable researcher to join as an AI evaluation specialist with a focus on multilingual, multicultural, and multimodal LLM evaluation. You will build and maintain evaluation frameworks, collect and translate evaluation data, and collaborate with partners to ensure reliable AI solutions.

You should hold a degree in CS/AI/data science, be proficient in Python with Git, and have experience with PyTorch and vLLM.

Qualifications

• Degree in computer science, AI, data science or related field.
• Deep understanding of LLM evaluation experimental design and the advantages/limitations of various LLM evaluation methods.
• Experience in LLM inference frameworks such as vLLM and AI/Deep Learning frameworks such as PyTorch.
• Writing production level code in Python and using version control systems such as Git.
• Fluent in English and one other Southeast Asian language for multilingual evaluations.

Responsibilities

• Develop and maintain evaluation frameworks and pipelines to measure the capabilities of Large Language Models (LLMs).
• Keep up to date and experiment with the latest research in multilingual, multicultural, and multimodal LLM evaluations such as the LLM-as-a-Judge paradigm.
• Work with partners to collect, translate and verify evaluation datasets.
• Perform the necessary data preparation and analysis, AI modelling, coding, testing, validation and deployment to ensure reliable and scalable AI solutions.
• Collaborate with cross-functional teams within AI Products to design and resolve issues.
• Maintain code repository and documentation standards.
• Contribute to community engagement activities such as sharings via technical session meet-ups and article write-ups, and participating in discussion forums.

Skills

Communication skills

Multilingual capability

Independent learning

Research paper literacy

Python programming

Education

Degree in computer science/AI/data science or related field

Tools

vLLM

PyTorch

Git

Job description

NTU Singapore’s AISG invites a capable researcher to join as an AI evaluation specialist with a focus on multilingual, multicultural, and multimodal LLM evaluation. You will build and maintain evaluation frameworks, collect and translate evaluation data, and collaborate with partners to ensure reliable AI solutions.

You should hold a degree in CS/AI/data science, be proficient in Python with Git, and have experience with PyTorch and vLLM.