Multilingual LLM Evaluation Engineer

Nanyang Technological University Singapore; Singapore; On-site; SGD 70,000 - 110,000; Full time

Job summary

AI Singapore, hosted at NTU, seeks a capable researcher/software engineer to build rigorous evaluations that test multilingual, multicultural, and multimodal capabilities of Large Language Models. You will develop evaluation pipelines, translate datasets, and validate AI solutions in collaboration with cross-functional teams.

The role requires production Python coding, strong communication, and the ability to read research papers. NTU hosts AISG and supports a culture of learning and innovation.

Qualifications

• Graduate in a degree program in computer science, AI, data science or related fields (or equivalent practical experience).
• Deep understanding of LLM evaluation experimental design and the advantages/limitations of various LLM evaluation methods.
• Experience in LLM inference frameworks such as vLLM and AI/Deep Learning frameworks such as PyTorch.
• Writing production level code in Python and using version control systems such as Git.
• Strong written and verbal communication skills.
• Independent learner that is capable of reading and understanding research papers.
• Fluent in English and one other Southeast Asian language for the purposes of understanding how to build quality evaluations from a multilingual and multicultural perspective.

Responsibilities

• Develop and maintain evaluation frameworks and pipelines to measure the capabilities of Large Language Models (LLMs).
• Keep up to date and experiment with the latest research in multilingual, multicultural, and multimodal LLM evaluations such as the LLM-as-a-Judge paradigm.
• Work with partners to collect, translate and verify evaluation datasets.
• Perform the necessary data preparation and analysis, AI modelling, coding, testing, validation and deployment to ensure reliable and scalable AI solutions.
• Collaborate with cross-functional teams within AI Products to design and resolve issues.
• Maintain code repository and documentation standards.
• Contribute to community engagement activities such as sharings via technical session meet-ups and article write-ups, and participating in discussion forums.

Skills

Python

Git

LLM evaluation

Multilingual proficiency

Education

Degree in CS/AI/Data Science

Tools

vLLM

PyTorch

Job description

AI Singapore, hosted at NTU, seeks a capable researcher/software engineer to build rigorous evaluations that test multilingual, multicultural, and multimodal capabilities of Large Language Models. You will develop evaluation pipelines, translate datasets, and validate AI solutions in collaboration with cross-functional teams.

The role requires production Python coding, strong communication, and the ability to read research papers. NTU hosts AISG and supports a culture of learning and innovation.