LLM Evaluation Engineer – Multilingual AI Systems

Nanyang Technological University Singapore; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

AI Singapore (AISG) is Singapore's national programme in artificial intelligence, launched by the National Research Foundation (NRF) to anchor deep national capabilities in AI.

Hosted at Nanyang Technological University (NTU), AI Singapore brings together Singapore-based research institutions and the vibrant ecosystem of AI start-ups and companies to perform use-inspired research, grow knowledge, create tools, and develop the talent to power Singapore's AI efforts.

Qualifications

• Graduate from a degree program in computer science, AI, data science or related fields (or equivalent practical experience).
• Deep understanding of LLM evaluation experimental design and the advantages/limitations of various LLM evaluation methods.
• Experience in LLM inference frameworks such as vLLM and AI/Deep Learning frameworks such as PyTorch.
• Writing production level code in Python and using version control systems such as Git.

Responsibilities

• Develop and maintain evaluation frameworks and pipelines to measure the capabilities of Large Language Models (LLMs).
• Keep up to date and experiment with the latest research in multilingual, multicultural, and multimodal LLM evaluations such as the LLM-as-a-Judge paradigm.
• Work with partners to collect, translate and verify evaluation datasets.
• Perform the necessary data preparation and analysis, AI modelling, coding, testing, validation and deployment to ensure reliable and scalable AI solutions.
• Collaborate with cross-functional teams within AI Products to design and resolve issues.
• Maintain code repository and documentation standards.
• Contribute to community engagement activities such as sharings via technical session meet-ups and article write-ups, and participating in discussion forums.

Skills

LLM evaluation

Python

PyTorch

Git

Communication

Independent learning

Multilingual ability

Education

Degree in CS/AI/data science or related

Tools

vLLM

Job description

AI Singapore (AISG) is Singapore's national programme in artificial intelligence, launched by the National Research Foundation (NRF) to anchor deep national capabilities in AI.

Hosted at Nanyang Technological University (NTU), AI Singapore brings together Singapore-based research institutions and the vibrant ecosystem of AI start-ups and companies to perform use-inspired research, grow knowledge, create tools, and develop the talent to power Singapore's AI efforts.