LLM Evaluation Automation Engineer

KUAILU SOFTWARE (SINGAPORE) PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

KUAILU SOFTWARE (SINGAPORE) PTE. LTD. is seeking a skilled engineer to build automated evaluation pipelines for large language models. The role focuses on general, agent, and persona-based evaluations using established benchmarks and tooling.

You will deploy and analyze results from MMLU, GSM8K, HumanEval, and more, while tracking performance across training runs. Strong Python skills and experience with vLLM/OpenCompass are preferred in a collaborative Singapore office.

Qualifications

• Bachelor's degree or above in Computer Science, Artificial Intelligence, or a related discipline.
• Familiar with mainstream LLM evaluation benchmarks and frameworks such as lm-eval-harness, OpenCompass, and EvalPlus.
• Proficient in Python with ability to independently build evaluation pipelines for model inference/deployment, batch evaluation and results analysis.
• Familiar with LLM inference frameworks such as vLLM and SGLang and deploying models for batch inference and evaluation.
• Experience in evaluation data analysis and visualisation.
• Detail-oriented and rigorous with focus on reproducibility and reliability of evaluation results.

Responsibilities

• Build and maintain automated LLM evaluation pipelines covering multiple dimensions including general capabilities, Agent capabilities, and persona/role-playing capabilities.
• Conduct general capability evaluations using benchmarks like MMLU, C-Eval, HumanEval, GSM8K, MATH, and IFEval including deployment, execution, and results analysis.
• Conduct Agent capability evaluations, setting up evaluation environments and tracking performance metrics for benchmarks such as BFCL, τ-bench, and GAIA.
• Design and execute persona and role-playing evaluation frameworks covering identity recognition, role consistency, multi-turn stability, and style consistency.
• Record and analyse evaluation results across model training runs, perform comparisons and anomaly detection, and prepare evaluation reports for training checkpoints.
• Conduct regular intermediate evaluations during the pre-training stage to track evolution and improvement of model capabilities.

Skills

Python

evaluation pipelines

benchmarking

data analysis

reproducibility

Education

Bachelor's degree or above in CS/AI or related

Tools

lm-eval-harness

OpenCompass

EvalPlus

vLLM

Job description

KUAILU SOFTWARE (SINGAPORE) PTE. LTD. is seeking a skilled engineer to build automated evaluation pipelines for large language models. The role focuses on general, agent, and persona-based evaluations using established benchmarks and tooling.

You will deploy and analyze results from MMLU, GSM8K, HumanEval, and more, while tracking performance across training runs. Strong Python skills and experience with vLLM/OpenCompass are preferred in a collaborative Singapore office.