LLM Evaluation Engineer

Kuailu Software; Singapore; On-site; SGD 90,000 - 150,000; Full time

Benefits offered by this job

Agent evaluation experience

Persona/role-playing evaluation

Evaluation automation/CI-CD

Training workflow understanding

Job summary

Kuailu Software seeks an experienced ML/NLP Engineer to build and maintain automated evaluation pipelines for LLMs, covering general, agent, and persona evaluation aspects. You will support one-click evaluations, benchmark deployments, and results analysis across training runs.

The role requires Python proficiency, familiarity with lm-eval-harness/OpenCompass/EvalPlus, and experience with vLLM/SGLang inference frameworks. Strong focus on reproducibility and automation is essential.

Qualifications

• Bachelor's degree or above in Computer Science, Artificial Intelligence, or a related discipline.
• Familiar with mainstream LLM evaluation benchmarks and frameworks, such as lm-eval-harness, OpenCompass, and EvalPlus.
• Proficient in Python, with the ability to independently build evaluation pipelines covering model inference/deployment, batch evaluation, and results analysis.
• Familiar with LLM inference frameworks such as vLLM and SGLang, with the ability to deploy models for batch inference and evaluation.
• Experience in evaluation data analysis and visualisation.
• Detail-oriented and rigorous, with a strong focus on ensuring the reproducibility and reliability of evaluation results.

Responsibilities

• Build and maintain automated LLM evaluation pipelines covering multiple dimensions, including general capabilities, Agent capabilities, and persona/role-playing capabilities.
• Support one-click evaluation, historical result comparison, and regression testing.
• Conduct general capability evaluations using benchmarks such as MMLU, C-Eval, HumanEval, GSM8K, MATH, and IFEval, including deployment and results analysis.
• Conduct Agent capability evaluations, including environment setup and tracking performance metrics for benchmarks like BFCL, τ-bench, and GAIA.
• Design and execute persona and role-playing evaluation frameworks, covering identity recognition, role consistency, multi-turn stability, and style consistency.
• Record and analyse evaluation results across model training runs, perform comparisons and anomaly detection, and prepare evaluation reports for training checkpoints.
• Conduct regular intermediate evaluations during the pre-training stage to track evolution and improvement of model capabilities.

Skills

Python

Evaluation pipelines

Benchmarks

Result analysis

CI/CD

Education

Bachelor's degree in CS/AI

Tools

lm-eval-harness

OpenCompass

EvalPlus

vLLM

SGLang

Job description

Job Description

• Build and maintain automated LLM evaluation pipelines covering multiple dimensions, including general capabilities, Agent capabilities, and persona/role-playing capabilities. Support one-click evaluation, historical result comparison, and regression testing.
• Conduct general capability evaluations using established benchmarks such as MMLU, C-Eval, HumanEval, GSM8K, MATH, and IFEval, including benchmark deployment, execution, and results analysis.
• Conduct Agent capability evaluations, including setting up evaluation environments and tracking performance metrics for benchmarks such as BFCL, τ-bench, and GAIA.
• Design and execute persona and role-playing evaluation frameworks, covering metrics such as identity recognition, role consistency, multi-turn stability, and style consistency.
• Record and analyse evaluation results across model training runs, perform comparisons and anomaly detection, and prepare evaluation reports for training checkpoints.
• Conduct regular intermediate evaluations during the pre-training stage to track the evolution and improvement of model capabilities.

Job Requirements

• Bachelor's degree or above in Computer Science, Artificial Intelligence, or a related discipline.
• Familiar with mainstream LLM evaluation benchmarks and frameworks, such as lm-eval-harness, OpenCompass, and EvalPlus.
• Proficient in Python, with the ability to independently build evaluation pipelines covering model inference/deployment, batch evaluation, and results analysis.
• Familiar with LLM inference frameworks such as vLLM and SGLang, with the ability to deploy models for batch inference and evaluation.
• Experience in evaluation data analysis and visualisation.
• Detail-oriented and rigorous, with a strong focus on ensuring the reproducibility and reliability of evaluation results.

Good to Have

• Experience with Agent evaluation, particularly BFCL, τ-bench, GAIA, or SWE-bench.
• Experience with persona or role-playing evaluation, such as CharacterBench or RMTBench.
• Experience in evaluation automation and CI/CD integration.
• Understanding of LLM training workflows, with the ability to interpret the relationship between training checkpoints and evaluation results.