LLM Evaluation Engineer: Automated Benchmarks & Analytics

Kuailu Software; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

Kuailu Software is seeking an experienced evaluation engineer to build and maintain automated LLM evaluation pipelines. You will cover general, agent, and persona-based benchmarks, enabling one-click evaluation, historical comparisons, and regression testing.

You will deploy and run benchmarks like MMLU, C-Eval, HumanEval, GSM8K, MATH, and IFEval, while tracking metrics for BFCL, τ-bench, and GAIA. Strong Python and evaluation framework experience are essential.

Qualifications

• Bachelor's degree or above in Computer Science, Artificial Intelligence, or a related field.
• Familiarity with mainstream LLM evaluation benchmarks and frameworks, such as lm-eval-harness, OpenCompass, and EvalPlus.
• Strong proficiency in Python, with the ability to independently build evaluation pipelines covering model inference/deployment, batch evaluation, and results analysis.

Responsibilities

• Build and maintain an automated LLM evaluation pipeline covering multiple dimensions, including general capabilities, Agent capabilities, and persona/role-playing. The pipeline should support one-click evaluation, historical result comparison, and regression testing.
• Conduct general capability evaluations using benchmarks such as MMLU, C-Eval, HumanEval, GSM8K, MATH, and IFEval, including benchmark deployment, execution, and results analysis.
• Conduct Agent capability evaluations, including setting up evaluation environments and tracking metrics for benchmarks such as BFCL, τ-bench, and GAIA.
• Design and execute persona/role-playing evaluation frameworks, covering metrics such as identity recognition, role compatibility, multi-turn stability, and style consistency.
• Record and analyze evaluation results from training runs, conduct comparative analysis and anomaly detection, and produce checkpoint evaluation reports.
• Conduct regular intermediate evaluations during the pre-training stage to track the evolution and improvement of model capabilities.

Skills

Python

LLM evaluation

Benchmark frameworks

CI/CD integration

Data visualization

Education

Bachelor's degree in Computer Science / AI

Tools

lm-eval-harness

OpenCompass

EvalPlus

vLLM

SGLang

Job description

Kuailu Software is seeking an experienced evaluation engineer to build and maintain automated LLM evaluation pipelines. You will cover general, agent, and persona-based benchmarks, enabling one-click evaluation, historical comparisons, and regression testing.

You will deploy and run benchmarks like MMLU, C-Eval, HumanEval, GSM8K, MATH, and IFEval, while tracking metrics for BFCL, τ-bench, and GAIA. Strong Python and evaluation framework experience are essential.