AI Evaluation Architect

Allegis Group Singapore Pte Ltd; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Allegis Group Singapore Pte Ltd is seeking an AI Evaluation Engineer to understand, analyze, and improve performance of advanced AI systems. You will design evaluation methodologies, analyze model outputs, and translate findings into actionable recommendations for stakeholders.

The role involves leading evaluation suites for LLMs, developing scoring frameworks, and driving improvements in prompt engineering, RAG strategies, and model fine-tuning. Strong ML and MLOps skills are essential.

Qualifications

• Bachelor's or Master's degree in Computer Science, ML, AI, or related field.
• Experience in ML Engineering or Applied Research.
• Proficiency in Python and deep learning ecosystems (PyTorch, JAX, Transformers).
• Experience building scalable ML inference pipelines and evaluation workflows.
• Ability to interpret unstructured model outputs and translate findings into engineering guidance.
• Hands-on experience with LLMs, multimodal models, and NLP systems.
• Familiarity with AI quality metrics, hallucination detection, and RLHF/DPO alignment.
• Experience building internal tools for ML workflows (MLflow, Weights & Biases).
• Strong knowledge of prompt engineering, RAG architectures, and fine-tuning.

Responsibilities

• Lead evaluation suites for LLMs and multimodal models, identifying edge cases in reasoning, factuality, robustness, safety, and alignment.
• Develop deterministic, heuristic, and LLM-assisted evaluation frameworks to quantify quality metrics.
• Translate failure modes into measurable loss patterns and actionable data adjustments for training and inference.
• Collaborate with engineering teams to refine model behavior using evaluation telemetry for prompt engineering and RAG strategies.
• Apply advanced ML techniques to map error taxonomies and latent failure manifolds in outputs.
• Develop robust MLOps workflows to codify metrics and automate regression testing across checkpoints.
• Architect distributed evaluation pipelines for high-throughput assessment and annotation at scale.
• Define human-centric metrics like trust calibration and interpretability.
• Build automated evaluation pipelines using LLMs to align with human annotations.
• Collaborate with ML researchers, software developers, and product managers to translate requirements into scalable evaluation infrastructure.

Skills

Python

Deep learning

PyTorch

JAX

Hugging Face

LLMs

NLP

MLOps

RAG

RLHF

Education

Master's degree in computer science or related field

Tools

MLflow

Weights & Biases

Vector DBs

Ray

vLLM

Job description

Allegis Group Singapore Pte Ltd is seeking an AI Evaluation Engineer to understand, analyze, and improve performance of advanced AI systems. You will design evaluation methodologies, analyze model outputs, and translate findings into actionable recommendations for stakeholders.

The role involves leading evaluation suites for LLMs, developing scoring frameworks, and driving improvements in prompt engineering, RAG strategies, and model fine-tuning. Strong ML and MLOps skills are essential.