AI Evaluation Engineer: Stats-Driven Model Validation

Tek Systems; Singapore; On-site; SGD 70,000 - 110,000; Full time

Job summary

Allegis Group Singapore Pte Ltd is seeking an ML Evaluation Engineer/Data Scientist to bring statistical rigor to evaluating AI and LLM systems. You will design analyses, validate results, and build dashboards to monitor evaluation outcomes.

The role focuses on ensuring reliability, reproducibility, and scientific validity of evaluation results, and will partner with ML Evaluation Engineers and Applied Scientists to answer critical questions about performance improvements and evaluator alignment.

Qualifications

• Bachelor's degree in Statistics, Data Science, Computer Science, Mathematics, or related field.
• 2+ years of professional Data Science/Evaluation Science/Applied Statistics experience.
• Strong foundation in statistical analysis, hypothesis testing, experimental design, and measurement methodologies.
• Experience with Python, SQL, and data analysis tools; ability to present statistics to technical and non-technical stakeholders.
• Experience building analytical dashboards and reporting solutions.

Responsibilities

• Design and execute statistical analyses for AI and LLM evaluation programs.
• Measure and improve inter-annotator agreement between human evaluators.
• Develop sampling methodologies and evaluation datasets to ensure representative results.
• Perform significance testing and statistical validation of model performance changes.
• Analyze golden sets and benchmark datasets to assess model quality and consistency.
• Build dashboards and reporting frameworks to monitor evaluation outcomes and trends.
• Identify sources of evaluation noise, bias, and inconsistency.
• Partner with ML Evaluation Engineers to validate automated judges against human raters.
• Provide data-driven recommendations regarding evaluation quality and model readiness.

Skills

Statistical analysis

Hypothesis testing

Experimental design

Communication

Dashboarding

Education

Bachelor's degree (Statistics/Data Science/CS/Math)

Tools

Python

SQL

Jupyter

Job description

Allegis Group Singapore Pte Ltd is seeking an ML Evaluation Engineer/Data Scientist to bring statistical rigor to evaluating AI and LLM systems. You will design analyses, validate results, and build dashboards to monitor evaluation outcomes.

The role focuses on ensuring reliability, reproducibility, and scientific validity of evaluation results, and will partner with ML Evaluation Engineers and Applied Scientists to answer critical questions about performance improvements and evaluator alignment.