ML Evaluation Engineer - Statistical QA for AI/LLMs

Allegis Group Singapore Pte Ltd; Singapore; On-site; SGD 70,000 - 110,000; Full time

Job summary

Allegis Group Singapore Pte Ltd is seeking an ML Evaluation Engineer/Data Scientist to bring statistical rigor to AI and LLM evaluation. You will design analyses, validate model performance, and ensure reliable, reproducible results in collaboration with ML Evaluation Engineers and Applied Scientists. The role focuses on answering critical questions like: Is this improvement real?

Are evaluators aligned? Can we trust these results?

Qualifications

• Bachelor’s degree in Statistics, Data Science, CS, Mathematics, ML, or related field.
• Strong foundation in statistical analysis, hypothesis testing, experimental design, and measurement methods.
• Experience with structured and unstructured data using Python, SQL, and data analysis tools.
• Ability to communicate statistical findings clearly to technical and non-technical stakeholders.
• Experience building analytical dashboards and reporting solutions.

Responsibilities

• Design and execute statistical analyses for AI and LLM evaluation programs.
• Measure and improve inter-annotator agreement between human evaluators.
• Develop sampling methodologies and evaluation datasets to ensure representative results.
• Perform significance testing and statistical validation of model performance changes.
• Analyze golden sets and benchmark datasets to assess model quality and consistency.
• Build dashboards and reporting frameworks to monitor evaluation outcomes and trends.
• Identify sources of evaluation noise, bias, and inconsistency.
• Partner with ML Evaluation Engineers to validate automated judges against human raters.
• Provide data-driven recommendations regarding evaluation quality and model readiness.

Skills

Statistics

Data Science

Hypothesis testing

Experimental design

Communication

Dashboards

Education

Bachelor's degree in Statistics, Data Science, Computer Science, Mathematics, Machine Learning, or a related field

Master's degree or PhD in Statistics, Data Science, Computer Science, Quantitative Social Sciences, or a related discipline

Tools

Python

SQL

Job description

Allegis Group Singapore Pte Ltd is seeking an ML Evaluation Engineer/Data Scientist to bring statistical rigor to AI and LLM evaluation. You will design analyses, validate model performance, and ensure reliable, reproducible results in collaboration with ML Evaluation Engineers and Applied Scientists. The role focuses on answering critical questions like: Is this improvement real?

Are evaluators aligned? Can we trust these results?