AI Model Evaluation & Benchmark Scientist

Tawteen; Muscat; On-site; OMR 12,000 - 18,000; Full time

Job summary

Tawteen is seeking a data science professional to design evaluation methods and build test data sets for evaluating AI models. You will run evaluations, benchmark models, and analyze errors in safety and privacy filters across multilingual and adversarial prompts.

The role involves building predictive and classification models on tabular/time-series data and integrating evaluation tests into CI pipelines. Skills in Python, pandas, NumPy, scikit-learn, SQL, and Git are essential, along with

Qualifications

• Degree in data science, statistics, computer science or a related field.
• Proficiency in Python with pandas, NumPy and scikit-learn.
• Knowledge of statistics: precision, recall, confidence intervals, hypothesis testing.
• SQL and Git proficiency.

Responsibilities

• Design evaluation methods and build test data sets including multilingual and adversarial prompts.
• Run evaluations and score results with automated metrics and LLM-as-judge.
• Benchmark models on quality, latency and cost and recommend which to use.
• Analyse errors including false positives and false negatives of safety and privacy filters.
• Build and validate predictive and classification models on tabular, time-series and signal data.
• Add evaluation tests to CI pipelines to prevent regressions.
• Analyse usage cost and quality data and build reports and dashboards.

Skills

Python

Pandas

NumPy

scikit-learn

Statistics

Precision

Recall

Confidence intervals

Hypothesis testing

SQL

Git

Education

Data Science degree

Tools

PyTorch

TensorFlow

MLflow

Weights & Biases

Langfuse

promptfoo

Job description

Tawteen is seeking a data science professional to design evaluation methods and build test data sets for evaluating AI models. You will run evaluations, benchmark models, and analyze errors in safety and privacy filters across multilingual and adversarial prompts.

The role involves building predictive and classification models on tabular/time-series data and integrating evaluation tests into CI pipelines. Skills in Python, pandas, NumPy, scikit-learn, SQL, and Git are essential, along with