Remote AI Evaluation Engineer

Joblogic Service Management Software; Lahore; On-site; PKR 900,000 - 1,500,000; Full time

Benefits offered by this job

Life Insurance

Medical Insurance (Family)

Provident Fund

Gym Facility

Remote Working (During Pandemic)

Company trip

29 Annual Leaves

Sick & uncapped Compassionate Leaves

Onsite with UK team

Job summary

Joblogic is building its AI Agent Platform and seeks an AI Evaluation Engineer to own model quality across channels and tools. You will define rubrics, design datasets, and implement release gates that guard every agent version in CI and live traffic.

You will collaborate with engineers, data, and product to ensure robust evaluations, actionable insights, and continuous improvement of agent performance in production. This is a hybrid role based in Lahore, Pakistan.

Qualifications

• 3+ years evaluating ML models.
• Strong Python engineering skills.
• Experience evaluating LLM-powered applications or AI agents.
• Experience grading tool-using agents and reporting reliability.
• Knowledge of ML evaluation metrics and drift monitoring.
• Data analysis using Pandas/NumPy/SQL.
• Awareness of AI safety and GDPR implications.
• Cross-functional collaboration with data/product teams.

Responsibilities

• Own release sign-off with a green regression suite and judge-scored evaluations.
• Run error analysis on production traces across channels.
• Build datasets and rubrics for multi-turn, tool-using agents.
• Build and calibrate LLM judges aligned to human labels.
• Run offline and online evaluation with CI and production traffic.
• Triage regressions to root cause and propose fixes.
• Define voice quality metrics and latency budgets.
• Evaluate ML models across vision/speech/predictive domains.
• Conduct adversarial testing and track regression coverage.
• Lead human-evaluation programs with clear guidelines.

Skills

Python

ML evaluation

LLM evaluation

Agent evaluation

LLM judges

ML fundamentals

Pandas

NumPy

SQL

Prompt engineering

Jira

Slack

Tools

LangSmith

LangGraph

MLflow

PromptFoo

Job description

Joblogic is building its AI Agent Platform and seeks an AI Evaluation Engineer to own model quality across channels and tools. You will define rubrics, design datasets, and implement release gates that guard every agent version in CI and live traffic.

You will collaborate with engineers, data, and product to ensure robust evaluations, actionable insights, and continuous improvement of agent performance in production. This is a hybrid role based in Lahore, Pakistan.