AI Quality & Evaluation Engineer

Systems Limited; Lahore; On-site; PKR 2,400,000 - 3,600,000; Full time

Job summary

Systems Limited is seeking a QA/test engineer to lead automation for AI-native features. You will design evaluation harnesses and run regression tests to catch quality drift across model changes, with emphasis on accuracy, hallucination, and bias.

You will collaborate with AI architects, own the eval framework, and communicate quality risk to leadership, driving gated releases and robust test coverage in a fast-paced delivery environment.

Qualifications

• 5–9 yrs QA/test engineering with AI/ML feature testing.
• Strong Python-based test automation and CI/CD integration.
• Familiarity with AI-specific failure modes and evaluation metrics.
• Ability to interpret model evaluation metrics beyond pass/fail.

Responsibilities

• Build test plans and automation for AI-native features (functional + AI-specific).
• Design evaluation harnesses for model outputs—accuracy, drift, hallucination rate.
• Run regression tests across model/prompt/config changes to detect quality drift.
• Red-team AI features for edge cases and adversarial inputs.
• Create CI/CD integrated eval pipelines and dashboards for teams.
• Define eval thresholds and gate releases on them, with clear risk communication.
• Own the evals framework including datasets, rubrics, and LLm calibration.

Skills

QA automation

Python

CI/CD

AI/ML testing

Statistical literacy

Red-teaming

Communication

Drift testing

Tools

OpenAI Evals

Ragas

DeepEval

LangSmith

Azure AI Foundry

Job description

Systems Limited is seeking a QA/test engineer to lead automation for AI-native features. You will design evaluation harnesses and run regression tests to catch quality drift across model changes, with emphasis on accuracy, hallucination, and bias.

You will collaborate with AI architects, own the eval framework, and communicate quality risk to leadership, driving gated releases and robust test coverage in a fast-paced delivery environment.