AI Evaluation Engineer

Brilliant Systems; Lahore; On-site; PKR 1,800,000 - 3,200,000; Full time

Job summary

Brilliant Systems is seeking an engineer to own the evaluation of ML/LLM systems, turning client requirements into measurable evaluation sets and robust tooling. You will build harnesses, monitor production drift, and ensure evaluation is integral to the product lifecycle.

You must demonstrate statistical literacy, strong Python skills, and a knack for developer-friendly tooling, while candidly reporting results that may be inconvenient.

Qualifications

• You have built evaluation for a machine learning or LLM system that others relied on.
• You understand what small samples can and cannot tell you.
• You write code and build tooling that engineers actually adopt.
• You are willing to report results that nobody wants to hear.

Responsibilities

• Turn client requirements into measurable evaluation sets and adversarial cases.
• Build evaluation harnesses and tooling so engineers can run and read them.
• Track drift in production and raise it early before clients detect it.
• Contribute to BotUp where evaluation is part of the product, not just internal tooling.
• Push back early on unmeasurable scope while changes are inexpensive.

Skills

Strong Python

Statistical literacy

Taste for tooling

Scepticism

Tools

Python

Job description

Decide what done means. If we cannot score it, we have not specified it, and we certainly cannot sell it.

We sell fixed-price phases, so we need a defensible answer to whether a thing is finished. For deterministic software that is a test suite. For a system that is probabilistic by design it is an evaluation harness, and building good ones is a speciality. This role owns that speciality across client engagements and our own products.

What you will do

• Turn client requirements into measurable evaluation sets, including the adversarial cases nobody asked for.
• Build the harnesses, and the tooling around them, so any engineer can run and read them.
• Track drift in production and raise it before a client does.
• Work on BotUp, where every worker is reviewed and every run is metered and logged, so evaluation is part of the product rather than internal tooling.
• Push back early on scope that cannot be measured, while it is still cheap to change.

What we need from you

• You have built evaluation for a machine learning or LLM system that other people then relied on.
• Statistical literacy: you know what a small sample can and cannot tell you.
• Strong Python, and a taste for tooling other engineers actually adopt.
• Scepticism. This role only works if you are willing to report a result nobody wants.

Useful but not required

• A testing or QA engineering background before moving into machine learning.
• Experience with human-in-the-loop annotation at any scale.
• You have written about evaluation somewhere public.