ML Evaluation Architect & Quality Harness Engineer

Brilliant Systems; Lahore; On-site; PKR 1,800,000 - 3,200,000; Full time

Job summary

Brilliant Systems is seeking an engineer to own the evaluation of ML/LLM systems, turning client requirements into measurable evaluation sets and robust tooling. You will build harnesses, monitor production drift, and ensure evaluation is integral to the product lifecycle.

You must demonstrate statistical literacy, strong Python skills, and a knack for developer-friendly tooling, while candidly reporting results that may be inconvenient.

Qualifications

• You have built evaluation for a machine learning or LLM system that others relied on.
• You understand what small samples can and cannot tell you.
• You write code and build tooling that engineers actually adopt.
• You are willing to report results that nobody wants to hear.

Responsibilities

• Turn client requirements into measurable evaluation sets and adversarial cases.
• Build evaluation harnesses and tooling so engineers can run and read them.
• Track drift in production and raise it early before clients detect it.
• Contribute to BotUp where evaluation is part of the product, not just internal tooling.
• Push back early on unmeasurable scope while changes are inexpensive.

Skills

Strong Python

Statistical literacy

Taste for tooling

Scepticism

Tools

Python

Job description

Brilliant Systems is seeking an engineer to own the evaluation of ML/LLM systems, turning client requirements into measurable evaluation sets and robust tooling. You will build harnesses, monitor production drift, and ensure evaluation is integral to the product lifecycle.

You must demonstrate statistical literacy, strong Python skills, and a knack for developer-friendly tooling, while candidly reporting results that may be inconvenient.