AI Evaluation Engineer

Brilliant Systems; United States; Hybrid; USD 5,400 - 9,700; Full time

Benefits offered by this job

Permanent, full-time

Remote-friendly

Job summary

Brilliant Systems in Pakistan is seeking an experienced Evaluation Engineer to shape how probabilistic AI systems are assessed. You will translate client requirements into measurable tests, build evaluation harnesses, and monitor drift in production.

This is a mid-to-senior, permanent, full-time role based in Lahore or remote. Compensation discussed in the first conversation; you will work across client projects and an AI product to make quality measurable throughout development and after

Qualifications

• Experience building evaluation for a machine learning or large language model system.
• Ability to turn qualitative requirements into concrete evaluation criteria and test cases.
• Skills in developing reusable harnesses and tools for engineering teams.
• Understanding of production monitoring and how system behaviour can change over time.

Responsibilities

• Translate client requirements into measurable evaluation sets, including challenging and adversarial cases.
• Build evaluation harnesses and supporting tools that engineers can run and interpret.
• Monitor production behavior for drift and surface problems before clients report them.
• Contribute evaluation capabilities to a product where worker activity is reviewed and runs are metered and logged.
• Identify requirements or project scope that cannot be measured, and raise the issue early enough to revise them.

Skills

Evaluation design

ML systems experience

Test case development

Production monitoring understanding

Tools

Evaluation harnesses

Monitoring dashboards

Job description

Role overview

Define how teams determine whether probabilistic AI systems meet their requirements. You will build evaluation methods and tooling across client projects and an AI product, making quality measurable throughout development and after release.

Responsibilities

• Translate client requirements into measurable evaluation sets, including challenging and adversarial cases.
• Build evaluation harnesses and supporting tools that engineers can run and interpret.
• Monitor production behavior for drift and surface problems before clients report them.
• Contribute evaluation capabilities to a product where worker activity is reviewed and runs are metered and logged.
• Identify requirements or project scope that cannot be measured, and raise the issue early enough to revise them.

Requirements

• Experience building evaluation for a machine learning or large language model system.
• Ability to turn qualitative requirements into concrete evaluation criteria and test cases.
• Skills in developing reusable harnesses and tools for engineering teams.
• Understanding of production monitoring and how system behavior can change over time.

Benefits and work setup

• Permanent, full-time role based in Lahore or remote.
• Mid-to-senior level; compensation is discussed in the first conversation.