Remote AI Quality Engineer: Eval Harness Lead

Sokoni Kwetu Limited; United States; Remote; USD 95,000 - 140,000; Full time

Job summary

Sokoni Kwetu Limited is hiring for a remote role owning the eval harness and quality gate from day one. You will replace the old Evals Specialist model with a standing owner responsible for measurable agent quality.

You will design MVP-style evals, integrate them into CI, and establish thresholds with Product and the Tech Lead, paving the way for future adversarial testing while avoiding MVP overbuild.

Qualifications

• Experience evaluating ML, LLM, or non-deterministic systems.
• Strong test and benchmark design capability.
• Comfort working with noisy metrics, thresholds, and probabilistic behavior.
• Good scripting and automation skills.

Responsibilities

• Build and maintain the MVP eval harness: golden tasks, exception tasks, scorecard metrics, and regression packs.
• Wire evals into CI so quality regressions fail builds and releases.
• Define and maintain release-gate thresholds with Product and the Tech Lead.
• Lay the path for later adversarial and drift-testing expansion without overbuilding MVP scope.

Skills

ML evaluation

Benchmark design

Scripting

Automation

Probabilistic reasoning

Tools

CI/CD pipelines

Job description

Sokoni Kwetu Limited is hiring for a remote role owning the eval harness and quality gate from day one. You will replace the old Evals Specialist model with a standing owner responsible for measurable agent quality.

You will design MVP-style evals, integrate them into CI, and establish thresholds with Product and the Tech Lead, paving the way for future adversarial testing while avoiding MVP overbuild.