Remote ML Evaluation Engineer

Zute Predictive; Northern (KY); Hybrid; USD 110,000 - 160,000; Full time

Job summary

Zute Predictive is seeking an engineer to design evaluation harnesses that verify model performance after deployments and upgrades. You will build pipelines that run in CI and help prevent regressions before they reach production.

You will define measurable criteria for what "working" means across customer integrations, design annotation workflows to keep ground truth aligned with evolving data, and research robustness under distribution shifts and adversarial inputs.

Qualifications

• Experience building eval infrastructure that caught real production problems.
• Familiar with frontier models at the API level.
• Able to maintain Python in production-ready form.
• Understanding of statistical significance for eval results.

Responsibilities

• Build eval pipelines that run automatically on every model upgrade and flag regressions before production.
• Define what “working” means for each customer integration in measurable terms.
• Design annotation workflows to keep ground truth current as data drifts.
• Research prompt robustness: adversarial inputs, distribution shift, context window edge cases.
• Publish internal findings as reference documents for the team.

Skills

Python

Eval pipelines

Statistical significance

Adversarial inputs

Tools

CI/CD tooling

PyTest

Job description

Zute Predictive is seeking an engineer to design evaluation harnesses that verify model performance after deployments and upgrades. You will build pipelines that run in CI and help prevent regressions before they reach production.

You will define measurable criteria for what "working" means across customer integrations, design annotation workflows to keep ground truth aligned with evolving data, and research robustness under distribution shifts and adversarial inputs.