ZP-ML-004 Applied ML -- eval & alignment Remote · US RESEARCH →

Zute Predictive; Northern (KY); Hybrid; USD 110,000 - 160,000; Full time

Job summary

Zute Predictive is seeking an engineer to design evaluation harnesses that verify model performance after deployments and upgrades. You will build pipelines that run in CI and help prevent regressions before they reach production.

You will define measurable criteria for what "working" means across customer integrations, design annotation workflows to keep ground truth aligned with evolving data, and research robustness under distribution shifts and adversarial inputs.

Qualifications

• Experience building eval infrastructure that caught real production problems.
• Familiar with frontier models at the API level.
• Able to maintain Python in production-ready form.
• Understanding of statistical significance for eval results.

Responsibilities

• Build eval pipelines that run automatically on every model upgrade and flag regressions before production.
• Define what “working” means for each customer integration in measurable terms.
• Design annotation workflows to keep ground truth current as data drifts.
• Research prompt robustness: adversarial inputs, distribution shift, context window edge cases.
• Publish internal findings as reference documents for the team.

Skills

Python

Eval pipelines

Statistical significance

Adversarial inputs

Tools

CI/CD tooling

PyTest

Job description

Design the evaluation harnesses that tell us whether a deployed model is actually working. Build them to run in CI and to survive model upgrades.

WHAT YOU'LL DO

• Build eval pipelines that run automatically on every model upgrade and flag regressions before they reach production.
• Define what \"working\" means for each customer integration -- in measurable, reproducible terms.
• Design the annotation workflows that keep ground truth current as customer data drifts.
• Research prompt robustness: adversarial inputs, distribution shift, context window edge cases.
• Publish internal findings as reference documents the whole team can act on.

WHAT WE'RE LOOKING FOR

• Track record building eval infrastructure that actually caught a real production problem.
• Familiar with the major frontier models at the API level -- not just the papers.
• Can write Python that a systems engineer would be comfortable maintaining.
• Understand statistical significance well enough to know when an eval result is meaningful.
• Have written a prompt that was still working six months later.