Production ML Engineer: Evaluation, Fine-Tuning & Task Routing

Zoro; Northern (KY); Hybrid; USD 120,000 - 180,000; Full time

Job summary

Zoro, LLC is seeking a candidate to own the eval harnesses, the fine-tuning runs, and the routing layer that decides which model sees which task. You will define the judgment boundary between model and human and monitor production behavior to reduce failure rates while maintaining quality.

The role emphasizes hands-on evaluation, model routing, and continuous improvement of production accuracy and cost-efficiency.

Qualifications

• Experience in evaluating language models and setting evaluation criteria.
• Proficient in Python and data analysis for model evaluation.
• Experience with fine-tuning methods (SFT, preference tuning) and deployment considerations.

Responsibilities

• Own the eval suites that gate model changes before production.
• Fine-tune and route models for drafting, extraction, and routing tasks.
• Define and enforce the judgment boundary between model and human.
• Observe production behavior and drive failure rates down week over week.
• Lower cost per correct action without lowering the standard.

Skills

LLM evaluation

Python

Model fine-tuning

Production systems

Experimentation

Tools

Python tooling

Job description

Zoro, LLC is seeking a candidate to own the eval harnesses, the fine-tuning runs, and the routing layer that decides which model sees which task. You will define the judgment boundary between model and human and monitor production behavior to reduce failure rates while maintaining quality.

The role emphasizes hands-on evaluation, model routing, and continuous improvement of production accuracy and cost-efficiency.