AI Evaluation Architect for Models & Agent Systems

Syntony; Durham (NC); On-site; USD 83,000 - 138,000; Full time

Job summary

Syntony is hiring for an AI Evaluation & Agent Systems Engineer in Durham, NC. You will build evaluation harnesses, record structured traces, and create replay tests to compare system behavior across changes.

You will connect outputs to governance materials and document scenarios, instructions, and results with a focus on reproducibility and clear technical writing. This contract role emphasizes rigorous evaluation workflows.

Qualifications

• Experience developing evaluation software and agent systems.
• Understanding of experiment design and measurement limitations.
• Ability to debug multi-step behavior across system boundaries.
• Careful distinction between observed results and conclusions supported by evidence.
• Clear technical writing and experience producing reproducible work.

Responsibilities

• Develop evaluation harnesses for model behavior, tool use, and multi-step agent tasks.
• Record inputs, configurations, tool calls, observations, and limitations in structured traces.
• Build replay and regression tests for comparing system behavior before and after changes.
• Connect evaluation outputs to Resonance attack-path records and GovTune AI review materials.
• Document scenarios, run instructions, results, and the limits of each evaluation.

Skills

Evaluation software

Model integration

Agent systems

Experiment design

Nondeterminism

Measurement limitations

Multi-step debugging

Failure tracing

Technical writing

Reproducible work

Job description

Syntony is hiring for an AI Evaluation & Agent Systems Engineer in Durham, NC. You will build evaluation harnesses, record structured traces, and create replay tests to compare system behavior across changes.

You will connect outputs to governance materials and document scenarios, instructions, and results with a focus on reproducibility and clear technical writing. This contract role emphasizes rigorous evaluation workflows.