AI Evaluation & Agent Systems Engineer

Syntony; Durham (NC); On-site; USD 83,000 - 138,000; Full time

Job summary

Syntony is hiring for an AI Evaluation & Agent Systems Engineer in Durham, NC. You will build evaluation harnesses, record structured traces, and create replay tests to compare system behavior across changes.

You will connect outputs to governance materials and document scenarios, instructions, and results with a focus on reproducibility and clear technical writing. This contract role emphasizes rigorous evaluation workflows.

Qualifications

• Experience developing evaluation software and agent systems.
• Understanding of experiment design and measurement limitations.
• Ability to debug multi-step behavior across system boundaries.
• Careful distinction between observed results and conclusions supported by evidence.
• Clear technical writing and experience producing reproducible work.

Responsibilities

• Develop evaluation harnesses for model behavior, tool use, and multi-step agent tasks.
• Record inputs, configurations, tool calls, observations, and limitations in structured traces.
• Build replay and regression tests for comparing system behavior before and after changes.
• Connect evaluation outputs to Resonance attack-path records and GovTune AI review materials.
• Document scenarios, run instructions, results, and the limits of each evaluation.

Skills

Evaluation software

Model integration

Agent systems

Experiment design

Nondeterminism

Measurement limitations

Multi-step debugging

Failure tracing

Technical writing

Reproducible work

Job description

Build evaluation software for AI models and agents. The work involves designing tests, recording evidence, and checking whether a mitigation changes the behavior under investigation. Evaluation outputs will support Syntony’s research and product workflows.

• Develop evaluation harnesses for model behavior, tool use, and multi-step agent tasks.
• Record inputs, configurations, tool calls, observations, and limitations in structured traces.
• Build replay and regression tests for comparing system behavior before and after changes.
• Connect evaluation outputs to Resonance attack-path records and GovTune AI review materials.
• Document scenarios, run instructions, results, and the limits of each evaluation.

Qualifications

• Experience developing evaluation software, model integrations, or agent systems.
• Understanding of experiment design, nondeterminism, and measurement limitations.
• Ability to debug multi-step behavior and trace failures across system boundaries.
• Careful distinction between observed results and conclusions that the evidence supports.
• Clear technical writing and experience producing reproducible work.

We assess relevant experience and demonstrated work. Graduate enrollment or a graduate degree is not required.

Compensation and scope

This is a paid contract role. Deliverables, milestones, rates, payment terms, expected availability, and acceptance criteria will be agreed in writing before work begins.

The agreement will also cover confidentiality, intellectual property, access requirements, and handover. This professional role is separate from Syntony’s fellowships and undergraduate internship.

Apply for this role

Tell us about your experience and share links to your work. You can also apply by emailing [e-mail] with “AI Evaluation & Agent Systems Engineer” in the subject line.

Fields marked with an asterisk are required.

I agree that Syntony may use this information to review my application and contact me about it, as described in the Privacy Policy . *