Coding Agent Evaluation Specialist

Thoth AI; United States; On-site; USD 90,000 - 150,000; Full time

Job summary

Thoth AI is seeking a highly skilled researcher to join its team in the United States to evaluate LLM code responses and agent trajectories using structured rubrics. The role requires strong programming proficiency, the ability to run and debug code, and clear, high-quality feedback for training and improvement.

Candidates should have a PhD or equivalent senior CS expertise, with evidence of professional or project experience in AI evaluation, data labeling, or related fields.

Qualifications

• PhD or professional CS degree required.
• Strong Python or main language proficiency and debugging ability.
• Excellent written and spoken English for feedback and collaboration.

Responsibilities

• Apply coding rubrics to evaluate LLM code responses and agent trajectories.
• Run, test, reproduce issues against test cases and acceptance criteria.
• Identify errors and provide structured, rubric-based feedback.
• Interpret rubrics and complete post-training qualification efforts.

Skills

Python

Git

Terminal

Debugging

Rubrics

Feedback

Education

PhD in CS or related field

Professional degree or equivalent

Tools

Git

Terminal

Python

Job description

Apply and adapt coding rubrics to evaluate LLM code responses and agent trajectories (correctness, code quality, efficiency, security, tool usage)

Run and test code, reproduce issues, and verify outcomes against test cases and acceptance criteria

Identify and attribute errors (reasoning, implementation, tool use, environment) and write structured feedback

Complete rubric interpretation and golden-task training; pass the post-training qualification before starting work

Relevant software development experience with evidence: professional role or project experience (internship or project acceptable; suggested 1+ year, 3+ years preferred)

LLM code evaluation / AI training or code annotation / data labeling experience (code response scoring, ranking, rubric-based review) with proven quality delivery

Proficient in Python or another main language; able to read, run and debug code; solid Git and terminal skills; able to apply rubrics under established standards, attribute errors and give reliable feedback

Task execution capability; high ability of guideline understanding and attention to detail

Device: laptop/PC with stable internet connection

Education: PhD, professional degree, or equivalent recognized senior expertise in CS / related field

Nice-to-have: Education: relevant Bachelor's / Master's / PhD (CS, software engineering or related field), or an equivalent professional / experience route

Nice-to-have: experience with coding agents / AI-assisted development tools; personal or open-source projects

Fluent English (B2+ required, C1+ preferred); written English for feedback, training and collaboration