LLM Code Evaluator & Feedback Architect

Thoth AI; United States; On-site; USD 90,000 - 150,000; Full time

Job summary

Thoth AI is seeking a highly skilled researcher to join its team in the United States to evaluate LLM code responses and agent trajectories using structured rubrics. The role requires strong programming proficiency, the ability to run and debug code, and clear, high-quality feedback for training and improvement.

Candidates should have a PhD or equivalent senior CS expertise, with evidence of professional or project experience in AI evaluation, data labeling, or related fields.

Qualifications

• PhD or professional CS degree required.
• Strong Python or main language proficiency and debugging ability.
• Excellent written and spoken English for feedback and collaboration.

Responsibilities

• Apply coding rubrics to evaluate LLM code responses and agent trajectories.
• Run, test, reproduce issues against test cases and acceptance criteria.
• Identify errors and provide structured, rubric-based feedback.
• Interpret rubrics and complete post-training qualification efforts.

Skills

Python

Git

Terminal

Debugging

Rubrics

Feedback

Education

PhD in CS or related field

Professional degree or equivalent

Tools

Git

Terminal

Python

Job description

Thoth AI is seeking a highly skilled researcher to join its team in the United States to evaluate LLM code responses and agent trajectories using structured rubrics. The role requires strong programming proficiency, the ability to run and debug code, and clear, high-quality feedback for training and improvement.

Candidates should have a PhD or equivalent senior CS expertise, with evidence of professional or project experience in AI evaluation, data labeling, or related fields.