Remote LLM Code Evaluator & AI Training Specialist

Thoth AI; Portugal; Remote; EUR 34,000 - 69,000; Full time

Job summary

Thoth AI is seeking a Coding Agent Evaluation Specialist (LLM Eval & AI Training) for a fully remote freelance role. You will evaluate AI-generated code, assess coding agent performance, and provide structured feedback to improve models.

Flexible hours and remote work cater to developers seeking project-based opportunities. Ideal candidates have professional software development experience, 1+ year in relevant tasks (3+ years a plus), strong English (B2+), and proficiency with Git and

Qualifications

• Professional software development experience demonstrated
• 1+ year of relevant experience preferred (3+ years advantageous)
• Experience in LLM code evaluation, AI training, code annotation, or data labeling is a plus
• Ability to read, run, test, and debug code effectively
• Knowledge of Git and terminal/CLI tools
• Strong written English for feedback and documentation

Responsibilities

• Evaluate LLM-generated code and coding agent trajectories for correctness and quality
• Run, test, and debug code against test cases and criteria
• Reproduce issues and identify root causes
• Categorize errors (reasoning, code, tools, environment)
• Provide structured, actionable feedback per evaluation guidelines
• Maintain consistency in scoring, ranking, and rubrics
• Complete rubric interpretation and golden-task training
• Pass post-training qualification before live project work

Skills

Analytical thinking

Attention to detail

Code reading and debugging

English proficiency (B2+)

Task execution

Education

Bachelor’s/Master’s/PhD in CS/SE or related

Tools

Git

Command-line

Job description

Thoth AI is seeking a Coding Agent Evaluation Specialist (LLM Eval & AI Training) for a fully remote freelance role. You will evaluate AI-generated code, assess coding agent performance, and provide structured feedback to improve models.

Flexible hours and remote work cater to developers seeking project-based opportunities. Ideal candidates have professional software development experience, 1+ year in relevant tasks (3+ years a plus), strong English (B2+), and proficiency with Git and