Coding Agent Evaluation Specialist - Portugal

Thoth AI; Portugal; Remote; EUR 34,000 - 69,000; Full time

Job summary

Thoth AI is seeking a Coding Agent Evaluation Specialist (LLM Eval & AI Training) for a fully remote freelance role. You will evaluate AI-generated code, assess coding agent performance, and provide structured feedback to improve models.

Flexible hours and remote work cater to developers seeking project-based opportunities. Ideal candidates have professional software development experience, 1+ year in relevant tasks (3+ years a plus), strong English (B2+), and proficiency with Git and

Qualifications

• Professional software development experience demonstrated
• 1+ year of relevant experience preferred (3+ years advantageous)
• Experience in LLM code evaluation, AI training, code annotation, or data labeling is a plus
• Ability to read, run, test, and debug code effectively
• Knowledge of Git and terminal/CLI tools
• Strong written English for feedback and documentation

Responsibilities

• Evaluate LLM-generated code and coding agent trajectories for correctness and quality
• Run, test, and debug code against test cases and criteria
• Reproduce issues and identify root causes
• Categorize errors (reasoning, code, tools, environment)
• Provide structured, actionable feedback per evaluation guidelines
• Maintain consistency in scoring, ranking, and rubrics
• Complete rubric interpretation and golden-task training
• Pass post-training qualification before live project work

Skills

Analytical thinking

Attention to detail

Code reading and debugging

English proficiency (B2+)

Task execution

Education

Bachelor’s/Master’s/PhD in CS/SE or related

Tools

Git

Command-line

Job description

Position : Coding Agent Evaluation Specialist (LLM Eval & AI Training)

Employment Type : Remote Freelance

Working Hour : Flexible

About the Opportunity

We are building a talent pool of qualified professionals for upcoming and future Coding Agent Evaluation and AI Training projects. This freelance, fully remote opportunity is ideal for software engineers, programmers, and coding professionals who are passionate about improving the quality, safety, and performance of Large Language Models (LLMs) and AI coding agents.

In this role, you will evaluate AI-generated code, assess coding agent performance, identify errors, and provide structured feedback that helps train and improve next-generation AI systems.

Job Responsibilities

• Apply and adapt coding evaluation rubrics to assess LLM-generated code responses and coding agent trajectories, including correctness, code quality, efficiency, security, and tool usage.
• Run, test, and debug code to verify outputs against defined test cases and acceptance criteria.
• Reproduce technical issues and identify their root causes.
• Accurately categorize errors related to reasoning, code implementation, tool usage, or the execution environment.
• Provide clear, structured, and actionable written feedback according to established evaluation guidelines.
• Maintain consistent quality and accuracy across scoring, ranking, and rubric-based evaluation tasks.
• Complete rubric interpretation and golden-task training.
• Pass the required post-training qualification before starting live project work.

Requirements

• Relevant software development experience demonstrated through professional employment, internships, academic work, or personal projects.
• At least 1 year of relevant experience preferred, with 3 or more years considered an advantage.
• Previous experience in LLM code evaluation, AI training, code annotation, or data labeling, including code-response scoring, ranking, and rubric-based reviews.
• Ability to read, run, test, and debug code effectively.
• Working knowledge of Git, version control, and terminal or command-line tools.
• Ability to understand complex guidelines, apply evaluation rubrics consistently, identify errors accurately, and provide reliable feedback.
• Strong analytical skills, task-execution ability, attention to detail, and commitment to quality.
• English proficiency at B2 level or above, with C1 level preferred.
• Strong written English skills for feedback, training, documentation, and collaboration.
• A personal laptop or desktop computer with a stable internet connection.
• A bachelor’s, master’s, or doctoral degree in Computer Science, Software Engineering, or a related field is preferred. Equivalent professional experience is also accepted.
• Experience with coding agents, AI-assisted development tools, open-source contributions, or personal software projects is an advantage.