Remote LLM Evaluation & Repo Validation Engineer

AI Trainer Jobs; United States; Remote; USD 69,000 - 138,000; Full time

Job summary

AI Trainer Jobs is seeking software engineers to validate real-world repositories used to evaluate large language models. You will triage issues, set up reproducible environments, assess test quality, and check how well models fix real bugs.

You will review and triage issues in open-source GitHub projects, prepare repositories for evaluation with containerisation, and write up findings for researchers. This role is remote, independent, and deadline-driven.

Qualifications

• 3+ years of software engineering experience.
• Strong experience in at least one of Python, C++, C#, Go, Rust or Ruby.
• Proficiency with Git, Docker and basic pipeline setup.
• Ability to navigate complex code bases and run real-world projects locally.
• Fluent written English.
• Reliable, detail-oriented, and able to work independently and meet deadlines in a remote setting.

Responsibilities

• Review and triage issues in open-source GitHub projects.
• Prepare repositories for evaluation, including containerisation and environment automation.
• Assess how thorough and well written the unit tests are.
• Edit and run code locally to see how LLMs perform on bug-fixing tasks.
• Help pick repositories and issues that are hard for LLMs.
• Write up findings clearly for researchers and reviewers.
• Follow project guidelines, take part in calibration and review sessions, and incorporate feedback to keep quality consistent.

Skills

Software engineering

English fluency

Tools

Git

Docker

CI/CD pipelines

Job description

AI Trainer Jobs is seeking software engineers to validate real-world repositories used to evaluate large language models. You will triage issues, set up reproducible environments, assess test quality, and check how well models fix real bugs.

You will review and triage issues in open-source GitHub projects, prepare repositories for evaluation with containerisation, and write up findings for researchers. This role is remote, independent, and deadline-driven.