Senior Software Engineer – LLM Evaluation & Repository Validation

Turing

United States

Remote

USD 55,000 - 83,000

Part time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Fully remote
Cutting-edge AI projects

Job summary

Aitrainergigs seeks a Senior Software Engineer to lead LLM evaluation and repository validation efforts. This role focuses on high-quality public GitHub projects, automation of environments, issue triage, and assessing test coverage and bug-fixing scenarios.

You will collaborate with researchers, potentially lead junior engineers, and help design tasks based on repositories with 500+ stars. This fully remote contractor role requires about 20 hours per week and starts next week.

Qualifications

  • Experience with one or more of the listed languages and strong navigation of large codebases.
  • Ability to run, test, and modify real-world projects locally.
  • Familiarity with public GitHub repos and issue triaging a plus.

Responsibilities

  • Analyze and triage GitHub issues across trending open-source libraries.
  • Set up and configure code repositories, including Dockerization and environment setup.
  • Evaluate unit test coverage and quality.
  • Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
  • Collaborate with researchers to identify repositories and issues challenging for LLMs.
  • Opportunity to lead a team of junior engineers on projects.

Skills

Python
JavaScript
Java
Go
Rust
C/C++
C#
Ruby

Tools

Git
Docker
CI/CD

Job description

Job Title: Senior Software Engineer - LLM Evaluation & Repository Validation
About the projects:

we are building LLM evaluation and training datasets to train LLM to work on realistic software engineering problems. One of our approaches, in this project, is to build verifiable SWE tasks based on public repository histories in a synthetic approach with human-in-the-loop; while expanding the dataset coverage to different types of tasks in terms of programming language, difficulty level, and etc.

About the Role:

We are looking for experienced software engineers (tech lead level) who are familiar with high-quality public GitHub repositories and can contribute to this project. You should have experience working with well-maintained, widely-used repos with 500+ stars. This role involves hands-on software engineering work, including development environment automation, issue triaging, and evaluating test coverage and quality

Why Join Us?

Turing is one of the world's fastest-growing AI companies accelerating the advancement and deployment of powerful AI systems. You'll be at the forefront of evaluating how LLMs interact with real code, influencing the future of AI-assisted software development. This is a unique opportunity to blend practical software engineering with AI research.

What does day-to-day look like
  • Analyze and triage GitHub issues across trending open-source libraries.
  • Set up and configure code repositories, including Dockerization and environment setup.
  • Evaluating unit test coverage and quality.
  • Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
  • Collaborate with researchers to design and identify repositories and issues that are challenging for LLMs.
  • Opportunities to lead a team of junior engineers to collaborate on projects.
Required Skills:
  • Strong experience with at least one of the following languages: Python, JavaScript, Java, Go, Rust, C/C++, C#, or Ruby.
  • Proficiency with Git, Docker, and basic software pipeline setup.
  • Ability to understand and navigate complex codebases.
  • Comfortable running, modifying, and testing real-world projects locally.
  • Experience contributing to or evaluating open-source projects is a plus.
Nice to Have:
  • Previous participation in LLM research or evaluation projects.
  • Experience building or testing developer tools or automation agents.
Perks of Freelancing With Turing:
  • Work in a fully remote environment.
  • Opportunity to work on cutting-edge AI projects with leading LLM companies.
Offer details:
  • Commitment required: 20 hours per week with some overlap with PST
  • Employment type: Contractor assignment (no medical/paid leave)
  • Duration of contract: 3 months with expected start date as next week
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer – Ruby (LLM Evaluation & Repository Validation)
Senior Software Engineer – Ruby (LLM Evaluation & Repository Validation)

Turing • United States

Remote
USD 83,000 - 124,000
Fully remote environment
Cutting-edge AI projects
Senior Software Engineer – Go (LLM Evaluation & Repository Validation)
Senior Software Engineer – Go (LLM Evaluation & Repository Validation)

Turing • United States

Remote
USD 83,000 - 138,000
Fully remote
Cutting-edge AI projects
Senior Software Engineer – C++(LLM Evaluation & Repository Validation)
Senior Software Engineer – C++(LLM Evaluation & Repository Validation)

Turing • United States

Remote
USD 83,000 - 138,000
Fully remote
Work on cutting-edge AI projects
Senior Software Engineer – Rust (LLM Evaluation & Repository Validation)
Senior Software Engineer – Rust (LLM Evaluation & Repository Validation)

Turing • United States

Remote
USD 41,000 - 83,000
Fully remote
Python
Python

Turing Global India • United States

Remote
USD 23,000 - 32,000
Work in a fully remote environment
Cutting-edge AI projects with leadingL
Remote Senior Software Engineer - 34123
Remote Senior Software Engineer - 34123

Turing • San Francisco (CA)

Remote
USD 150,000 - 175,000
Work in a fully remote environment
Cutting-edge AI projects
SWE Bench Engineer - C#
SWE Bench Engineer - C#

Turing • United States

Remote
USD 46,000 - 68,000
Fully remote
C# Engineer
C# Engineer

Turing Global India • United States

Remote
USD 14,000 - 27,000
Fully remote
Cutting-edge AI projects
Remote Senior Software Engineer - 34123
Remote Senior Software Engineer - 34123

Turing • San Francisco (CA)

Remote
USD 120,000 - 175,000
Fully remote work environment
Opportunity to work on cutting-edge AI projects
Java Developer
Java Developer

Turing Global India • United States

Remote
USD 23,000 - 32,000
Fully remote
Cutting-edge AI projects