Senior Software Engineer - LLM Benchmarking Lead (Remote)

24-Mag Llc; New York (NY); Remote; USD 83,000 - 138,000; Part time

Benefits offered by this job

Fully remote

Job summary

24-MAG LLC is seeking a part-time Senior Software Engineer for a remote consulting role focused on advanced LLM evaluation, repository validation, and benchmark development.

You will analyze public repositories, configure reproducible environments using Docker, modify real-world code, and assess model performance on bug-fixing tasks. The contract runs about 20 hours per week for 3 months, with some overlap to PST hours.

Qualifications

• Senior engineer with expert-level software development experience.
• Proficient in Python, JavaScript, Go, or C-family languages.
• Strong debugging and issue-triage skills.
• Experience with open-source software is advantageous.
• Familiarity with large language model evaluation is a plus.

Responsibilities

• Analyse issues across public repositories for LLM evaluation.
• Set up reproducible development environments and Docker containers.
• Modify code and evaluate model performance on tasks.
• Conduct testing and verify repository quality.
• Lead or support junior engineers; collaborate remotely.

Skills

Senior Engineer

Tech Lead

LLM evaluation

Open-source analysis

Python

Git

Docker

CI/CD

Tools

Docker

Git

Job description

24-MAG LLC is seeking a part-time Senior Software Engineer for a remote consulting role focused on advanced LLM evaluation, repository validation, and benchmark development.

You will analyze public repositories, configure reproducible environments using Docker, modify real-world code, and assess model performance on bug-fixing tasks. The contract runs about 20 hours per week for 3 months, with some overlap to PST hours.