Remote | Senior Software Engineer – LLM Evaluation

24-Mag Llc; New York (NY); Remote; USD 14,000 - 55,000; Part time

Job summary

24-MAG LLC invites experienced Senior Software Engineers to join a part-time, remote consulting engagement focused on evaluating AI-generated software, code benchmarking, and research-driven verification. You will curate code, craft reference solutions, and build automated assessment mechanisms across multiple languages and platforms.

The role requires deep software-engineering expertise, strong communication, and the ability to work with research teams across time zones.

Qualifications

• Several years of professional software-engineering experience.
• At least 2 years of continuous full-time experience at a top-tier product company.
• Strong full-stack development capabilities.
• Experience building scalable, production-grade software.
• Deep understanding of software architecture and system design.
• Strong knowledge of debugging and code-quality assessment.
• Experience reviewing and improving complex production code.
• Proficiency in one or more of Python, JavaScript, C/C++, Java, Rust, or Go.
• ReactJS experience is valuable for frontend-oriented assignments.
• Strong understanding of API design and production implementation.
• Familiarity with software monitoring and operational maintenance.
• Ability to reason across the complete software-engineering lifecycle.
• Strong analytical and problem-solving capabilities.
• Excellent written and verbal communication skills.
• Ability to produce clear and structured evaluation rationales.
• Comfortable collaborating with research and technical teams in a remote environment.

Responsibilities

• Code Curation & Solution Development: Curate high-quality code examples for model training and benchmarking.
• Develop precise reference solutions to software-engineering tasks.
• Correct and improve code across multiple programming languages.
• Work with Python, JavaScript, ReactJS, C/C++, Java, Rust, and Go.
• Maintain strong standards for correctness, clarity, and maintainability.

Skills

Full-stack development

Production-grade software

Software architecture

Debugging & code-quality

API design

Remote collaboration

Python

JavaScript

C/C++

Java

Rust

Go

Tools

ReactJS

Job description

We are sharing a specialised part-time consulting opportunity for experienced Senior Software Engineers to contribute to advanced large language model evaluation, coding benchmark development, and AI-assisted software-engineering research.

Selected professionals will curate and evaluate code, develop verification mechanisms, assess AI-generated software across multiple programming languages, and help researchers understand where advanced models succeed or fail throughout the software-development lifecycle. The work combines senior-level software engineering judgement with structured evaluation and benchmark design.

Key Responsibilities

Code Curation & Solution Development

• Curate high-quality code examples for model training and benchmarking
• Develop precise reference solutions to software-engineering tasks
• Correct and improve code across multiple programming languages
• Work with Python, JavaScript, ReactJS, C/C++, Java, Rust, and Go
• Maintain strong standards for correctness, clarity, and maintainability

AI-Generated Code Evaluation

• Evaluate AI-generated code for technical correctness
• Assess solutions for efficiency, scalability, and reliability
• Identify implementation weaknesses and recurring error patterns
• Review code quality against professional engineering standards
• Provide structured rationales supporting evaluation decisions

Verification & Automated Assessment

• Design mechanisms that automatically verify software-engineering solutions
• Build agents capable of assessing code quality
• Develop approaches for identifying common model failure patterns
• Create deterministic or structured checks where appropriate
• Support reliable evaluation across repeated coding tasks

Software Engineering Lifecycle Evaluation

• Evaluate model capabilities across the software-development lifecycle
• Assess reasoning around prototyping and architecture design
• Review API design and production implementation decisions
• Evaluate launch, experimentation, monitoring, and operational scenarios
• Identify where models struggle with real-world engineering workflows

Research & Benchmark Collaboration

• Collaborate with researchers and cross-functional technical teams
• Contribute to datasets used for training and benchmarking
• Help define engineering evaluation strategies and quality standards
• Compare model performance against professional engineering expectations
• Support iterative improvements to coding-focused evaluation systems

Ideal Profile

• Several years of professional software-engineering experience
• At least 2 years of continuous full-time experience at a top-tier product company
• Strong full-stack development capabilities
• Experience building scalable, production-grade software
• Deep understanding of software architecture and system design
• Strong knowledge of debugging and code-quality assessment
• Experience reviewing and improving complex production code
• Proficiency in one or more of Python, JavaScript, C/C++, Java, Rust, or Go
• ReactJS experience is valuable for frontend-oriented assignments
• Strong understanding of API design and production implementation
• Familiarity with software monitoring and operational maintenance
• Ability to reason across the complete software-engineering lifecycle
• Strong analytical and problem-solving capabilities
• Excellent written and verbal communication skills
• Ability to produce clear and structured evaluation rationales
• Comfortable collaborating with research and technical teams in a remote environment

Engagement Details

• Part-time independent contractor engagement
• Fully remote
• Minimum commitment: 10 hours per week
• Flexible workload of up to 40 hours per week
• Partial overlap with PST working hours is required
• Initial project duration is approximately 1 month
• Project is expected to begin the following week
• Extension may be available depending on performance and project fit
• No medical or paid-leave benefits are included under the contractor arrangement
• Work will involve coding evaluation, benchmark development, verification systems, and software-engineering research
• Assignments may span Python, JavaScript, ReactJS, C/C++, Java, Rust, Go, and related technologies
• Work must be completed without using confidential, proprietary, unreleased, employer-restricted, client-restricted, or otherwise protected code, datasets, architecture materials, or technical information belonging to any employer, client, institution, or other third party

About the Platform

This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.