Remote | Senior Software Engineer – LLM Evaluation (US/Canada/WEU based)

24-Mag Llc; United States; Remote; USD 14,000 - 55,000; Part time

Benefits offered by this job

Fully remote

Flexible hours

Contract-based

Job summary

24-MAG LLC is seeking experienced software engineers for a part-time, fully remote consulting role focused on evaluating advanced language models and coding benchmarks. You will curate code, verify solutions, and assess AI-generated software across languages, contributing to research collaboration and reproducible evaluation processes.

Ideal candidates have 3+ years in software engineering, strong full-stack skills, and familiarity with Python, JavaScript, Java, C++, Rust, or Go.

Qualifications

• 3+ years of professional software-engineering experience.
• Strong full-stack development capabilities.
• Experience building scalable, production-grade software.
• Deep knowledge of development, debugging, and code-quality assessment.

Responsibilities

• Curate high-quality code examples for model training and benchmarking.
• Develop precise solutions to software-engineering tasks across multiple languages.
• Evaluate AI-generated code for technical correctness and efficiency.
• Assess software design, architecture, and production implementation decisions.
• Collaborate with research and cross-functional teams on evaluation strategies.

Skills

Python

JavaScript

Java

C++

Rust

Go

ReactJS

Job description

We are sharing a specialised part-time consulting opportunity for experienced software engineers to contribute to advanced large language model evaluation, coding benchmark development, and AI-assisted software-engineering research.

Selected professionals will curate and evaluate code, develop verification mechanisms, assess AI-generated software across multiple programming languages, and help research teams understand how advanced models perform throughout realistic software-development workflows.

Key Responsibilities

Code Curation & Solution Development

• Curate high-quality code examples for model training and benchmarking
• Develop precise solutions to software-engineering tasks
• Correct and improve code across multiple programming languages
• Work with Python, JavaScript, ReactJS, C/C++, Java, Rust, and Go
• Maintain strong standards for correctness and maintainability

AI-Generated Code Evaluation

• Evaluate AI-generated code for technical correctness
• Assess solutions for efficiency, scalability, and reliability
• Identify implementation weaknesses and recurring error patterns
• Review code quality against professional engineering standards
• Provide structured rationales supporting evaluation decisions

Verification & Automated Assessment

• Build agents that assess code quality
• Design mechanisms for automatically verifying software solutions
• Identify recurring model-generated coding errors
• Develop reliable checks for engineering tasks
• Support reproducible evaluation across repeated assignments

Software Engineering Lifecycle Evaluation

• Evaluate model capabilities across the software-development lifecycle
• Assess reasoning around prototyping and architecture design
• Review API design and production implementation decisions
• Evaluate launch, experimentation, monitoring, and maintenance scenarios
• Identify areas where models struggle with real-world engineering workflows

Research & Benchmark Collaboration

• Collaborate with research and cross-functional technical teams
• Contribute to datasets used for training and benchmarking
• Help define engineering evaluation strategies
• Compare model performance against professional engineering expectations
• Support iterative improvements to coding-focused evaluation systems

Ideal Profile

• 3+ years of professional software-engineering experience
• Strong full-stack development capabilities
• Experience building scalable, production-grade software
• Strong understanding of software architecture and system design
• Deep knowledge of development, debugging, and code-quality assessment
• Experience reviewing and improving complex software implementations
• Proficiency in one or more of Python, JavaScript, Java, C++, Rust, or related languages
• ReactJS, C, or Go experience may also be relevant to project assignments
• Strong understanding of API design and production implementation
• Familiarity with software monitoring and operational maintenance
• Ability to reason across the complete software-engineering lifecycle
• Strong analytical and problem-solving capabilities
• Excellent written and verbal communication skills
• Ability to provide clear, structured evaluation rationales
• Comfortable collaborating remotely with research and technical teams

Engagement Details

• Part-time independent contractor engagement
• Fully remote
• Candidates must be based in the United States, Canada, or eligible Western European (WEU) countries
• Source examples of WEU locations include Austria, Belgium, France, and Germany
• Minimum commitment: 10 hours per week
• Flexible workload of up to 40 hours per week
• Initial project duration is approximately 1 month
• Extension may be available depending on performance and project fit
• No medical or paid-leave benefits are included under the contractor arrangement
• Application process takes approximately 15–30 minutes
• Completion of an AI video interview is required
• Compensation is not specified in the source materials
• Work must be completed without using confidential, proprietary, unreleased, employer-restricted, client-restricted, or otherwise protected code, datasets, architecture materials, or technical information belonging to any employer, client, institution, or other third party

By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy