Remote | Software Engineer — $90–$140/hour

24-Mag Llc; New York, Northern (NY, KY); Hybrid; USD 124,000 - 193,000; Part time

Job summary

24-MAG LLC invites experienced Software Engineers to join a part-time, fully remote evaluation project. You will analyze AI-generated technical outputs, assess accuracy and clarity, and craft detailed written rationales to support each evaluation.

No software development required. Ideal candidates bring strong software engineering experience, exceptional written English, and a proven ability to review technical content with precision.

Qualifications

• Professional Software Engineer with a modern technology stack.
• Excellent written communication in English.
• Experience reviewing technical documentation and API references.
• Ability to provide structured feedback on technical writing.

Responsibilities

• Evaluate and score AI-generated technical responses for correctness and usefulness.
• Review technical content (docs, API references, specs) for clarity and completeness.
• Write detailed rationales distinguishing factual accuracy, communication quality, and relevance.
• Work asynchronously within remote evaluation workflows.

Skills

Software engineering

Technical writing

Code review

Analytical thinking

Remote collaboration

Attention to detail

Job description

We are sharing a specialised part-time consulting opportunity for experienced Software Engineers to contribute technical expertise to an advanced AI development project focused on evaluating technical content, software-related reasoning, and AI-generated system outputs.

This engagement focuses on critical analysis and evaluation rather than software development or feature delivery. Selected professionals will assess AI-generated responses to technical prompts, determine whether outputs are accurate, clearly communicated, appropriate for their intended audience, and genuinely useful, and provide detailed written reasoning to support each evaluation. No prior experience in AI is required.

Key Responsibilities

Technical Content & AI Output Evaluation

• Evaluate and compare AI-generated responses to technical prompts for correctness, clarity, tone, and usefulness
• Review outputs involving technical documentation, error messages, code comments, architectural explanations, instructions, and related software-engineering content
• Determine whether generated responses accurately address the underlying technical question or requirement
• Identify outputs that appear fluent or convincing but contain factual, logical, or technical inaccuracies
• Assess whether technically correct responses are communicated clearly and appropriately for their intended audience

Technical Documentation & Communication Review

• Review technical writing for precision, completeness, structure, and instructional value
• Evaluate documentation, design documents, API references, specifications, and comparable technical materials
• Identify unclear explanations, ambiguous wording, misleading statements, inconsistencies, and missing information
• Assess whether technical content is appropriately tailored to developers, technical stakeholders, or other intended users
• Apply professional software-engineering judgement when reviewing the quality of technical communication

Scoring, Analysis & Written Rationale

• Assign analytical quality scores to AI-generated technical responses
• Write detailed rationales explaining the reasoning behind each evaluation
• Clearly distinguish between factual correctness, communication quality, relevance, and overall usefulness
• Document subtle errors, inconsistencies, or ambiguities with precision and consistency
• Provide actionable written feedback that can support improvements in AI model performance

Remote Collaboration & Quality Consistency

• Work independently within structured remote evaluation workflows
• Collaborate asynchronously with project managers, reviewers, and other expert contributors
• Follow established evaluation guidelines, rubrics, and project-specific quality standards
• Apply consistent judgement across repeated technical assessment tasks
• Complete assigned work within agreed project requirements and defined timeframes

Ideal Profile

• Professional experience as a Software Engineer working with a modern technology stack
• Native-level English fluency and exceptional written communication skills
• Proven experience authoring or reviewing technical documentation, design documents, API references, specifications, or similar materials
• Experience conducting code reviews, design reviews, or providing structured feedback on technical writing
• Strong ability to assess technical communication for correctness, audience appropriateness, and instructional value
• Excellent analytical skills and meticulous attention to detail
• Ability to identify subtle factual inaccuracies, logical inconsistencies, ambiguous explanations, and misleading technical language
• Strong independent-working skills and comfort operating within remote, asynchronous project environments
• Familiarity with AI model evaluation, annotation, RLHF, or related structured evaluation workflows is advantageous but not required

Engagement Details

• Part-time independent contractor engagement
• Fully remote
• Evaluation and analysis role focused on software-engineering and technical content
• No software development or feature-delivery responsibilities
• Compensation: $90–$140/hour
• No prior experience in AI is required
• Work will involve structured assessment, scoring, written rationale development, and technical quality review
• Project scope, workload, timing, and duration may evolve depending on project requirements
• Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, or other third party

About the Platform

This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.

By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy