Remote Contract Software Engineer for AI Evaluation

Patronusaiinc; United States; Remote; USD 69,000 - 138,000; Full time

Job summary

Patronus AI is seeking experienced software engineers for task-based contracts to design realistic, multi-step coding tasks for frontier AI models and to judge how well those models handle them.

You will apply your day-to-day engineering judgment to show where AI gets code, reasoning and technical instructions right, and where it falls short. This remote India contract pays $50–$100 per hour, with flexible hours and 15–20 hours per week.

Qualifications

• 2+ years of professional software engineering experience.
• Experience with system design, data pipelines, ML or LLM-based applications.
• Prior work in code review, technical writing, AI training or model evaluation.
• Strong coding skills in Python, Java, C++, JavaScript, TypeScript or Go.
• Experience building, reviewing and debugging production-grade code.
• Ability to follow detailed technical guidelines, work independently, respond to feedback, and meet deadlines.
• Excellent written communication and attention to detail.
• A bachelor’s degree in computer science, engineering or related field, or equivalent experience.

Responsibilities

• Design realistic, multi-step coding and technical tasks that test an AI model's ability to reason, write code and follow precise instructions.
• Build tasks around codebases, technical specs, architecture docs and datasets.
• Evaluate AI-generated code and technical content for correctness, efficiency, clarity and alignment with real-world engineering practice.
• Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.
• Help improve how AI understands programming, system design, data workflows and engineering best practices.
• Work independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high.
• Test and refine content to reflect real-world complexity.
• Review other contributors’ work for accuracy, clarity, and practical relevance.
• Incorporate feedback promptly and flag gaps.

Skills

System design

Data pipelines

ML/LLM-based applications

Code review

Technical writing

Model evaluation

Programming in major language

Education

Bachelor degree in CS/Engineering or related field

Job description

Patronus AI is seeking experienced software engineers for task-based contracts to design realistic, multi-step coding tasks for frontier AI models and to judge how well those models handle them.

You will apply your day-to-day engineering judgment to show where AI gets code, reasoning and technical instructions right, and where it falls short. This remote India contract pays $50–$100 per hour, with flexible hours and 15–20 hours per week.