Software Engineer – AI Evaluation (Task-based Contract, Remote)

Patronus AI; United States; Remote; USD 69,000 - 138,000; Full time

Job summary

Patronus AI seeks experienced software engineers for task-based contracts to design realistic, challenging tasks for frontier AI models and judge how well those models handle them.

You will apply day-to-day engineering judgment to test coding, reasoning and technical instructions, grade outputs, provide structured feedback, and work asynchronously with the Patronus team to maintain high quality.

Responsibilities

• Design realistic, multi-step coding and technical tasks that test an AI model's ability to reason, write code and follow precise instructions.
• Build tasks around realistic workspace materials such as codebases, technical specs, architecture docs and datasets.
• Evaluate AI-generated code and technical content for correctness, efficiency, clarity and alignment with real-world engineering practice.
• Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.
• Help improve how AI understands programming, system design, data workflows and engineering best practices.
• Work independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high.

Job description

Pay: $50–$100/hr

Commitment: Flexible, typically 15–20 hours per week

Patronus AI is a frontier lab building simulation research and infrastructure to accelerate progress toward human-aligned AGI. We built some of the earliest research in AI evaluation. That includes FinanceBench, the first large-scale benchmark of how language models handle financial analysis; CopyrightCatcher, which tested leading models for copyright infringement; Humanity's Last Exam; SimpleSafetyTests; and Lynx, a model that detects hallucinations in AI output. Our team comes from Meta AI, Amazon AGI and Google. Our customers include foundation model labs and Fortune 500 companies, and we are backed by Lightspeed Venture Partners, Notable Capital and others.

At Patronus AI we are looking for experienced software engineers (task based contract) to design realistic, challenging tasks for frontier AI models and to judge how well those models handle them. You'll apply your day-to-day engineering judgment to show where AI gets code, reasoning and technical instructions right, and where it falls short.

What you'll do

• Design realistic, multi-step coding and technical tasks that test an AI model's ability to reason, write code and follow precise instructions.
• Build tasks around realistic workspace materials such as codebases, technical specs, architecture docs and datasets.
• Evaluate AI-generated code and technical content for correctness, efficiency, clarity and alignment with real-world engineering practice.
• Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.
• Help improve how AI understands programming, system design, data workflows and engineering best practices.
• Work independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high.