AI Evaluation Engineer: Real-World Coding & Systems Tasks

Patronus AI; United States; Remote; USD 69.000 - 138.000; Vollzeit

Zusammenfassung

Patronus AI seeks experienced software engineers for task-based contracts to design realistic, challenging tasks for frontier AI models and judge how well those models handle them.

You will apply day-to-day engineering judgment to test coding, reasoning and technical instructions, grade outputs, provide structured feedback, and work asynchronously with the Patronus team to maintain high quality.

Aufgaben

• Design realistic, multi-step coding and technical tasks that test an AI model's ability to reason, write code and follow precise instructions.
• Build tasks around realistic workspace materials such as codebases, technical specs, architecture docs and datasets.
• Evaluate AI-generated code and technical content for correctness, efficiency, clarity and alignment with real-world engineering practice.
• Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.
• Help improve how AI understands programming, system design, data workflows and engineering best practices.
• Work independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high.

Jobbeschreibung

Patronus AI seeks experienced software engineers for task-based contracts to design realistic, challenging tasks for frontier AI models and judge how well those models handle them.

You will apply day-to-day engineering judgment to test coding, reasoning and technical instructions, grade outputs, provide structured feedback, and work asynchronously with the Patronus team to maintain high quality.