Expert: Software Engineer – AI Evaluation (Task-based Contract, Remote)

Patronus AI, Inc.

United States

A distancia

USD 69.000 - 138.000

A tiempo parcial

Hace 12 días

Consigue una respuesta de este empleador

Tu currículum y tu carta de presentación, adaptados exactamente a lo que buscan contratar.

Supera los filtros ATS

Descripción de la vacante

Patronus AI is seeking experienced software engineers for task-based contracts to design realistic, multi-step coding and technical tasks for frontier AI models and to judge how well those models handle them. You’ll apply day-to-day engineering judgment to assess code, reasoning and technical instructions, grade outputs against rubrics, and provide clear, structured feedback.

This is a remote, hourly project-based role based in India, with typical commitment of 15–20 hours per week and the

Formación

  • 2+ years of professional experience in software engineering. (internships excluded)
  • Experience with system design, data pipelines, ML or LLM-based applications.
  • Prior work in code review, technical writing, AI training or model evaluation.
  • Strong coding skills in at least one major language, such as Python, Java, C++, JavaScript, TypeScript or Go.
  • Experience building, reviewing and debugging production-grade code.
  • Ability to follow detailed technical guidelines, work independently, respond to feedback, and meet deadlines.
  • Excellent written communication and close attention to detail.
  • A bachelor’s degree in computer science, engineering or a related field, or equivalent experience.

Responsabilidades

  • Design realistic, multi-step coding and technical tasks that test an AI model’s ability to reason, write code and follow precise instructions.
  • Build tasks around realistic workspace materials such as codebases, technical specs, architecture docs and datasets.
  • Evaluate AI-generated code and technical content for correctness, efficiency, clarity and alignment with real-world engineering practice.
  • Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.
  • Help improve how AI understands programming, system design, data workflows and engineering best practices.
  • Work independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high.
  • Test and refine content to reflect real-world complexity.
  • Review other contributors’ work for accuracy, clarity, and practical relevance.
  • Incorporate feedback promptly and flag gaps.

Conocimientos

System design
Data pipelines
ML/LLM applications
Code review
Technical writing
AI training/evaluation
Python/Java/C++/JS/TS/Go
Production-grade code

Educación

Bachelor's degree in CS/Engineering or related field

Descripción del empleo

About Patronus AI

Patronus AI is a frontier lab developing simulation research and infrastructure to accelerate progress toward human-aligned AGI. We are on a mission to simulate all of the world’s intelligence.

We are the team behind some of the earliest and most influential research in AI evaluation like FinanceBench, Lynx, SimpleSafetyTests, CopyrightCatcher, Humanity’s Last Exam, and more. We are formerly AI researchers and engineers from companies like Meta AI, Amazon AGI, and Google. Our customers include foundation model labs and Fortune 500 enterprises like Adobe. We are backed by top-tier investors like Lightspeed Venture Partners, Notable Capital, Stanford University, Noam Brown, Gokul Rajaram, and more.

About the role

We are looking for experienced software engineers (task based contract) to design realistic, challenging tasks for frontier AI models and to judge how well those models handle them. You’ll apply your day-to-day engineering judgment to show where AI gets code, reasoning and technical instructions right, and where it falls short.

What you’ll do

Depending on the project, you may:

  • Design realistic, multi-step coding and technical tasks that test an AI model’s ability to reason, write code and follow precise instructions.
  • Build tasks around realistic workspace materials such as codebases, technical specs, architecture docs and datasets.
  • Evaluate AI-generated code and technical content for correctness, efficiency, clarity and alignment with real-world engineering practice.
  • Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.
  • Help improve how AI understands programming, system design, data workflows and engineering best practices.
  • Work independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high.
  • Test and refine content to reflect real-world complexity.
  • Review other contributors’ work for accuracy, clarity, and practical relevance.
  • Incorporate feedback promptly and flag gaps.

What we’re looking for

  • 2+ years of professional experience in software engineering. (internships excluded)
  • Experience with system design, data pipelines, ML or LLM-based applications.
  • Prior work in code review, technical writing, AI training or model evaluation.
  • Strong coding skills in at least one major language, such as Python, Java, C++, JavaScript, TypeScript or Go.
  • Experience building, reviewing and debugging production-grade code.
  • Ability to follow detailed technical guidelines, work independently, respond to feedback, and meet deadlines.
  • Excellent written communication and close attention to detail.
  • A bachelor’s degree in computer science, engineering or a related field, or equivalent experience.
Our Standards
  • All tasks, project files and solutions are human-authored. LLM-generated code, content or text is not permitted.
  • Tasks and solutions must be original. Nothing may be copied from published or easily searchable sources, including tutorials and marketplace content.
  • Never include code, assets or project files from a current or former employer, client or studio, or anything covered by an NDA. All studios, projects and scenarios are fictional, and all code and content is your own.
  • Quality is judged on accuracy, rubric validity, difficulty and reviewer agreement, not volume.

Pay: $50–$100 per hour and some tasks would have a flat rate.

Location: Remote (India)

Type: Hourly, project/task-based contract (not a full-time role)

Commitment: Flexible, typically 15–20 hours per week; can be done alongside a full-time job

Applications are reviewed on a rolling basis. Placement depends on project needs and assessment results.

Patronus AI is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.

Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Expert: Software Engineer – AI Evaluation (Task-based Contract, Remote)
Expert: Software Engineer – AI Evaluation (Task-based Contract, Remote)

Patronusaiinc • EE. UU.

A distancia
USD 69.000 - 138.000
Software Engineer – AI Evaluation (Task-based Contract, Remote)
Software Engineer – AI Evaluation (Task-based Contract, Remote)

Patronus AI • EE. UU.

A distancia
USD 69.000 - 138.000
Domain Expert, AI Evaluation Task Writer: Finance (Contract, Remote)
Domain Expert, AI Evaluation Task Writer: Finance (Contract, Remote)

Patronus AI, Inc. • EE. UU.

A distancia
USD 69.000 - 207.000
Remote work
Trial task paid
Performance-based pay
Expert: Mechanical Engineering (Contract, Remote)
Expert: Mechanical Engineering (Contract, Remote)

Patronus AI, Inc. • EE. UU.

A distancia
USD 69.000 - 207.000
Expert: Mechanical Engineering (Contract, Remote)
Expert: Mechanical Engineering (Contract, Remote)

Patronus AI • EE. UU.

A distancia
USD 69.000 - 207.000
Remote work
Rolling start
Paid trial task
Sales (Contract, Remote)
Sales (Contract, Remote)

Patronus AI • EE. UU.

A distancia
USD 69.000 - 207.000
Domain Expert, AI Evaluation Task Writer: Sales (Contract, Remote)
Domain Expert, AI Evaluation Task Writer: Sales (Contract, Remote)

Patronus AI, Inc. • EE. UU.

A distancia
USD 69.000 - 207.000
Domain Expert, AI Evaluation Task Writer: Unreal Game Development (Contract, Remote)
Domain Expert, AI Evaluation Task Writer: Unreal Game Development (Contract, Remote)

Patronus AI, Inc. • EE. UU.

A distancia
USD 69.000 - 207.000
Interactive Coding, Site Reliability Engineering (Contract, Remote)
Interactive Coding, Site Reliability Engineering (Contract, Remote)

Patronus AI • EE. UU.

A distancia
USD 170.000 - 243.000
Internal Strategic Projects Lead
Internal Strategic Projects Lead

Patronus AI • EE. UU.

A distancia
USD 115.000 - 225.000
Equity
Health insurance
Vacation days
+1