AI Software Evaluation Engineer - Remote Contract

agency; United States; Remote; USD 8,300 - 90,000; Full time

Job summary

agency is seeking a role focused on testing AI systems’ software engineering capabilities through coding tasks and programming questions. You will assess responses, check code correctness, and document errors to improve model performance.

The work spans algorithms, debugging, APIs, concurrency, cloud-native development, and DevOps, with remote contract terms and pay based on experience. The position emphasizes clear reasoning, documentation, and the ability to articulate technical conclusions

Qualifications

• Strong software development knowledge including algorithms, data structures, debugging, testing, version control, and code optimization.
• Familiarity with programming paradigms, API design, concurrency, cloud-native development, or DevOps.
• Clear written communication and ability to show how you reached a technical conclusion.
• A degree in computer science, software engineering, or a related field is preferred.
• Relevant projects, research publications, or open-source contributions can demonstrate fit.

Responsibilities

• Present coding scenarios and technical questions to AI models, then assess the responses.
• Check code for correctness, sound logic, and alignment with software engineering practices.
• Record reproducible errors and describe the conditions that cause them.
• Recommend improvements to prompts and evaluation methods based on observed results.
• Explain your reasoning clearly so findings can be reviewed and used in model training.

Skills

Algorithms

Data structures

Debugging

Testing

Version control

Code optimization

Clear communication

Education

CS/Software Eng degree preferred

Tools

API design

Concurrency

Cloud-native development

DevOps

Job description

agency is seeking a role focused on testing AI systems’ software engineering capabilities through coding tasks and programming questions. You will assess responses, check code correctness, and document errors to improve model performance.

The work spans algorithms, debugging, APIs, concurrency, cloud-native development, and DevOps, with remote contract terms and pay based on experience. The position emphasizes clear reasoning, documentation, and the ability to articulate technical conclusions