AI Software Evaluator — Python & ML Systems (Contract)

turing; California (MO); Hybrid; USD 14,000 - 55,000; Part time

Job summary

Turing in San Francisco, CA seeks a Software Engineering evaluator to create datasets for training and benchmarking large language models, collaborating with researchers.

The role emphasizes Python across backend services, data pipelines, and ML infrastructure, with JavaScript (including ReactJS), C/C++, Java, Rust, and Go also used; you will evaluate and refine AI-generated code for efficiency and reliability.

Qualifications

• 3+ years of software engineering experience.
• Strong Python expertise with production-grade software.
• Experience building full-stack applications and deploying scalable software.
• Deep understanding of software architecture, design, development, debugging, and code quality.

Responsibilities

• Curate code examples and build solutions, primarily in Python.
• Evaluate and refine AI-generated code to ensure efficiency, scalability, and reliability.
• Collaborate with cross-functional teams to enhance AI-driven coding solutions against industry benchmarks.
• Build agents and automated verification tools in Python to verify code quality and detect error patterns.
• Hypothesize on steps in the software engineering cycle and evaluate model capabilities on them.
• Design verification mechanisms that can automatically verify solutions to software engineering tasks.

Skills

Python

JavaScript (ReactJS)

C/C++

Java

Rust

Go

Job description

Turing in San Francisco, CA seeks a Software Engineering evaluator to create datasets for training and benchmarking large language models, collaborating with researchers.

The role emphasizes Python across backend services, data pipelines, and ML infrastructure, with JavaScript (including ReactJS), C/C++, Java, Rust, and Go also used; you will evaluate and refine AI-generated code for efficiency and reliability.