AI Evaluation Engineer: Benchmarking & Automation

Akraya, Inc.; San Diego (CA); On-site; USD 90,000 - 101,000; Part time

Job summary

Akraya, Inc. in San Diego, CA is seeking a role focused on building and scaling evaluation systems for AI-powered software development tools. You will engineer evaluation frameworks and automate benchmarking to ensure reproducible, reliable results with emphasis on measurement accuracy.

The role requires strong Python and Git skills, Docker containerization, and experience with AI coding tools and benchmarking. Contract is 6 months with potential extension, paid hourly at $65–$73.

Qualifications

• Experience with AI coding tools and harnesses.
• Proven evaluation and benchmarking experience.
• Strong proficiency in Git and containerization with Docker.
• Python, Java or JavaScript

Responsibilities

• Develop and integrate evaluation systems for AI development tools.
• Automate benchmark runs with versioned, reproducible environments.
• Calibrate evaluation metrics against human judgment for accuracy.
• Support performance benchmarking across quality and efficiency.
• Analyze and improve the reliability of evaluation workflows.

Skills

Python

Git

AI coding tools

API Integration

Docker

Tools

Docker

Job description

Akraya, Inc. in San Diego, CA is seeking a role focused on building and scaling evaluation systems for AI-powered software development tools. You will engineer evaluation frameworks and automate benchmarking to ensure reproducible, reliable results with emphasis on measurement accuracy.

The role requires strong Python and Git skills, Docker containerization, and experience with AI coding tools and benchmarking. Contract is 6 months with potential extension, paid hourly at $65–$73.