AI Tool Evaluation Engineer: Benchmarking & Automation

Akraya; San Diego (CA); Remote; USD 90,000 - 101,000; Full time

Job summary

Akraya, Inc. is recruiting for a role focused on building and scaling systems that measure the performance of AI-powered software development tools in San Diego. You will engineer evaluation frameworks and automate benchmarking to ensure reproducible results.

The emphasis is on measurement accuracy as much as automation. The position requires strong Python and Git skills, experience with AI coding tools, and Docker-based environments, with a 6-month contract and potential extension.

Qualifications

• Experience with Python and Git in software development.
• Proven ability to benchmark AI tools and measure performance.
• Familiarity with containerization and Docker-based environments.

Responsibilities

• Develop and integrate evaluation systems for AI development tools.
• Automate benchmark runs with versioned, reproducible environments.
• Calibrate evaluation metrics against human judgment for accuracy.
• Support performance benchmarking across quality and efficiency.
• Analyze and improve the reliability of evaluation workflows.

Skills

Python

Git

AI coding tools

API Integration

Tools

Docker

Job description

Akraya, Inc. is recruiting for a role focused on building and scaling systems that measure the performance of AI-powered software development tools in San Diego. You will engineer evaluation frameworks and automate benchmarking to ensure reproducible results.

The emphasis is on measurement accuracy as much as automation. The position requires strong Python and Git skills, experience with AI coding tools, and Docker-based environments, with a 6-month contract and potential extension.