Software Engineer AI Evaluation & Automation : 26-02760

Akraya; San Diego (CA); Remote; USD 90,000 - 101,000; Full time

Job summary

Akraya, Inc. is recruiting for a role focused on building and scaling systems that measure the performance of AI-powered software development tools in San Diego. You will engineer evaluation frameworks and automate benchmarking to ensure reproducible results.

The emphasis is on measurement accuracy as much as automation. The position requires strong Python and Git skills, experience with AI coding tools, and Docker-based environments, with a 6-month contract and potential extension.

Qualifications

• Experience with Python and Git in software development.
• Proven ability to benchmark AI tools and measure performance.
• Familiarity with containerization and Docker-based environments.

Responsibilities

• Develop and integrate evaluation systems for AI development tools.
• Automate benchmark runs with versioned, reproducible environments.
• Calibrate evaluation metrics against human judgment for accuracy.
• Support performance benchmarking across quality and efficiency.
• Analyze and improve the reliability of evaluation workflows.

Skills

Python

Git

AI coding tools

API Integration

Tools

Docker

Job description

Primary Skills: Python (advanced), Git (advanced), AI coding tools (advanced), API Integration (intermediate), Docker (Proficient). Contract Type: W2. Duration: 6 months (Possible extension). Location: San Diego, CA (#LI-Remote PST). Pay Range: $65.00- $73.00 per hour #LP

Job Summary

This role focuses on building and scaling the systems used to measure the performance of AI-powered software development tools. You will engineer evaluation frameworks and automate benchmarking processes to ensure reproducible and reliable results. The core of this position is ensuring the accuracy of measurements, not solely the automation of tasks.

Key Responsibilities

• Develop and integrate evaluation systems for AI development tools.
• Automate benchmark runs with versioned, reproducible environments.
• Calibrate evaluation metrics against human judgment for accuracy.
• Support performance benchmarking across quality and efficiency.
• Analyze and improve the reliability of evaluation workflows.

Must-Have Skills

• Experience with AI coding tools and harnesses.
• Proven evaluation and benchmarking experience.
• Strong proficiency in Git and containerization with Docker.
• Python, Java or JavaScript.

Industry Experience

• Experience in software development, particularly with AI-assisted coding tools and their evaluation, is highly beneficial.

ABOUT AKRAYA

Akraya is an award-winning IT staffing firm consistently recognized for our commitment to excellence and a thriving work environment. Most recently, we were recognized Stevie Employer of the Year 2025, SIA Best Staffing Firm to work for 2025, Inc 5000 Best Workspaces in US (2025 & 2024) and Glassdoor's Best Places to Work (2023 & 2022)! Industry Leaders in Tech Staffing As Talent solutions provider for Fortune 100 Organizations, Akraya's industry recognitions solidify our leadership position in the IT staffing space. We don't just connect you with great jobs, we connect you with a workplace that inspires!