Software Engineer — AI Evaluation & Automation : 26-02760

Akraya, Inc.; San Diego (CA); On-site; USD 90,000 - 101,000; Part time

Job summary

Akraya, Inc. in San Diego, CA is seeking a role focused on building and scaling evaluation systems for AI-powered software development tools. You will engineer evaluation frameworks and automate benchmarking to ensure reproducible, reliable results with emphasis on measurement accuracy.

The role requires strong Python and Git skills, Docker containerization, and experience with AI coding tools and benchmarking. Contract is 6 months with potential extension, paid hourly at $65–$73.

Qualifications

• Experience with AI coding tools and harnesses.
• Proven evaluation and benchmarking experience.
• Strong proficiency in Git and containerization with Docker.
• Python, Java or JavaScript

Responsibilities

• Develop and integrate evaluation systems for AI development tools.
• Automate benchmark runs with versioned, reproducible environments.
• Calibrate evaluation metrics against human judgment for accuracy.
• Support performance benchmarking across quality and efficiency.
• Analyze and improve the reliability of evaluation workflows.

Skills

Python

Git

AI coding tools

API Integration

Docker

Tools

Docker

Job description

Primary Skills: Python (advanced), Git (advanced), AI coding tools (advanced), API Integration (intermediate), Docker (Proficient)
Contract Type: W2
Duration: 6 months (Possible extension)
Location: San Diego, CA ( PST)
Pay Range: $65.00- $73.00 per hour
#LP

Job Summary:

This role focuses on building and scaling the systems used to measure the performance of AI-powered software development tools. You will engineer evaluation frameworks and automate benchmarking processes to ensure reproducible and reliable results. The core of this position is ensuring the accuracy of measurements, not solely the automation of tasks.

Key Responsibilities:

• Develop and integrate evaluation systems for AI development tools.
• Automate benchmark runs with versioned, reproducible environments.
• Calibrate evaluation metrics against human judgment for accuracy.
• Support performance benchmarking across quality and efficiency.
• Analyze and improve the reliability of evaluation workflows.

Must-Have Skills:

• Experience with AI coding tools and harnesses.
• Proven evaluation and benchmarking experience.
• Strong proficiency in Git and containerization with Docker.
• Python, Java or JavaScript

Industry Experience:

Experience in software development, particularly with AI-assisted coding tools and their evaluation, is highly beneficial.

ABOUT AKRAYA:

Akraya is an award-winning IT staffing firm consistently recognized for our commitment to excellence and a thriving work environment. Most recently, we were recognized Stevie Employer of the Year 2025, SIA Best Staffing Firm to work for 2025, Inc 5000 Best Workspaces in US (2025 & 2024) and Glassdoor's Best Places to Work (2023 & 2022)!

Industry Leaders in Tech Staffing

As Talent solutions provider for Fortune 100 Organizations, Akraya's industry recognitions solidify our leadership position in the IT staffing space. We don't just connect you with great jobs, we connect you with a workplace that inspires!