AI Evaluation Engineer — Benchmarking Automation Lead

Wipro Technologies; San Diego (CA); On-site; USD 60,000 - 149,000; Full time

Benefits offered by this job

Medical and dental benefits

Disability insurance

Paid time off

Job summary

Wipro Limited is seeking an engineer to help build and scale tools that measure AI-powered software development performance. The role involves creating evaluation harnesses, automating benchmarks, and ensuring reproducible results across environments.

You will work with engineering and data teams to document findings for technical and leadership audiences. The position emphasizes hands-on automation, versioned workflows, and calibration against human judgments, with exposure to Docker, Git, and

Qualifications

• Strong software engineering background with automation and tooling experience.
• Experience evaluating AI coding agents and calibrating against human judgments.
• Experience building reproducible evaluation workflows (test execution, env setup, validation).
• Familiar with Git, CI/CD, and containerized workloads.

Responsibilities

• Build and integrate evaluation harnesses and automation for software dev use cases.
• Create versioned, repeatable evaluation processes with pinned deps and containerized runs.
• Validate evaluation approaches against human judgment for accuracy.
• Support benchmarking across quality, productivity, and cost metrics.
• Analyze results for variance, failures, and cost per outcome; improve workflows.
• Collaborate with engineering and data teams to document evaluations for technical and leadership audiences.

Skills

Software engineering

Automation

Dev tooling

Test & validation

AI evaluation concepts

LLM as a judge

Python/Java/JS

Git & CI/CD

Troubleshooting

AI coding tools

Tools

Docker

Git

Claude Code

Devin

Cursor

Job description

Wipro Limited is seeking an engineer to help build and scale tools that measure AI-powered software development performance. The role involves creating evaluation harnesses, automating benchmarks, and ensuring reproducible results across environments.

You will work with engineering and data teams to document findings for technical and leadership audiences. The position emphasizes hands-on automation, versioned workflows, and calibration against human judgments, with exposure to Docker, Git, and