AI Evaluation Engineer

Wipro Limited

San Diego (CA)

On-site

USD 60,000 - 149,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Medical and dental benefits
Disability insurance
Paid time off

Job summary

Wipro Limited in San Diego, CA, is seeking an engineer to build and scale evaluation tooling for AI-powered software development. You will develop evaluation harnesses, automate benchmark runs, and ensure results are reproducible and auditable through CI/CD environments.

The role blends engineering with measurement discipline, demanding strong programming skills (Python/Java/JavaScript), experience with Docker and Git, and familiarity with LLM-based evaluation and agent benchmarking.

Qualifications

  • Strong software engineering background with automation experience.
  • Experience evaluating AI coding agents and validating code changes.
  • Ability to build reproducible evaluation workflows and environments.
  • Familiarity with Git history, branches, and CI/CD pipelines.
  • Knowledge of Docker/containerization and API-driven workflows.
  • Experience with AI/LLM-based evaluation and tool benchmarking.

Responsibilities

  • Build and integrate evaluation harnesses and automation for software development use cases.
  • Create versioned, repeatable evaluation processes with reproducible runs.
  • Validate evaluation approaches against human judgment to ensure correctness.
  • Support benchmarking across quality, productivity, and cost metrics.
  • Analyze results across runs and improve reliability and automation.
  • Document evaluations for technical and leadership audiences.

Skills

Software engineering
Automation tooling
CI/CD
Git
Docker
Python/Java/JavaScript
LLM evaluation
Benchmarking
Troubleshooting
Team collaboration

Tools

Docker
Git
CI/CD tools
Claude Code
Devin
Cursor

Job description

Wipro Limited (NYSE: WIT, BSE: 507685, NSE: WIPRO) is a leading technology services and consulting company focused on building innovative solutions that address clients’ most complex digital transformation needs. Leveraging our holistic portfolio of capabilities in consulting, design, engineering, and operations, we help clients realize their boldest ambitions and build future-ready, sustainable businesses. With over 230,000 employees and business partners across 65 countries, we deliver on the promise of helping our customers, colleagues, and communities thrive in an ever-changing world.

For additional information, visit us at www.wipro.com.

Job Description Role Overview

Help build and scale the tooling we use to measure how well AI-powered software development tools actually perform. You'll develop evaluation harnesses, automate benchmark runs, and help make sure the results we produce are reproducible and hold up to scrutiny. This is an engineering role, but a lot of the work is about getting the measurement right, not just automating it.

Key Responsibilities
  • Build and integrate evaluation harnesses and automation for software development use cases, including turning real engineering artifacts like merged pull requests into repeatable benchmark tasks.
  • Build versioned, repeatable processes to evaluate AI tools, models, and harnesses, with reproducible run environments (pinned dependencies, containerized runs, isolated worktrees) so results stay comparable over time.
  • Validate and calibrate evaluation approaches against human judgment, so scores are consistent and correct rather than just repeatable.
  • Support execution-based benchmarking across quality, productivity, and e iciency measures, including cost and latency.
  • Analyze results across repeated runs, looking at variance, failure patterns, and cost per outcome, and find ways to make the workflows more reliable and more automated.
  • Work with engineering and data teams to improve the tooling, and document how the evaluations work and what they found for both technical and leadership audiences.
Required Skills & Experience
  • Strong software engineering background, with real experience building automation, developer tooling, or test and validation systems.
  • Coding-agent evaluation: Evaluating AI coding agents that modify code repositories, including validating generated code changes against expected outcomes.
  • Evaluation harnesses: Building automated, reproducible evaluation workflows, including test execution, environment setup, and result validation.
  • Benchmarking and reliability: Establishing baselines, measuring run-to-run variance, analyzing failures, and ensuring consistent evaluation results.
  • Git and CI/CD integration: Working with repository history, branches, pull requests, and automated testing within CI/CD pipelines.
  • LLM-as-a-Judge: Using LLMs to evaluate coding-agent outputs, including calibration against human assessments.
  • Proficient in at least one general-purpose language such as Python, Java, or JavaScript — the specific language background is flexible.
  • Solid working knowledge of Git, including how branches, history, and working trees behave, and of containerization with Docker.
  • Experience with APIs, development environments, CI/CD pipelines, and typical engineering workflows.
  • Understanding of how AI, LLM, or agent evaluation works and where it goes wrong, such as why a judge can be consistent but still wrong, why a single run can mislead, and how benchmark contamination happens.
  • Able to troubleshoot technical problems, think clearly about whether a measurement is valid, and analyze results carefully.
  • Hands-on experience using AI coding tools and agentic harnesses such as Claude Code, Devin, or Cursor, and command of the best practices for working with them effectively.

Mandatory Skills: Cloud Product & Platform Testing . Experience: 5-8 Years . The expected compensation for this role ranges from $60,000 to $148,500 . Final compensation will depend on various factors, including your geographical location, minimum wage obligations, skills, and relevant experience.

  • Based on the position, the role is also eligible for Wipro's standard benefits including a full range of medical and dental benefits options, disability insurance, paid time off (inclusive of sick leave), other paid and unpaid leave options.

Applicants are advised that employment in some roles may be conditioned on successful completion of a post-offer drug screening, subject to applicable state law.

Wipro provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.

Applications from veterans and people with disabilities are explicitly welcome.

Reinvent your world. We are building a modern Wipro. We are an end-to-end digital transformation partner with the boldest ambitions. To realize them, we need people inspired by reinvention. Of yourself, your career, and your skills. We want to see the constant evolution of our business and our industry. It has always been in our DNA - as the world around us changes, so do we. Join a business powered by purpose and a place that empowers you to design your own reinvention.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

DEVELOPER L3(CONTRACT)
DEVELOPER L3(CONTRACT)

Wipro Limited • Town of Alden (NY)

On-site
USD 45,000 - 110,000
Medical and dental benefits
Disability insurance
Paid time off and leave options
DEVELOPER L4
DEVELOPER L4

Wipro Ltd. • Washington

On-site
USD 60,000 - 149,000
Medical and dental benefits
Disability insurance
Paid time off
+1
AI LEAD L1
AI LEAD L1

Wipro • Chicago (IL)

On-site
USD 60,000 - 135,000
Medical and dental benefits
Disability insurance
Paid time off
Senior Data Scientist - AI/ML & Generative AI Lead
Senior Data Scientist - AI/ML & Generative AI Lead

Wipro Ltd. • Atlanta (GA)

On-site
USD 45,000 - 121,000
Medical and Dental Benefits
Disability Insurance
Paid Time Off
+1
AI Engineer
AI Engineer

Wipro Limited • Chicago (IL)

On-site
USD 45,000 - 110,000
Medical and dental benefits
Disability insurance
Paid time off
AI Lead
AI Lead

Wipro • City of Syracuse (NY)

On-site
USD 60,000 - 135,000
Medical and dental benefits
Disability insurance
Paid time off
AI LEAD L1(CONTRACT)
AI LEAD L1(CONTRACT)

Wipro • Wilmington (DE)

On-site
USD 60,000 - 135,000
AI/LLM Enginer
AI/LLM Enginer

Wipro Limited • San Jose (CA)

On-site
USD 80,000 - 174,000
AI/ ML engineer
AI/ ML engineer

Wipro Limited • Minneapolis (MN)

On-site
USD 80,000 - 158,000
Forward Deployed Engineer (Junior FDE)
Forward Deployed Engineer (Junior FDE)

NextBigWhat • Seattle (WA)

On-site
USD 60,000 - 149,000
Medical and dental benefits
Paid time off
Disability insurance