Software Engineer — AI Evaluation & Automation

Virtue America; California; Presencial; USD 120.000 - 160.000; Jornada completa

Descripción de la vacante

Virtue America is seeking a Software Engineer for AI Evaluation & Automation to design and grow evaluation harnesses, automate benchmark runs, and ensure reproducible results across AI-powered tooling. You will work across engineering and data teams to pin dependencies, containerize runs, calibrate approaches against human judgment, and communicate findings to technical and leadership audiences.

You will contribute to building scalable evaluation infrastructure, work with modern tooling, and

Formación

• Strong software engineering background with automation or tooling experience.
• Git knowledge and containerization with Docker.
• Experience with APIs, development environments, and CI/CD pipelines.
• Familiarity with AI/LLM evaluation concepts and benchmarking concerns.
• Ability to troubleshoot, reason about measurement validity, and analyze results.

Responsabilidades

• Build evaluation harnesses and automation for software development use cases.
• Create reproducible evaluation processes with pinned dependencies and containers.
• Calibrate evaluation approaches against human judgment.
• Support benchmarking across quality, productivity, and cost metrics.
• Analyze results for variance and reliability, and automate workflows.
• Collaborate with engineering and data teams and document findings.

Conocimientos

Strong software engineering

Git

Docker

CI/CD pipelines

APIs

AI/LLM evaluation basics

Herramientas

Git

Docker

CI/CD pipelines

Descripción del empleo

Title: Software Engineer — AI Evaluation & Automation

Role Overview

Help build and scale the tooling we use to measure how well AI-powered software development tools actually perform. You'll develop evaluation harnesses, automate benchmark runs, and help make sure the results we produce are reproducible and hold up to scrutiny. This is an engineering role, but a lot of the work is about getting the measurement right, not just automating it.

Key Responsibilities

• Build and integrate evaluation harnesses and automation for software development use cases, including turning real engineering artifacts like merged pull requests into repeatable


• Build versioned, repeatable processes to evaluate AI tools, models, and harnesses, with reproducible run environments (pinned dependencies, containerized runs, isolated worktrees) so results stay comparable over time.


• Validate and calibrate evaluation approaches against human judgment, so scores are consistent and correct rather than just repeatable.


• Support execution-based benchmarking across quality, productivity, and efficiency measures, including cost and latency.


• Analyze results across repeated runs, looking at variance, failure patterns, and cost per outcome, and find ways to make the workflows more reliable and more automated.


• Work with engineering and data teams to improve the tooling, and document how the evaluations work and what they found for both technical and leadership audiences.

Required Skills & Experience

• Strong software engineering background, with real experience building automation, developer tooling, or test and validation systems.


• language.


• Solid working knowledge of Git, including how branches, history, and working trees behave, and of containerization with Docker.


• Experience with APIs, development environments, CI/CD pipelines, and typical


• Some familiarity with how AI, LLM, or agent evaluation works and where it goes wrong, such as why a judge can be consistent but still wrong, why a single run can mislead, and how benchmark contamination happens.


• Able to troubleshoot technical problems, think clearly about whether a measurement is valid, and analyze results carefully.

Preferred Experience

• Hands-on work with AI-powered coding tools and agentic applications, such as Claude Code, Devin, or OpenCode.


• Experience designing benchmarks or evaluations for software systems, especially execution-based grading that verifies against tests.


• Familiarity with LLM-as-judge or agent-as-judge approaches, and how to check them


• Experience with build-system-aware test selection, such as Bazel or mapping changed files to the tests that cover them.


• Experience building reproducible test environments and managing versioned evaluation datasets.


• Comfortable writing up methodology and results for engineering leadership.