AI Evaluation & Automation Engineer

Virtue America; California; Sur place; USD 120 000 - 160 000; Plein temps

Résumé du poste

Virtue America is seeking a Software Engineer for AI Evaluation & Automation to design and grow evaluation harnesses, automate benchmark runs, and ensure reproducible results across AI-powered tooling. You will work across engineering and data teams to pin dependencies, containerize runs, calibrate approaches against human judgment, and communicate findings to technical and leadership audiences.

You will contribute to building scalable evaluation infrastructure, work with modern tooling, and

Qualifications

• Strong software engineering background with automation or tooling experience.
• Git knowledge and containerization with Docker.
• Experience with APIs, development environments, and CI/CD pipelines.
• Familiarity with AI/LLM evaluation concepts and benchmarking concerns.
• Ability to troubleshoot, reason about measurement validity, and analyze results.

Responsabilités

• Build evaluation harnesses and automation for software development use cases.
• Create reproducible evaluation processes with pinned dependencies and containers.
• Calibrate evaluation approaches against human judgment.
• Support benchmarking across quality, productivity, and cost metrics.
• Analyze results for variance and reliability, and automate workflows.
• Collaborate with engineering and data teams and document findings.

Connaissances

Strong software engineering

Git

Docker

CI/CD pipelines

APIs

AI/LLM evaluation basics

Outils

Git

Docker

CI/CD pipelines

Description du poste

Virtue America is seeking a Software Engineer for AI Evaluation & Automation to design and grow evaluation harnesses, automate benchmark runs, and ensure reproducible results across AI-powered tooling. You will work across engineering and data teams to pin dependencies, containerize runs, calibrate approaches against human judgment, and communicate findings to technical and leadership audiences.

You will contribute to building scalable evaluation infrastructure, work with modern tooling, and