Software Engineer - AI Evaluation & Automation

IDR, Inc.; Santa Clara (CA); À distance; USD 120 000 - 170 000; Plein temps

Avantages offerts par ce poste

Competitive compensation package

Full Benefits: Medical, Vision, Dental

Opportunity with industry leading IDR,

Dedicated Engagement Manager

Résumé du poste

IDR, Inc. is seeking a Software Engineer - AI Evaluation & Automation for a remote role.

You will help build tooling to measure performance of AI-powered development tools and develop evaluation harnesses, automating benchmark runs to ensure reproducible, trustworthy results. You will validate measurement approaches against human judgment and collaborate with engineering and data teams to enhance tooling and document evaluation processes.

Qualifications

• Strong software engineering background with automation/developer tooling experience.
• Experience with Git, containers (Docker), and branch management.
• Familiarity with APIs, CI/CD pipelines, and engineering workflows.
• Some exposure to AI, LLM, or agent evaluation concepts.

Responsabilités

• Build and scale tooling to measure performance of AI-powered software development tools.
• Develop evaluation harnesses and automate benchmark runs.
• Ensure results are reproducible and trustworthy.
• Validate measurement approaches against human judgment and analyze results.
• Collaborate with engineering and data teams to improve tooling and document processes.
• Support execution-based benchmarking across quality, productivity, and efficiency metrics.

Connaissances

Automation tooling

Validation systems

Git & Docker

APIs & CI/CD

AI/LLM evaluation familiarity

Outils

Git

Docker

Description du poste

IDR is seeking a Software Engineer - AI Evaluation & Automation to join one of our top clients for a remote opportunity. This role is within the technology sector, focusing on developing tools to measure the performance of AI-powered software development solutions. The position offers the chance to work on innovative projects that ensure the reliability and reproducibility of AI evaluations.

Position Overview for the Software Engineer - AI Evaluation & Automation:

• Help build and scale tooling used to measure the performance of AI-powered software development tools.
• Develop evaluation harnesses, automate benchmark runs, and ensure results are reproducible and trustworthy.
• Focus on validating measurement approaches against human judgment and analyzing benchmarking results.
• Collaborate with engineering and data teams to improve tooling and document evaluation processes.
• Support execution-based benchmarking across quality, productivity, and efficiency measures, including cost and latency.

Requirements for the Software Engineer - AI Evaluation & Automation:

• Strong software engineering background, with experience building automation, developer tooling, or validation systems.
• Solid working knowledge of Git, including branch management, history, and worktrees, and containerization with Docker.
• Experience with APIs, development environments, CI/CD pipelines, and engineering workflows.
• Some familiarity with AI, LLM, or agent evaluation processes, including understanding common pitfalls.

What's in it for you?

• Competitive compensation package
• Full Benefits; Medical, Vision, Dental, and more!
• Opportunity to get in with an industry leading organization.

Why IDR?

• 25+ Years of Proven Industry Experience in 4 major markets
• Employee Stock Ownership Program
• Dedicated Engagement Manager who is committed to you and your success.
• Medical, Dental, Vision, and Life Insurance
• ClearlyRated's Best of Staffing® Client and Talent Award winner 12 years in a row.