Remote AI Evaluation & Automation Engineer

IDR, Inc.; Santa Clara (CA); Remote; USD 120,000 - 170,000; Full time

Benefits offered by this job

Competitive compensation package

Full Benefits: Medical, Vision, Dental

Opportunity with industry leading IDR,

Dedicated Engagement Manager

Job summary

IDR, Inc. is seeking a Software Engineer - AI Evaluation & Automation for a remote role.

You will help build tooling to measure performance of AI-powered development tools and develop evaluation harnesses, automating benchmark runs to ensure reproducible, trustworthy results. You will validate measurement approaches against human judgment and collaborate with engineering and data teams to enhance tooling and document evaluation processes.

Qualifications

• Strong software engineering background with automation/developer tooling experience.
• Experience with Git, containers (Docker), and branch management.
• Familiarity with APIs, CI/CD pipelines, and engineering workflows.
• Some exposure to AI, LLM, or agent evaluation concepts.

Responsibilities

• Build and scale tooling to measure performance of AI-powered software development tools.
• Develop evaluation harnesses and automate benchmark runs.
• Ensure results are reproducible and trustworthy.
• Validate measurement approaches against human judgment and analyze results.
• Collaborate with engineering and data teams to improve tooling and document processes.
• Support execution-based benchmarking across quality, productivity, and efficiency metrics.

Skills

Automation tooling

Validation systems

Git & Docker

APIs & CI/CD

AI/LLM evaluation familiarity

Tools

Git

Docker

Job description

IDR, Inc. is seeking a Software Engineer - AI Evaluation & Automation for a remote role.

You will help build tooling to measure performance of AI-powered development tools and develop evaluation harnesses, automating benchmark runs to ensure reproducible, trustworthy results. You will validate measurement approaches against human judgment and collaborate with engineering and data teams to enhance tooling and document evaluation processes.