AI Engineer — Agent Evaluation & LLM Benchmarking

ZealHire Inc.; Los Angeles (CA); Sur place; USD 83 000 - 179 000; Temps partiel

Résumé du poste

ZealHire Inc. is seeking an AI Engineer specialized in evaluating LLM-powered agents. The role focuses on hands-on design, implementation, and execution of agent evaluation frameworks to assess response quality, reliability, and tool usage.

Candidate will develop golden datasets with Business Analysts, instrument agent executions using Langfuse, and drive end-to-end agent evaluation programs. This is a 3+ month contract in the Los Angeles area with potential extension.

Qualifications

• Must have hands-on experience designing, implementing, and running LLM-based evaluation frameworks for agent performance.
• Experience collaborating with analysts to build gold datasets used as benchmarks.
• Practical experience instrumenting agents with Langfuse and tracing executions.
• Strong understanding of end-to-end agent evaluation, tool use, and reasoning quality.

Responsabilités

• Lead practical evaluation campaigns for AI agents to measure accuracy, relevance and reliability.
• Develop and maintain evaluation datasets and benchmarks.
• Instrument agent runs with Langfuse and monitor performance metrics.

Connaissances

LLM evaluation frameworks

Golden dataset development

Langfuse instrumentation

Agent evaluation expertise

Description du poste

ZealHire Inc. is seeking an AI Engineer specialized in evaluating LLM-powered agents. The role focuses on hands-on design, implementation, and execution of agent evaluation frameworks to assess response quality, reliability, and tool usage.

Candidate will develop golden datasets with Business Analysts, instrument agent executions using Langfuse, and drive end-to-end agent evaluation programs. This is a 3+ month contract in the Los Angeles area with potential extension.