AI Engineer – Agent Evaluation Specialist

ZealHire Inc.; Los Angeles (CA); On-site; USD 83,000 - 179,000; Part time

Job summary

ZealHire Inc. is seeking an AI Engineer specialized in evaluating LLM-powered agents. The role focuses on hands-on design, implementation, and execution of agent evaluation frameworks to assess response quality, reliability, and tool usage.

Candidate will develop golden datasets with Business Analysts, instrument agent executions using Langfuse, and drive end-to-end agent evaluation programs. This is a 3+ month contract in the Los Angeles area with potential extension.

Qualifications

• Must have hands-on experience designing, implementing, and running LLM-based evaluation frameworks for agent performance.
• Experience collaborating with analysts to build gold datasets used as benchmarks.
• Practical experience instrumenting agents with Langfuse and tracing executions.
• Strong understanding of end-to-end agent evaluation, tool use, and reasoning quality.

Responsibilities

• Lead practical evaluation campaigns for AI agents to measure accuracy, relevance and reliability.
• Develop and maintain evaluation datasets and benchmarks.
• Instrument agent runs with Langfuse and monitor performance metrics.

Skills

LLM evaluation frameworks

Golden dataset development

Langfuse instrumentation

Agent evaluation expertise

Job description

Duration: 3+ Months (Possible Extension)

Visa: USC, GC ( with PP number)

LinkedIn should match resume and should have proper connection, photo and created before 2021

References with LinkedIn

Primary Focus: Hands-on Agent Evaluation

Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & Evaluation

Critical Screening Criteria:

Candidates must demonstrate actual implementation experience with agent evaluation frameworks, not just theoretical knowledge or general GenAI/LLM development.

Agent evaluation is the core requirement and will be the primary selection criterion.

We are looking for an AI Engineer with strong, practical experience in evaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.

Key Requirements (Must-Have):

• LLM-as-a-Judge: Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
• Golden Dataset Development: Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
• Langfuse: Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
• Agent Evaluation: Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.

$000,000 – $000,000 Salary range is not shown on the candidate portal