Hiring | AI Engineer – Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote

Technology Management Solutions LLC; Los Angeles (CA); Remote; USD 96,000 - 165,000; Full time

Job summary

Technology Management Solutions LLC is seeking an AI Engineer for hands-on agent evaluation work. The role focuses on evaluating LLM-powered agents, not building them, and requires practical experience with evaluation frameworks and agent testing.

Remote contract position, 3+ months, with emphasis on ensuring response quality, tool usage effectiveness, and reliability across agent workflows.

Qualifications

• Hands-on experience evaluating LLM-powered agents.
• Ability to design and run evaluation frameworks.
• Experience creating benchmark datasets for agents.
• Instrumenting agents to capture evaluation metrics.

Skills

LLM evaluation

Langfuse

Agent evaluation

Golden dataset

Tools

AWS Bedrock

Claude Agent SDK

Job description

Hiring | AI Engineer - Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote

• Contract
• Duration: 3+ Months (Possible Extension)
• Primary Focus: Hands-on Agent Evaluation
• Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & Evaluation

We are looking for an AI Engineer with strong, practical experience in evaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.

Key Requirements (Must-Have):

• LLM-as-a-Judge: Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
• Golden Dataset Development: Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
• Langfuse: Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
• Agent Evaluation: Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.

Critical Screening Criteria:

Candidates must demonstrate actual implementation experience with agent evaluation frameworks, not just theoretical knowledge or general GenAI/LLM development.

Agent evaluation is the core requirement and will be the primary selection criterion.

All your information will be kept confidential according to EEO guidelines.