Remote AI Engineer — Agent Evaluation Specialist

Technology Management Solutions LLC; Los Angeles (CA); Remote; USD 96,000 - 165,000; Full time

Job summary

Technology Management Solutions LLC is seeking an AI Engineer for hands-on agent evaluation work. The role focuses on evaluating LLM-powered agents, not building them, and requires practical experience with evaluation frameworks and agent testing.

Remote contract position, 3+ months, with emphasis on ensuring response quality, tool usage effectiveness, and reliability across agent workflows.

Qualifications

• Hands-on experience evaluating LLM-powered agents.
• Ability to design and run evaluation frameworks.
• Experience creating benchmark datasets for agents.
• Instrumenting agents to capture evaluation metrics.

Skills

LLM evaluation

Langfuse

Agent evaluation

Golden dataset

Tools

AWS Bedrock

Claude Agent SDK

Job description

Technology Management Solutions LLC is seeking an AI Engineer for hands-on agent evaluation work. The role focuses on evaluating LLM-powered agents, not building them, and requires practical experience with evaluation frameworks and agent testing.

Remote contract position, 3+ months, with emphasis on ensuring response quality, tool usage effectiveness, and reliability across agent workflows.