Remote AI Evaluation & Quality Assurance Engineer

Equiliem; United States; Remote; USD 69,000 - 74,000; Full time

Job summary

Equiliem is seeking an AI Evaluation/Test Engineer to test AI-powered agent experiences across customer and internal workflows. The role focuses on prompt and agent evaluation, QA testing, observability, and defect management.

You will collaborate with product managers, engineers, and stakeholders to identify quality gaps and improve reliability, accuracy, and user experience in AI agents. Remote work is offered.

Qualifications

• Experience testing AI-powered agent experiences across customer and internal workflows.
• Familiarity with prompt evaluation and agent quality assurance.
• Ability to design evaluation frameworks and scoring methodologies for AI output quality.

Responsibilities

• Test and evaluate prompts and agent workflows for accuracy, groundedness, and user satisfaction.
• Design and execute test plans for AI agents, web apps, APIs, and backend services.
• Create test cases covering accuracy, hallucination detection, safety, tone and style adherence, instruction following, context retention and response consistency.
• Track, prioritize, validate, and retest defects using Azure DevOps.
• Analyze model and agent performance with observability tools like Arize and develop quality metrics and dashboards.

Skills

Prompt evaluation

AI QA/testing

Azure DevOps

Arize observability

Test framework design

Data analysis

Communication

Tools

Azure DevOps

Arize

Job description

Equiliem is seeking an AI Evaluation/Test Engineer to test AI-powered agent experiences across customer and internal workflows. The role focuses on prompt and agent evaluation, QA testing, observability, and defect management.

You will collaborate with product managers, engineers, and stakeholders to identify quality gaps and improve reliability, accuracy, and user experience in AI agents. Remote work is offered.