AI Evaluation/Test Engineer

Equiliem; United States; Remote; USD 69,000 - 74,000; Full time

Job summary

Equiliem is seeking an AI Evaluation/Test Engineer to test AI-powered agent experiences across customer and internal workflows. The role focuses on prompt and agent evaluation, QA testing, observability, and defect management.

You will collaborate with product managers, engineers, and stakeholders to identify quality gaps and improve reliability, accuracy, and user experience in AI agents. Remote work is offered.

Qualifications

• Experience testing AI-powered agent experiences across customer and internal workflows.
• Familiarity with prompt evaluation and agent quality assurance.
• Ability to design evaluation frameworks and scoring methodologies for AI output quality.

Responsibilities

• Test and evaluate prompts and agent workflows for accuracy, groundedness, and user satisfaction.
• Design and execute test plans for AI agents, web apps, APIs, and backend services.
• Create test cases covering accuracy, hallucination detection, safety, tone and style adherence, instruction following, context retention and response consistency.
• Track, prioritize, validate, and retest defects using Azure DevOps.
• Analyze model and agent performance with observability tools like Arize and develop quality metrics and dashboards.

Skills

Prompt evaluation

AI QA/testing

Azure DevOps

Arize observability

Test framework design

Data analysis

Communication

Tools

Azure DevOps

Arize

Job description

AI Evaluation/Test Engineer

Remote

Pay: $50.00 - $54.00 per hour

26-09816

Job summary

The AI Evaluation/Test Engineer is responsible for testing, evaluating, and validating AI-powered agent experiences across customer and internal business workflows. This role involves prompt and agent quality assurance, evaluation framework execution, defect management, and AI observability. The engineer collaborates with product managers, software engineers, and business stakeholders to identify quality gaps and improve agent reliability, accuracy, and user experience.

Responsibilities

Prompt & Agent Evaluation

• Test and evaluate existing prompts, system instructions, and agent workflows for accuracy, consistency, groundedness, and user satisfaction.
• Conduct structured prompt experiments and A/B test result analysis (execution and analysis, not prompt authorship).
• Document evaluation findings, failure patterns, and recommended refinements for the engineering team to implement.

AI Quality Assurance & Testing

• Design and execute comprehensive test plans covering AI agents, web applications, APIs, and backend services.
• Develop evaluation frameworks and scoring methodologies for AI output quality.
• Create test cases covering accuracy, hallucination detection, safety, tone and style adherence, instruction following, context retention, and response consistency.
• Perform regression testing for prompt and model changes.
• Track, prioritize, validate, and retest defects using Azure DevOps.

AI Evaluation & Observability

• Utilize Arize or similar observability platforms to monitor model and agent performance.
• Analyze user interactions and evaluation results to identify trends and improvement opportunities.
• Develop quality metrics and dashboards for accuracy, relevance, latency, user satisfaction, defect rates, and prompt performance.
• Investigate production issues and recommend prompt, workflow, or architectural improvements.

Documentation & Collaboration

• Maintain documentation for evaluation methodologies, testing procedures, and defect logs.
• Participate in sprint planning, backlog refinement, and Agile ceremonies.
• Communicate findings and recommendations to technical and non-technical stakeholders.
• Share best practices and lessons learned across the team.

Location

Remote work (home-based)

About Equiliem

Equiliem believes in empowering success. It's our job to cultivate relationships that connect people and employers in a way that is inclusive, intelligent, and allows both to thrive.

Across the U.S., leading companies in healthcare, government, engineering, manufacturing, professional services, and energy rely on us for their workforce solutions. Our recruiting and HR services include contract and direct hire staffing, Payrolling/EOR, Independent Contractor Compliance, and Managed Services.

For almost 30 years, we've helped shape our industry. Today, we continue to research, ask questions, and continuously enhance the candidate journey and client experience.

Benefits

We offer a comprehensive benefits package to our employees, which includes:

• Medical Insurance
• Vision & Dental Insurance
• Life Insurance
• Matching 401(k)
• Commuter Benefits
• Employee Discounts & Rewards
• Payroll Payment Options

Future Communication Consent

By applying, you are providing consent for Equiliem to engage with you via phone calls, AI-generated calls, AI-generated text messages, standard text messages, and/or emails to share job opportunities. Consistent with our Privacy Policy, information obtained will not be shared with third parties for marketing or promotional purposes. Message frequency may vary for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You may reply STOP to cancel or HELP for assistance.

EEO Employer

Equiliem is an equal opportunity employer. We do not discriminate or allow discrimination based on race, color, religion, creed, sex, age, sexual orientation, gender identity, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, disability, status as a crime victim, protected veteran status, political affiliation, union membership, or any other characteristic protected by law. Equiliem will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law. If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation in using our website for a search or application, please contact our Employee Services Department at [emailprotected] or [phone].