AI Evaluation Specialist

Onsite It Advisors LLC; Atlanta (GA); Remote; USD 90,000 - 130,000; Full time

Job summary

Onsite It Advisors LLC in the United States seeks an experienced AI Evaluation Specialist to test and evaluate AI tools for accuracy, reliability, safety, and business readiness. You will work across platforms, including OpenAI products and Rosalind, to design tests using real-world scenarios.

Collaborating with product managers, data scientists, security and privacy teams, you will develop evaluation scorecards, report findings, and recommend the best AI solutions for enterprise use while

Qualifications

• 5+ years in software QA, AI testing, model evaluation or related QA field.
• Experience creating test plans, scenarios, tests, scorecards, benchmarks, and measurable criteria.
• Hands-on with Generative AI tools and LLMs, including prompts and scenario testing.
• Ability to evaluate AI tools on accuracy, quality, reliability, safety, bias, privacy, security, usability, performance, and cost.
• Knowledge of GenAI technologies (LLMs, RAG, APIs, enterprise search) and integrations.

Responsibilities

• Test AI platforms using real-world business scenarios, prompts, datasets, and workflows.
• Evaluate AI-generated responses for accuracy, relevance, completeness, and bias.
• Compare AI products on speed, reliability, usability, security, and cost.
• Develop testing plans, scorecards, guidelines, and success criteria.
• Identify AI issues such as incorrect answers, hallucinations, bias, and inconsistencies.
• Review tools for security, privacy, governance with relevant teams.
• Compare multiple products and vendors via hands-on testing and PoC exercises.
• Test capabilities including AI agents, RAG, MCP-based tools, and enterprise connectors.

Skills

AI testing

Quality assurance

Test plan development

Test case development

Data analysis

Education

Bachelor's degree in any relevant discipline

Tools

OpenAI tools

Rosalind

LLMs

APIs

Job description

Job Description

Job Description

Job description

AI Evaluation Specialist

** Location: Continental United States**

Position Overview

We are looking for an experienced AI Evaluation Specialist to test and evaluate AI tools and help determine whether they are accurate, reliable, safe, secure, and ready for business use.

In this role, you will work with different AI platforms and Generative AI solutions, including OpenAI products, life sciences AI models such as Rosalind, and similar enterprise AI tools.

You will work closely with product managers, data scientists, AI engineers, software developers, security and privacy teams, and business users. Your primary responsibility will be to test AI tools using real-world business scenarios, check the quality of their responses, identify issues, and recommend improvements.

You will compare different AI products based on accuracy, response quality, reliability, performance, security, privacy, ease of use, and cost. You will also develop standard testing methods and evaluation reports to help organizations select and adopt the right AI solutions.

Key Responsibilities

• Test different AI platforms, including OpenAI and life sciences AI models such as Rosalind, using real-world business scenarios, prompts, datasets, and workflows.
• Check AI-generated responses for accuracy, relevance, completeness, consistency, reasoning, and ability to follow instructions.
• Evaluate how well AI tools perform, including response speed, reliability, ease of use, accessibility, and their ability to complete assigned tasks.
• Create and maintain AI testing plans, test cases, evaluation scorecards, rating guidelines, and success criteria.
• Identify AI-related issues such as incorrect answers, misleading information, hallucinations, bias, and inconsistent results.
• Review AI tools for security, privacy, data protection, governance, and responsible use in coordination with the appropriate teams.
• Compare multiple AI products and vendors through hands-on testing and proof-of-concept exercises. Evaluate their features, performance, integration needs, and overall cost.
• Test advanced AI capabilities, including AI agents, Retrieval-Augmented Generation (RAG), enterprise search, document and data analysis, APIs, connectors, and Model Context Protocol (MCP)-based tools.
• Work with technical and business teams to share test results, explain identified problems, and recommend the best AI solutions for business needs.
• Keep up with new developments in Generative AI, AI testing methods, model evaluation, AI safety, and responsible AI practices.
• Share knowledge with team members and contribute to a positive and collaborative work environment.

Required Qualifications

• 5+ years of experience in software quality assurance, AI testing, model evaluation, or a related quality assurance field, including hands-on experience testing AI-based products or services.
• Experience creating test plans, test scenarios, test cases, evaluation scorecards, benchmarks, and measurable success criteria.
• Hands-on experience using Generative AI tools and Large Language Models (LLMs), including writing prompts, testing different scenarios, reviewing responses, and identifying incorrect answers or other AI issues.
• Ability to evaluate AI tools based on accuracy, response quality, consistency, reliability, safety, bias, privacy, security, usability, performance, and cost.
• Good understanding of Generative AI technologies, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), AI agents, APIs, enterprise search, and integrations with external tools and systems.
• Strong analytical, problem-solving, communication, and documentation skills.
• Bachelor's degree in any relevant discipline.
• Must be a U.S. Citizen or Lawful Permanent Resident (Green Card Holder).

Preferred Qualifications – A Plus

• Experience testing enterprise AI platforms and AI assistants, including OpenAI products, Rosalind, or similar Generative AI solutions.
• Experience with automated AI testing and evaluation tools, AI-as-a-judge approaches, human review of AI responses, red teaming, or security and adversarial testing.
• Experience testing RAG applications, AI agents, MCP-enabled tools, enterprise connectors, and multi-step AI workflows.
• Experience creating dashboards and reports to track AI quality, testing results, performance trends, risks, and regression issues.
• Ability to work with test data and evaluation results using Python, SQL, Excel, spreadsheets, Jupyter notebooks, or other analytics tools.
• Industry-recognized certifications in Artificial Intelligence, Machine Learning, Data Science, Cybersecurity, Cloud Computing, or Responsible AI.
• Master's degree preferred.