AI Evaluation Lead — Testing, QA & Responsible AI

Onsite It Advisors LLC; Atlanta (GA); Remote; USD 90,000 - 130,000; Full time

Job summary

Onsite It Advisors LLC in the United States seeks an experienced AI Evaluation Specialist to test and evaluate AI tools for accuracy, reliability, safety, and business readiness. You will work across platforms, including OpenAI products and Rosalind, to design tests using real-world scenarios.

Collaborating with product managers, data scientists, security and privacy teams, you will develop evaluation scorecards, report findings, and recommend the best AI solutions for enterprise use while

Qualifications

• 5+ years in software QA, AI testing, model evaluation or related QA field.
• Experience creating test plans, scenarios, tests, scorecards, benchmarks, and measurable criteria.
• Hands-on with Generative AI tools and LLMs, including prompts and scenario testing.
• Ability to evaluate AI tools on accuracy, quality, reliability, safety, bias, privacy, security, usability, performance, and cost.
• Knowledge of GenAI technologies (LLMs, RAG, APIs, enterprise search) and integrations.

Responsibilities

• Test AI platforms using real-world business scenarios, prompts, datasets, and workflows.
• Evaluate AI-generated responses for accuracy, relevance, completeness, and bias.
• Compare AI products on speed, reliability, usability, security, and cost.
• Develop testing plans, scorecards, guidelines, and success criteria.
• Identify AI issues such as incorrect answers, hallucinations, bias, and inconsistencies.
• Review tools for security, privacy, governance with relevant teams.
• Compare multiple products and vendors via hands-on testing and PoC exercises.
• Test capabilities including AI agents, RAG, MCP-based tools, and enterprise connectors.

Skills

AI testing

Quality assurance

Test plan development

Test case development

Data analysis

Education

Bachelor's degree in any relevant discipline

Tools

OpenAI tools

Rosalind

LLMs

APIs

Job description

Onsite It Advisors LLC in the United States seeks an experienced AI Evaluation Specialist to test and evaluate AI tools for accuracy, reliability, safety, and business readiness. You will work across platforms, including OpenAI products and Rosalind, to design tests using real-world scenarios.

Collaborating with product managers, data scientists, security and privacy teams, you will develop evaluation scorecards, report findings, and recommend the best AI solutions for enterprise use while