AI Evaluation Specialist

Metas Solutions; Decatur (GA); Remote; USD 110,000 - 160,000; Full time

Job summary

Metas Solutions seeks an AI Evaluation Specialist to assess enterprise AI tools and platforms, including OpenAI and Rosalind, in a remote role. You will define evaluation criteria, design repeatable frameworks, and partner with product teams, data scientists, and security to ensure safe, effective AI adoption.

The role requires 5+ years in model evaluation, strong test design, and hands-on experience with LLMs, RAG, and enterprise AI workflows. U.S. citizenship or green card is required.

Qualifications

• 5+ years of experience in model evaluation and QA for AI-enabled products or services.
• Experience developing structured evaluation plans, test cases, rubrics, benchmarks, and acceptance criteria.
• Hands-on experience with generative AI tools and large language models, including prompt design and testing.
• Familiarity with evaluating AI systems across accuracy, relevance, safety, privacy, security, usability, and cost.
• Knowledge of architectures like LLMs, RAG, AI agents, APIs, enterprise search, and tool integrations.
• Bachelor's degree required; U.S. citizenship or Lawful Permanent Resident required.

Responsibilities

• Plan and execute hands-on evaluations of AI tools and models using practical business scenarios and datasets.
• Assess response quality, accuracy, relevance, and usability across AI products and platforms.
• Design and maintain evaluation frameworks, scorecards, rubrics, and acceptance criteria.
• Evaluate safety, privacy, governance, and compliance with security teams and stakeholders.
• Conduct comparative evaluations across products, configurations, and vendors.
• Stay updated on advancements in generative AI and share knowledge with the team.
• Collaborate with product managers, data scientists, security, and legal teams.

Skills

Model evaluation

Evaluation plans

Prompt design

LLM tools

Benchmarks

Education

Bachelor's degree

Master's degree preferred

Tools

Python

SQL

Notebooks

Analytics platforms

Job description

Metas Solutions excels in providing strategic consulting, program management, and data-driven analytics to federal public health agencies. Our expertise enhances health systems and delivers measurable outcomes, helping agencies achieve their mission. We collaborate with federal partners to implement agile solutions for national public health initiatives, driving impactful changes through cutting-edge programs that strengthen communities and promote well-being. See www.metassolutions.com for further details about us and careers with Metas.

Job Description

Metas Solutions has an "immediate opening" for a talented AI Evaluation Specialist to evaluate enterprise AI tools and capabilities to determine their effectiveness, safety, usability, and readiness for business adoption. In this Remote position, you’ll partner with product managers, data scientists, AI engineers, software engineers, security and privacy teams, and business stakeholders to define evaluation criteria and conduct structured assessments of generative AI products and platforms, including tools such as OpenAI and life sciences models like Rosalind. You will design repeatable evaluation frameworks, test real-world use cases, analyze model and product behavior, and translate findings into clear recommendations. You will be responsible for comparing AI solutions across quality, accuracy, reliability, safety, privacy and security.

Responsibilities

• Plan and execute hands-on evaluations of AI tools such as OpenAI and life sciences models like Rosalind using representative business scenarios, workflows, prompts, datasets, and user personas
• Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness
• Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications
• Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations in collaboration with security, privacy, legal, and governance stakeholders
• Conduct comparative evaluations and proof-of-concept assessments across multiple AI products, models, configurations, and vendors, including analysis of functionality, integration requirements, performance, and cost
• Evaluate AI agents and tool-enabled workflows, including retrieval-augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)-enabled capabilities where applicable
• Stay current with advancements in generative AI, model evaluation, AI safety, responsible AI, and emerging enterprise AI products and incorporate relevant methods into the evaluation program
• Engage in knowledge sharing and contribute to building a strong, collaborative team environment

Qualifications

• 5+ years of experience in model evaluation, model quality assurance with hands-on experience evaluating AI-enabled products or services
• Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria
• Experience with hands-on generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes
• Experience assessing AI systems across dimensions such as accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost
• Ability to demonstrate familiarity with common generative AI architectures and capabilities, including large language models, retrieval-augmented generation (RAG), AI agents, APIs, enterprise search, and tool or connector integrations
• Bachelor's degree
• Must be a U.S. Citizen or Lawful Permanent Resident (Green Card Holder)

Additional Qualifications

• Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models like Rosalind or comparable generative AI solutions
• Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing
• Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows
• Experience developing dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results
• Ability to work with datasets and evaluation results using tools such as Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics, identify patterns, and summarize findings
• Possession of an industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI
• Master's degree preferred

Security Requirements

• Must be US Citizen and with the ability to obtain a US Government security clearance (Public Trust 5) within a reasonable period
• Market competitive salary, commensurate with experience and education
• Comprehensive benefits package available, Medical, Dental, Vision and Life Insurance, Paid Time Off (PTO), 401K with company match, growth, and promotion opportunities

Notice on Candidate AI Usage

Metas Solutions is committed to maintaining a fair and competitive interview process for all candidates, based on their experience, skills, and education. To preserve the integrity and fairness of our interview process, candidates are expected to respond to all interview questions based on their own knowledge, skills, and experience. The use of artificial intelligence (AI) tools or other third-party resources to generate, assist with, or provide interview responses during phone, virtual, or in-person interviews is not permitted. Any use of AI-assisted responses during the interview process may result in disqualification from further consideration for employment.

We are an Equal Opportunity Employer/Veterans/Disabled