Remote Enterprise AI Evaluation Specialist

Metas Solutions; Decatur (GA); Remote; USD 110,000 - 160,000; Full time

Job summary

Metas Solutions seeks an AI Evaluation Specialist to assess enterprise AI tools and platforms, including OpenAI and Rosalind, in a remote role. You will define evaluation criteria, design repeatable frameworks, and partner with product teams, data scientists, and security to ensure safe, effective AI adoption.

The role requires 5+ years in model evaluation, strong test design, and hands-on experience with LLMs, RAG, and enterprise AI workflows. U.S. citizenship or green card is required.

Qualifications

• 5+ years of experience in model evaluation and QA for AI-enabled products or services.
• Experience developing structured evaluation plans, test cases, rubrics, benchmarks, and acceptance criteria.
• Hands-on experience with generative AI tools and large language models, including prompt design and testing.
• Familiarity with evaluating AI systems across accuracy, relevance, safety, privacy, security, usability, and cost.
• Knowledge of architectures like LLMs, RAG, AI agents, APIs, enterprise search, and tool integrations.
• Bachelor's degree required; U.S. citizenship or Lawful Permanent Resident required.

Responsibilities

• Plan and execute hands-on evaluations of AI tools and models using practical business scenarios and datasets.
• Assess response quality, accuracy, relevance, and usability across AI products and platforms.
• Design and maintain evaluation frameworks, scorecards, rubrics, and acceptance criteria.
• Evaluate safety, privacy, governance, and compliance with security teams and stakeholders.
• Conduct comparative evaluations across products, configurations, and vendors.
• Stay updated on advancements in generative AI and share knowledge with the team.
• Collaborate with product managers, data scientists, security, and legal teams.

Skills

Model evaluation

Evaluation plans

Prompt design

LLM tools

Benchmarks

Education

Bachelor's degree

Master's degree preferred

Tools

Python

SQL

Notebooks

Analytics platforms

Job description

Metas Solutions seeks an AI Evaluation Specialist to assess enterprise AI tools and platforms, including OpenAI and Rosalind, in a remote role. You will define evaluation criteria, design repeatable frameworks, and partner with product teams, data scientists, and security to ensure safe, effective AI adoption.

The role requires 5+ years in model evaluation, strong test design, and hands-on experience with LLMs, RAG, and enterprise AI workflows. U.S. citizenship or green card is required.