Senior AI Quality & Evaluation Lead (Remote)

Metas Solutions; United States; Remote; USD 110,000 - 150,000; Full time

Benefits offered by this job

Medical, Dental, Vision Insurance

401(k) with company match

Paid Time Off

Career growth opportunities

Job summary

Metas Solutions has an immediate opening for an AI Evaluation Specialist to evaluate enterprise AI tools and capabilities for effectiveness, safety, usability, and readiness for business adoption.

This remote position partners with product managers, data scientists, AI engineers, security and governance teams to define evaluation criteria and conduct structured assessments of AI products, including OpenAI and Rosalind models.

Qualifications

• Bachelor's degree required.
• 5+ years of model evaluation experience.
• Experience with generative AI tools and LLMs.
• Ability to design evaluation plans, rubrics, and acceptance criteria.
• Familiarity with RAG, enterprise search, and tool integrations.

Responsibilities

• Plan and execute hands-on evaluations of AI tools using business scenarios.
• Assess AI solutions for quality, accuracy, usability, safety, and cost.
• Design evaluation frameworks, scorecards, rubrics, and acceptance criteria.
• Evaluate safety, privacy, and governance with stakeholders.
• Conduct comparative evaluations across multiple AI products and workflows.
• Stay current with advances in generative AI and evaluation methods.
• Contribute to a collaborative team environment.

Skills

AI evaluation

Structured evaluation plans

Prompt design

LLM evaluation

Dashboard/reporting

Python

SQL

Ethical AI

Education

Bachelor's degree

Master's degree preferred

Tools

Python

SQL

Spreadsheets

Notebooks

Job description

Metas Solutions has an immediate opening for an AI Evaluation Specialist to evaluate enterprise AI tools and capabilities for effectiveness, safety, usability, and readiness for business adoption.

This remote position partners with product managers, data scientists, AI engineers, security and governance teams to define evaluation criteria and conduct structured assessments of AI products, including OpenAI and Rosalind models.