Remote AI Evaluation Specialist - Enterprise Tools

Metas Solutions; Atlanta (GA); Teletrabalho; USD 110 000 - 170 000; Tempo integral

Resumo da oferta

Metas Solutions is seeking an AI Evaluation Specialist for a remote role to evaluate enterprise AI tools for safety, accuracy, and usability, including OpenAI and Rosalind. You will collaborate with product managers, data scientists, and security teams to design repeatable evaluation frameworks and provide clear recommendations.

The ideal candidate has 5+ years in model evaluation, experience with prompts and LLM testing, and a bachelor's degree. U.S.

Qualificações

• 5+ years of experience in model evaluation.
• Experience with structured evaluation plans, test cases, and rubrics.
• Experience with hands-on generative AI tools and LLM testing.
• Experience evaluating AI systems across multiple dimensions (accuracy, safety, cost).
• Bachelor's degree required; U.S. citizenship or green card.

Responsabilidades

• Plan and execute hands-on evaluations of AI tools using business scenarios.
• Assess AI solutions for response quality, factual accuracy, relevance, and usability.
• Design and maintain standardized evaluation frameworks, scorecards, and acceptance criteria.
• Evaluate safety, privacy, security, data handling, and governance considerations.
• Conduct comparative evaluations across multiple AI products and configurations.
• Stay current with advancements in generative AI and evaluation methods.
• Collaborate with security, privacy, legal, and governance teams.

Conhecimentos

Model evaluation

LLM evaluation

Rubrics & benchmarks

Prompt design

Formação académica

Bachelor's degree

Master's degree preferred

Ferramentas

Python

SQL

Descrição da oferta de emprego

Metas Solutions is seeking an AI Evaluation Specialist for a remote role to evaluate enterprise AI tools for safety, accuracy, and usability, including OpenAI and Rosalind. You will collaborate with product managers, data scientists, and security teams to design repeatable evaluation frameworks and provide clear recommendations.

The ideal candidate has 5+ years in model evaluation, experience with prompts and LLM testing, and a bachelor's degree. U.S.