Remote AI Evaluation Specialist

Chenega Services & Federal Solutions; Oregon; Hybrid; USD 102,000 - 120,000; Full time

Benefits offered by this job

PTO

Holidays

Medical

Dental

Vision

401k

Voluntary Benefits

Life Insurance

Commuter Benefits

Legal Insurance

Job summary

Chenega Services & Federal Solutions is seeking an AI Evaluation Specialist to assess enterprise AI tools and platforms for effectiveness, safety, and readiness for business adoption. The role collaborates with product managers, data scientists, and security teams to design evaluation criteria, develop frameworks, and deliver actionable recommendations.

The position emphasizes evaluating OpenAI tools and life sciences models like Rosalind, with a focus on quality, safety, privacy, and cost.

Qualifications

• 5+ years of experience in model evaluation and QA with hands-on experience evaluating AI-enabled products or services.
• Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.
• Experience with hands-on generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes.
• Experience assessing AI systems across dimensions such as accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost.
• Ability to demonstrate familiarity with common generative AI architectures and capabilities, including large language models, retrieval-augmented generation (RAG), AI agents, APIs, enterprise search, and tool or connector integrations.
• Bachelor's degree

Responsibilities

• Plan and execute hands-on evaluations of AI tools using representative business scenarios, prompts, datasets, and user personas.
• Assess AI solutions for quality metrics such as accuracy, relevance, and safety.
• Design and maintain evaluation frameworks, scorecards, rubrics, and acceptance criteria.
• Collaborate with security, privacy, legal, and governance teams on responsible AI considerations.
• Conduct comparative evaluations across products, models, configurations, and vendors.

Skills

Model evaluation

Evaluation plans

Prompt design

LLMs

RAG

Security & privacy

Education

Bachelor's degree

Master's degree

Tools

Python

SQL

Spreadsheets

Notebooks

Analytics platforms

Job description

Chenega Services & Federal Solutions is seeking an AI Evaluation Specialist to assess enterprise AI tools and platforms for effectiveness, safety, and readiness for business adoption. The role collaborates with product managers, data scientists, and security teams to design evaluation criteria, develop frameworks, and deliver actionable recommendations.

The position emphasizes evaluating OpenAI tools and life sciences models like Rosalind, with a focus on quality, safety, privacy, and cost.