Remote AI Evaluation Lead

Chenega Professional Services Strategic Business Unit; Georgia; Remote; USD 102,000 - 120,000; Full time

Benefits offered by this job

Teleworking permitted

Health & Wellness benefits

Retirement plan (401k/401a)

Flexible Spending Account

Commuter Benefits

Job summary

Chenega Services & FederalSolutions, LLC seeks an AI Evaluation Specialist to lead structured assessments of enterprise AI tools, including OpenAI and life sciences models like Rosalind. You’ll design evaluation frameworks, test real-world use cases, and compare solutions across quality, safety, privacy, and cost while collaborating with product, security, and business teams. Telework is permitted across the Continental US.

Qualifications

• 5+ years of experience in model evaluation and QA for AI-enabled products/services
• Experience developing structured evaluation plans, test cases, rubrics, benchmarks, and acceptance criteria
• Hands-on with generative AI tools and large language models, including prompt design and testing
• Ability to assess AI systems across accuracy, safety, privacy, security, usability, and cost
• Familiarity with LLM architectures (RAG, APIs, enterprise search) and MCP-enabled tools
• Bachelor's degree in a related field

Responsibilities

• Plan and execute hands-on evaluations of AI tools using representative business scenarios
• Assess solution quality: accuracy, relevance, reliability, usability, and latency
• Design evaluation frameworks, rubrics, and acceptance criteria for tools and models
• Collaborate with security, privacy, legal, and governance teams on responsible AI considerations
• Conduct comparative evaluations across products, configurations, and vendors
• Stay updated on generative AI advancements and integrate relevant methods
• Share knowledge to strengthen the evaluation program

Skills

AI evaluation

Prompt design

Model evaluation

RAG systems

Data analysis

Education

Bachelor's degree

Master’s degree preferred

Tools

OpenAI tools

Rosalind models

Job description

Chenega Services & FederalSolutions, LLC seeks an AI Evaluation Specialist to lead structured assessments of enterprise AI tools, including OpenAI and life sciences models like Rosalind. You’ll design evaluation frameworks, test real-world use cases, and compare solutions across quality, safety, privacy, and cost while collaborating with product, security, and business teams. Telework is permitted across the Continental US.