AI Evaluation Specialist

Chenega Corporation

Georgia

Remote

USD 102,000 - 120,000

Full time

2 days ago

Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Teleworking permitted

Health & Wellness benefits

Retirement plan (401k/401a)

Flexible Spending Account

Commuter Benefits

Job summary

Chenega Services & FederalSolutions, LLC seeks an AI Evaluation Specialist to lead structured assessments of enterprise AI tools, including OpenAI and life sciences models like Rosalind. You’ll design evaluation frameworks, test real-world use cases, and compare solutions across quality, safety, privacy, and cost while collaborating with product, security, and business teams. Telework is permitted across the Continental US.

Qualifications

5+ years of experience in model evaluation and QA for AI-enabled products/services
Experience developing structured evaluation plans, test cases, rubrics, benchmarks, and acceptance criteria
Hands-on with generative AI tools and large language models, including prompt design and testing
Ability to assess AI systems across accuracy, safety, privacy, security, usability, and cost
Familiarity with LLM architectures (RAG, APIs, enterprise search) and MCP-enabled tools
Bachelor's degree in a related field

Responsibilities

Plan and execute hands-on evaluations of AI tools using representative business scenarios
Assess solution quality: accuracy, relevance, reliability, usability, and latency
Design evaluation frameworks, rubrics, and acceptance criteria for tools and models
Collaborate with security, privacy, legal, and governance teams on responsible AI considerations
Conduct comparative evaluations across products, configurations, and vendors
Stay updated on generative AI advancements and integrate relevant methods
Share knowledge to strengthen the evaluation program

Skills

AI evaluation

Prompt design

Model evaluation

RAG systems

Data analysis

Education

Bachelor's degree

Master’s degree preferred

Tools

OpenAI tools

Rosalind models

Job description

Summary

Position contingent on contract award - details below are subject to change based on final award.

Come join a company that strives for Extraordinary People and Exceptional Performance! Chenega Services &FederalSolutions, LLC, a Chenega Professional Services’ company, is looking for a AI Evaluation Specialist. The AI Evaluation Specialist will evaluate enterprise AI tools and capabilities to determine their effectiveness, safety, usability, and readiness for business adoption. They’ll partner with product managers, data scientists, AI engineers, software engineers, security and privacy teams, and business stakeholders to define evaluation criteria and conduct structured assessments of generative AI products and platforms, including tools such as OpenAI and life sciences models like Rosalind. They will design repeatable evaluation frameworks, test real-world use cases, analyze model and product behavior, and translate findings into clear recommendations. They will be responsible for comparing AI solutions across quality, accuracy, reliability, safety, privacy and security.

Our company offers employees the opportunity to join a team where there is a robust employee benefits program, management engagement, quality leadership, an atmosphere of teamwork, recognition for performance, and promotion opportunities. We actively strive to channel our highly engaged employee’s knowledge, critical thinking, innovative solutions for our clients.

Responsibilities

Plan and execute hands‑on evaluations of AI tools such as OpenAI and life sciences models like Rosalind using representative business scenarios, workflows, prompts, datasets, and user personas.
Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness.
Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI‑enabled applications.
Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations in collaboration with security, privacy, legal, and governance stakeholders.
Conduct comparative evaluations and proof‑of‑concept assessments across multiple AI products, models, configurations, and vendors, including analysis of functionality, integration requirements, performance, and cost.
Evaluate AI agents and tool‑enabled workflows, including retrieval‑augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)‑enabled capabilities where applicable.
Stay current with advancements in generative AI, model evaluation, AI safety, responsible AI, and emerging enterprise AI products and incorporate relevant methods into the evaluation program.
Engage in knowledge sharing and contribute to building a strong, collaborative team environment.

Qualifications

5+ years of experience in model evaluation, model quality assurance with hands‑on experience evaluating AI‑enabled products or services
Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria
Experience with hands‑on generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes
Experience assessing AI systems across dimensions such as accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost
Ability to demonstrate familiarity with common generative AI architectures and capabilities, including large language models, retrieval‑augmented generation (RAG), AI agents, APIs, enterprise search, and tool or connector integrations
Bachelor's degree

Preferred

Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models like Rosalind or comparable generative AI solutions
Experience with automated LLM evaluation frameworks, model‑as‑a‑judge approaches, human‑in‑the‑loop evaluation, red teaming, or adversarial testing
Experience evaluating RAG systems, AI agents, MCP‑enabled tools, enterprise connectors, or multi‑step AI workflows
Experience developing dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results
Ability to work with datasets and evaluation results using tools such as Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics, identify patterns, and summarize findings
Possession of an industry‑recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI
Master’s degree preferred

Final salary determination based on skill‑set, qualifications, and approved funding.

Many of our jobs come with great benefits – Some offerings are dependent upon the role, work schedule, or location, and may include the following:

Paid Time Off