AI Evaluation Specialist

JRSS; United States; Remote; USD 120,000 - 150,000; Full time

Benefits offered by this job

401(k) plan with match

Paid time off

Medical, dental, and vision insurance

Professional growth opportunities

Job summary

JR Software Solutions Inc. is seeking an AI Evaluation Specialist to assess enterprise AI tools for federal public health environments.

You will partner with product managers, data scientists, AI engineers, and security teams to run structured evaluations of generative AI products, including OpenAI tools and Rosalind. The role focuses on evaluating quality, safety, privacy, and cost, translating findings into actionable recommendations for leadership, with a strong emphasis on governance and

Qualifications

• 5+ years in model evaluation or model QA for AI-enabled products.
• Experience developing evaluation plans, test cases, rubrics, benchmarks, and acceptance criteria.
• Hands-on with generative AI tools, prompt design, systematic testing, and failure mode analysis.
• Assess AI systems for accuracy, safety, privacy, security, usability, performance, and cost.
• Familiarity with LLMs, RAG, AI agents, APIs, enterprise search, and tool/connector integrations.
• U.S. Citizen or Lawful Permanent Resident; ability to obtain Public Trust clearance.

Responsibilities

• Plan and execute hands-on evaluations of AI tools using representative scenarios and datasets.
• Assess response quality, factual accuracy, relevance, consistency, and usability.
• Design and maintain evaluation frameworks, scorecards, rubrics, and acceptance criteria for tools and models.
• Evaluate safety, privacy, security, data governance, and responsible AI with stakeholders.
• Conduct comparative evaluations across products, models, configurations, and vendors.
• Translate evaluation findings into clear, actionable recommendations for technical and business stakeholders.
• Stay current with advancements in generative AI and model evaluation methods.
• Contribute to knowledge sharing and a collaborative team environment.

Skills

Model evaluation

Evaluation plans

Prompt design

LLMs

APIs

Tools

OpenAI tools

Rosalind

Job description

Location: Remote (Continental U.S.)

Company: JR Software Solutions Inc. (JRSS)

Employment Type: Full-time

About JR Software Solutions Inc.

JRSS is a certified Women-Owned Small Business (WOSB), Small Disadvantaged Business (SDB), and HUBZone business headquartered in Tampa, FL. We specialize in Cloud Services, AI and Machine Learning, Big Data and BI Analytics, Enterprise Resource Planning, IT Project Management, and Staff Augmentation for Federal Civilian agencies (e.g., CDC) and Fortune 500 clients. Our AI/ML teams help public health agencies adopt generative AI responsibly, with the rigor, transparency, and safeguards that federal missions require.

Position Overview

The JRSS AI/ML Team is seeking an AI Evaluation Specialist to determine the effectiveness, safety, usability, and readiness of enterprise AI tools for adoption in a federal public health environment. You will partner with product managers, data scientists, AI engineers, and security and privacy teams to run structured assessments of generative AI products and platforms, including OpenAI tools and life sciences models such as Rosalind. Your findings will guide which AI solutions are adopted, comparing them across quality, accuracy, reliability, safety, privacy, and security.

Key Responsibilities

• Plan and execute hands-on evaluations of AI tools, including OpenAI products and life sciences models such as Rosalind, using representative business scenarios, workflows, prompts, datasets, and user personas.
• Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness.
• Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications.
• Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations in collaboration with security, privacy, legal, and governance stakeholders.
• Conduct comparative evaluations and proof-of-concept assessments across multiple AI products, models, configurations, and vendors, including analysis of functionality, integration requirements, performance, and cost.
• Evaluate AI agents and tool-enabled workflows, including retrieval-augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)-enabled capabilities.
• Translate evaluation findings into clear, actionable recommendations for technical and business stakeholders.
• Stay current with advancements in generative AI, model evaluation, AI safety, and enterprise AI products, and incorporate relevant methods into the evaluation program.
• Contribute to knowledge sharing and a strong, collaborative team environment.

Required Qualifications

• 5+ years of experience in model evaluation or model quality assurance, with hands-on experience evaluating AI-enabled products or services.
• Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.
• Hands-on experience with generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes.
• Experience assessing AI systems for accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost.
• Familiarity with common generative AI architectures and capabilities, including LLMs, RAG, AI agents, APIs, enterprise search, and tool or connector integrations.
• Must be a U.S. Citizen or Lawful Permanent Resident (Green Card holder).
• Ability to obtain and maintain a Public Trust clearance.

Preferred Qualifications

• Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models such as Rosalind, or comparable generative AI solutions.
• Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing.
• Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows.
• Experience building dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results.
• Proficiency with Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics and summarize findings.
• Industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI.

What We Offer

• 401(k) plan with a 4% company match.
• Two weeks of paid time off.
• Comprehensive medical, dental, and vision insurance.
• Ongoing investment in professional growth and development.

Equal Opportunity Statement

JR Software Solutions Inc. is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status.