Remote AI Evaluation Specialist — Enterprise Safety

JRSS; United States; À distance; USD 120 000 - 150 000; Plein temps

Avantages offerts par ce poste

401(k) plan with match

Paid time off

Medical, dental, and vision insurance

Professional growth opportunities

Résumé du poste

JR Software Solutions Inc. is seeking an AI Evaluation Specialist to assess enterprise AI tools for federal public health environments.

You will partner with product managers, data scientists, AI engineers, and security teams to run structured evaluations of generative AI products, including OpenAI tools and Rosalind. The role focuses on evaluating quality, safety, privacy, and cost, translating findings into actionable recommendations for leadership, with a strong emphasis on governance and

Qualifications

• 5+ years in model evaluation or model QA for AI-enabled products.
• Experience developing evaluation plans, test cases, rubrics, benchmarks, and acceptance criteria.
• Hands-on with generative AI tools, prompt design, systematic testing, and failure mode analysis.
• Assess AI systems for accuracy, safety, privacy, security, usability, performance, and cost.
• Familiarity with LLMs, RAG, AI agents, APIs, enterprise search, and tool/connector integrations.
• U.S. Citizen or Lawful Permanent Resident; ability to obtain Public Trust clearance.

Responsabilités

• Plan and execute hands-on evaluations of AI tools using representative scenarios and datasets.
• Assess response quality, factual accuracy, relevance, consistency, and usability.
• Design and maintain evaluation frameworks, scorecards, rubrics, and acceptance criteria for tools and models.
• Evaluate safety, privacy, security, data governance, and responsible AI with stakeholders.
• Conduct comparative evaluations across products, models, configurations, and vendors.
• Translate evaluation findings into clear, actionable recommendations for technical and business stakeholders.
• Stay current with advancements in generative AI and model evaluation methods.
• Contribute to knowledge sharing and a collaborative team environment.

Connaissances

Model evaluation

Evaluation plans

Prompt design

LLMs

APIs

Outils

OpenAI tools

Rosalind

Description du poste

JR Software Solutions Inc. is seeking an AI Evaluation Specialist to assess enterprise AI tools for federal public health environments.

You will partner with product managers, data scientists, AI engineers, and security teams to run structured evaluations of generative AI products, including OpenAI tools and Rosalind. The role focuses on evaluating quality, safety, privacy, and cost, translating findings into actionable recommendations for leadership, with a strong emphasis on governance and