Enterprise AI Evaluation & Risk Specialist

System One, Inc.; Atlanta (GA); Vor Ort; USD 140.000 - 171.000; Vollzeit

Zusammenfassung

System One, Inc. is seeking an AI Evaluation Specialist to assess enterprise AI tools and platforms for safety, usability, and readiness.

You will design repeatable evaluation frameworks and collaborate with product managers, data scientists, security, and business stakeholders to deliver actionable recommendations. The role requires extensive hands-on evaluation of generative AI tools, including OpenAI-style models and life sciences models, with a focus on accuracy, reliability, and responsible

Qualifikationen

• 5+ years of experience in model evaluation or QA for AI-enabled products/services.
• Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable criteria.
• Hands-on experience with generative AI tools and large language models, including prompt design and testing.
• Ability to assess AI systems across accuracy, safety, privacy, security, usability, performance, and cost.
• Familiarity with architectures like LLMs, RAG, AI agents, APIs, and enterprise search.
• Bachelor's degree required; US Citizenship or Green Card holder as per government contract.

Aufgaben

• Plan and execute hands-on evaluations of AI tools using business scenarios, prompts, datasets, and personas.
• Assess response quality, factual accuracy, relevance, safety, reliability, latency, usability, and task effectiveness.
• Design and maintain evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools and models.
• Evaluate safety, privacy, security, data handling, governance, and responsible AI with security and legal teams.
• Conduct comparative evaluations across products, models, configurations, and vendors including cost analysis.
• Evaluate AI agents and MCP-enabled tools, including connectors and APIs, where applicable.
• Stay current with advancements in generative AI and model evaluation; share knowledge with the team.
• Collaborate with cross-functional stakeholders to drive evaluation program improvements.

Kenntnisse

AI evaluation

Model evaluation

Prompt design

Structured evaluation plans

RAG

LLMs

Enterprise AI

Security awareness

Ausbildung

Bachelor's degree

Tools

Python

SQL

Jobbeschreibung

System One, Inc. is seeking an AI Evaluation Specialist to assess enterprise AI tools and platforms for safety, usability, and readiness.

You will design repeatable evaluation frameworks and collaborate with product managers, data scientists, security, and business stakeholders to deliver actionable recommendations. The role requires extensive hands-on evaluation of generative AI tools, including OpenAI-style models and life sciences models, with a focus on accuracy, reliability, and responsible