Senior AI Evaluation & Knowledge Engineer

Supermicro; Wayne (CA); Vor Ort; USD 160.000 - 200.000; Vollzeit

Zusammenfassung

Supermicro is building an AI Agent-powered platform to handle customer issues related to AI Servers and Racks. This role focuses on the quality of agent outputs, evaluation systems, and continuous improvement of the Knowledge Base, ensuring the agent produces accurate and actionable troubleshooting steps.

The ideal candidate will design evaluation frameworks, own prompt engineering, manage knowledge base quality, analyze errors, and work with domain SMEs to validate steps, while tracking metrics

Qualifikationen

• 2+ years experience in LLM / Gen AI evaluation.
• Strong prompt engineering skills to reduce hallucinations.
• Experience evaluating LLM/Agent outputs.
• Familiarity with RAG or Knowledge Base management.
• Excellent analytical and cross-team communication.
• Ability to turn domain feedback into concrete improvements.
• Proficient in Python for basic evaluation scripts.
• Experience in technical support or AI agent applications is a plus.
• Experience with LLM-as-judge or A/B testing is a plus.
• Experience with knowledge engineering or data annotation is a plus.
• Ability to work with non-AI domain experts.

Aufgaben

• Design and maintain agent evaluation framework (test sets, scoring, sampling, regression tests).
• Own prompt engineering and output optimization to improve accuracy.
• Manage knowledge base quality processes (reviews, versioning, retirement, updates).
• Analyze agent errors and hallucinations; implement improvements.
• Collaborate with domain SMEs to validate troubleshooting steps.
• Establish feedback loop from on-site results to agent improvements.
• Define and track success metrics (first-time accuracy, step usability).
• Support RAG retrieval quality tuning and knowledge presentation improvements.

Kenntnisse

LLM / Gen AI

Prompt Engineering

Evaluation pipelines

RAG Knowledge Base

Analytics

Domain feedback

Python

Tech support AI

LLM-as-judge

Knowledge engineering

Cross-domain collab

Tools

Python

Jobbeschreibung

Supermicro is building an AI Agent-powered platform to handle customer issues related to AI Servers and Racks. This role focuses on the quality of agent outputs, evaluation systems, and continuous improvement of the Knowledge Base, ensuring the agent produces accurate and actionable troubleshooting steps.

The ideal candidate will design evaluation frameworks, own prompt engineering, manage knowledge base quality, analyze errors, and work with domain SMEs to validate steps, while tracking metrics