Senior AI Evaluation & Knowledge Engineer

Supermicro; San Jose (CA); Vor Ort; USD 160.000 - 200.000; Vollzeit

Zusammenfassung

Supermicro is seeking a Sr. Software Engineer to advance an AI evaluation and knowledge platform for handling AI Server and rack-related customer issues.

You will ensure agent outputs are accurate, actionable, and continuously improved through evaluation systems and a robust knowledge base. You will work with Domain SMEs to validate troubleshooting steps, design evaluation frameworks, and optimize prompts for higher first-time accuracy and usability.

Qualifikationen

• 2+ years of experience with LLM / Generative AI applications and evaluation.
• Experience in prompt engineering and reducing hallucinations.
• Experience systematically evaluating LLM or agent outputs.
• Experience with knowledge bases / RAG and retrieval pipelines.
• Strong analytical and cross-team communication skills.
• Ability to translate domain feedback into actionable improvements.
• Familiarity with Python to implement evaluation/data processing scripts.
• Experience in technical support or vertical AI agent applications is a plus.

Aufgaben

• Design and maintain the agent evaluation framework (test sets, automated scoring, human sampling review, regression testing).
• Own prompt engineering and structured output optimization to improve step accuracy.
• Manage Knowledge Base quality processes (content review, version control, retirement of outdated content, writing back new cases).
• Analyze agent errors and hallucinations, propose and implement improvements.
• Work with Domain SMEs to validate troubleshooting steps.
• Establish feedback loops from on-site results to agent improvements.
• Define and track success metrics (e.g., first-time accuracy, step usability).
• Support RAG retrieval quality tuning and knowledge presentation improvements.

Kenntnisse

LLM / Generative AI experience

Prompt engineering

Evaluation of LLM outputs

Cross-team collaboration

Data analysis

Python programming

Tools

Python

Jobbeschreibung

Supermicro is seeking a Sr. Software Engineer to advance an AI evaluation and knowledge platform for handling AI Server and rack-related customer issues.

You will ensure agent outputs are accurate, actionable, and continuously improved through evaluation systems and a robust knowledge base. You will work with Domain SMEs to validate troubleshooting steps, design evaluation frameworks, and optimize prompts for higher first-time accuracy and usability.